ENH: dataframe memory usage
State: closed

Closes [#6852](https://github.com/pydata/pandas/issues/6852)

For a sample DataFrame:

``` python
df = pd.DataFrame({ 'float' : np.random.randn(10000000), 'int' : np.random.randint(0,5,size=10000000), 'date' : Timestamp('20130101'), 'string' : 'foo', 'smallint' : np.random.randint(0,5,size=10000000).astype('int16') })
```

The memory usage of the elements of each column (in units of bytes) are returned by accessing the `meminfo` method. This returns a Series:

``` python
>>> df.meminfo()
date        80000000
float       80000000
int         80000000
smallint    20000000
string      80000000
dtype: int64
```

Also, the total memory usage (in a human readable format) can be found via the `info(memory_usage=True)` method:

``` python
>>> df.info(memory_usage=True)
<class 'pandas.core.frame.DataFrame'>
Int64Index: 10000000 entries, 0 to 9999999
Data columns (total 5 columns):
date        datetime64[ns]
float       float64
int         int64
smallint    int16
string      object
dtypes: datetime64[ns](1), float64(1), int16(1), int64(1), object(1)
memory usage: 324.2 MB
```


Labels:
name: Performance
description: Memory or execution speed performance
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
Need some tests for this (in test_format). You don't need to test the exact results, but rather that the right fields appear (e.g. the mem usage field when specifying the option).

Also would need a small example for v0.14.1.txt (which will serve as the release note as well). You can use the same example in the docs (I guess in FAQ unless you can find a better place).

@asobrien can you rebase on master?

@jreback I think we're ready to roll.

thinking that you may want to have the memory usage of the index included as well. not sure how to represent this in the Series return (which is indexed by columns).

also  need to squash these commits down to a single commit

@jreback Thanks for the notes... I'll make the above changes.

@jreback This should make the fixes you made in the comments.

@jreback The FAQ has been updated. Here's a screenshot of the renderer output:

![faq_mem_1](https://cloud.githubusercontent.com/assets/223897/3945729/6f405996-264f-11e4-865d-8ced00a6c823.png)

![faq_mem_2](https://cloud.githubusercontent.com/assets/223897/3945731/769d5c3e-264f-11e4-9c75-19e5adba654d.png)

@asobrien 

what is the purpose of the `as_names` parameter?
this should act like other routines and return the index (columns) as the index of the returned series. anything else is really odd and doesn't follow the pandas API.

further why does this not handle duplicates?

@jreback My thoughts on the `as_names` parameter are as follows:

In the case that a DataFrame has columns with duplicate names, such as:

```
df = DataFrame(np.random.randint(100, size=(25,4)), columns=['a','b','c','c'])
```

The `meminfo` method is designed to return a Series with memory usage of each column. To do this, the columns must be uniquely identified in the Series index that is returned, so by default the columns are referred to by index. If the columns were not referred to by index then the Series yielded by `meminfo` would appear like:

```
Index    200
a        200
b        200
c        400
dtype: int64
```

rather than as currently implemented:

```
Index    200
0        200
1        200
2        200
3        200
dtype: int64
```

The columns which have identical names will collapse in the Series index that is returned. The `as_names` parameter was included to force the Series index to use names in the case all columns names are unique. I anticipate such behavior useful during interactive work.

Perhaps the `as_names` parameter could be changed to dissuade ambiguity? Let me know if you have any other thoughts on how to implement the method.

@asobrien Why would an output like this not be possible (or desirable)?

```
index    200
a        200
b        200
c        200
c        200
dtype: int64
```

so just with the duplicate column names preserved?

Furthermore, would it be possible to handle multi-index seperately (I presume the different levels are now all summed in 'Index')?

The way `meminfo` is currently implemented:

```
d = Series(dict([ (key, self[key].values.nbytes)
     for key in self.keys() ]))
return sizes.append(Series(d))
```

collapses duplicate names on the Series index. We could implement it
another way to get the desired result.

you need to iterate over the index, use iloc and construct a list, then set the columns

what u r doing only working for non duplicates
no reason for it not to work always

@asobrien can you rebase and address comments above. like to get this in for 0.15.0

@jreback I removed the `as_names`. The Series from `meminfo` method always returns Index with column names even if they are duplicates. Hopefully we're good to roll...

The faq was also updated to reflect this:

![faq_meminfo](https://cloud.githubusercontent.com/assets/223897/4228960/c7244f04-3965-11e4-96f1-d75755c0443e.png)

@jorisvandenbossche should we make this default True (so it DOES print with `df.info()` by default)? (currently looks False)

@jreback made the mentions to `display.memory_usage` in the above files. 

@jreback @jorisvandenbossche Currently `display.memory_usage` is set to False in config_init.py. The default value False was chosen based on a previous conversation https://github.com/pydata/pandas/issues/6852#issuecomment-46896394. Let me know what you decide the default option should be and I'll push the changes.

@jorisvandenbossche ?

hmm, no strong opinion. First thought was to keep it False as default, but on the other hand maybe not many people will find it otherwise

@TomAugspurger @hayd @cpcloud 
cc @shoyer

opinions on making the default True (so it DOES show mem usage) on `df.info()` by default

@asobrien can you write an e-mail to pydata@googlegroups.com, put in a link to this PR, show the proposed changing default to True for 0.15.0 and asking for opinions? thanks

My inclination is to default it to false, but as long as there's an option I don't have a strong opinion.

@jreback will do...

On Fri, Sep 19, 2014 at 8:41 AM, jreback notifications@github.com wrote:

> @asobrien https://github.com/asobrien can you write an e-mail to
> pydata@googlegroups.com, put in a link to this PR, show the proposed
> changing default to True for 0.15.0 and asking for opinions? thanks
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/7619#issuecomment-56171700.

I'm +1 on defaulting to `memory_usage=True` or even dropping the option to turn it off at all. This is a nice addition, and I think I would always want to see it in `df.info()`.

IO posted here: https://groups.google.com/forum/#!topic/pydata/D_BLGew_3Vo

I am thinking that making default True is a good thing, but let's see feedback

Also +1 on including this by default in df.info, very interesting/useful.

@jreback Thanks for posting to google groups... I posted to it over the weekend but it needed approval. Glad to see a message on the board.

ok why don't u change the default to True 

@jreback Default has been set to True. We're ready to roll.

Can someone comment on how to proceed with Categorical? Currently this line will error:

```
result = Series([ c.values.nbytes for col, c in self.iteritems() ],
                        index=self.columns)
```

as `c.values.nbytes` is not available if `values` is a Categorical? Should I simple add a `nbytes` property as `self._codes.nbytes + self._categories._values.nbytes`?