Repository: pandas-dev/pandas

Issue:
ENH: showing memory used
maybe a function `nbytes()` to return memory usage

maybe as a tuple : the Series in [6] and then a series of the axis data?

```
In [5]: df = DataFrame({ 'float' : np.random.randn(1000), 'int' : np.random.randint(0,5,size=1000), 'date' : Timestamp('20130101'), 'string' : 'foo' })

In [6]: Series(dict([ (b.dtype,b.values.nbytes) for b in df._data.blocks ]))
Out[6]: 
datetime64[ns]    8000
int64             8000
float64           8000
object            8000
dtype: int64

In [7]: df.index.nbytes
Out[7]: 8000

In [8]: df.columns.nbytes
Out[8]: 32
```



Discussion:
my 2c. i think `meminfo` should return your series and a property `nbytes` should return `df.meminfo().sum()`

Using the following DataFrame as an example:

``` python
In [11]:  df = pd.DataFrame({ 'float' : np.random.randn(10000000), 'int' : np.random.randint(0,5,size=10000000), 'date' : Timestamp('20130101'), 'string' : 'foo', 'smallint' : np.random.randint(0,5,size=10000000).astype('int16') })
```

I have implemented `meminfo()` as as DataFrame method, resulting in a Series. Note that units can be specified:

``` python

In [13]: # Default units = B
In [14]: df.meminfo()
Out[14]: 
date        80000000
float       80000000
int         80000000
smallint    20000000
string      80000000
dtype: float64

In [15]: df.meminfo(units="MB")
Out[15]: 
date        76.293945
float       76.293945
int         76.293945
smallint    19.073486
string      76.293945
dtype: float64
```

Total memory usage of the DataFrame has been implemented as part of `df.info()` where the units representation can be specified:

``` python
In [16]: df.info()             
<class 'pandas.core.frame.DataFrame'>
Int64Index: 10000000 entries, 0 to 9999999
Data columns (total 5 columns):
date        datetime64[ns]
float       float64
int         int64
smallint    int16
string      object
dtypes: datetime64[ns](1), float64(1), int16(1), int64(1), object(1)
memory_usage: 340000000.00 B

In [18]: df.info(mem_unit='MB')
<class 'pandas.core.frame.DataFrame'>
Int64Index: 10000000 entries, 0 to 9999999
Data columns (total 5 columns):
date        datetime64[ns]
float       float64
int         int64
smallint    int16
string      object
dtypes: datetime64[ns](1), float64(1), int16(1), int64(1), object(1)
memory_usage: 324.25 MB
```

Any more thoughts?

I think that would be useful!

would need a couple of display options I think:
- default unit (MB?)
- whether to display this by default (in say df.info()), default False for now

Let's use MB as the default unit and add a `memory_usage` argument to the DataFrame `info()` method to specify unit type (where default is None).

Total memory usage of the DataFrame (by default in MB) can always be determined by using:

``` python
df.meminfo().sum()
```

Seems like it would be easy to infer the unit by taking the log10 of the number of bytes, dividing by 10 to that number and using the bisect module to lookup the corresponding name of the unit in an array. 

I have units mapped in a dictionary with unit options consisting of (B, KB, MB, GB, TB, PB)

``` python
        byte_conv = {
                    "B": 2**0,
                    "KB": 2**10,
                    "MB": 2**20,
                    "GB": 2**30,
                    "TB": 2**40,
                    "PB": 2**50}
```

Yet, this requires an explicit indication of desired units (as implemented [here](https://github.com/asobrien/pandas/blob/df-mem-info/pandas/core/frame.py#L1489). Are you suggesting to use a method to automatically infer the appropriate unit based on DataFrame size?

Yep. Just an intuition, haven't thought it out completely. 

Inferring units would be nice for `df.info()` but not desirable for the machine readable `df.meminfo()`.

@shoyer `df.meminfo()` would have no arguments, so that the series would always return memory usage in columns in terms of bytes. In this manner, usage would be consistent.

@cpcloud a [small function](http://stackoverflow.com/a/1094933) could be used to infer the memory usage in `df.info()` with the appropriate order of magnitude if a`memory_usage` argument is `True`.

``` python
def sizeof_fmt(num):
    for x in ['bytes', 'KB', 'MB', 'GB', 'TB']:
        if num < 1024.0:
            return "%3.1f %s" % (num, x)
        num /= 1024.0
    return "%3.1f %s" % (num, 'PB')
```


Resolving patch:
diff --git a/pandas/core/config_init.py b/pandas/core/config_init.py
--- a/pandas/core/config_init.py
+++ b/pandas/core/config_init.py
@@ -203,6 +203,12 @@
     Setting this to None/False restores the values to their initial value.
 """
 
+pc_memory_usage_doc = """
+: bool or None
+    This specifies if the memory usage of a DataFrame should be displayed when
+    df.info() is called.
+"""
+
 style_backup = dict()
 
 
@@ -274,6 +280,8 @@ def mpl_style_cb(key):
     # redirected to width, make defval identical
     cf.register_option('line_width', get_default_val('display.width'),
                        pc_line_width_doc)
+    cf.register_option('memory_usage', True, pc_memory_usage_doc,
+                        validator=is_instance_factory([type(None), bool]))
 
 cf.deprecate_option('display.line_width',
                     msg=pc_line_width_deprecation_warning,
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -1390,7 +1390,7 @@ def to_latex(self, buf=None, columns=None, col_space=None, colSpace=None,
         if buf is None:
             return formatter.buf.getvalue()
 
-    def info(self, verbose=None, buf=None, max_cols=None):
+    def info(self, verbose=None, buf=None, max_cols=None, memory_usage=None):
         """
         Concise summary of a DataFrame.
 
@@ -1404,6 +1404,12 @@ def info(self, verbose=None, buf=None, max_cols=None):
         max_cols : int, default None
             Determines whether full summary or short summary is printed.
             None follows the `display.max_info_columns` setting.
+        memory_usage : boolean, default None
+            Specifies whether total memory usage of the DataFrame
+            elements (including index) should be displayed. None follows
+            the `display.memory_usage` setting. True or False overrides
+            the `display.memory_usage` setting. Memory usage is shown in
+            human-readable units (base-2 representation).
         """
         from pandas.core.format import _put_lines
 
@@ -1462,6 +1468,14 @@ def _verbose_repr():
         def _non_verbose_repr():
             lines.append(self.columns.summary(name='Columns'))
 
+        def _sizeof_fmt(num):
+            # returns size in human readable format
+            for x in ['bytes', 'KB', 'MB', 'GB', 'TB']:
+                if num < 1024.0:
+                    return "%3.1f %s" % (num, x)
+                num /= 1024.0
+            return "%3.1f %s" % (num, 'PB')
+
         if verbose:
             _verbose_repr()
         elif verbose is False:  # specifically set to False, not nesc None
@@ -1474,9 +1488,46 @@ def _non_verbose_repr():
 
         counts = self.get_dtype_counts()
         dtypes = ['%s(%d)' % k for k in sorted(compat.iteritems(counts))]
-        lines.append('dtypes: %s\n' % ', '.join(dtypes))
+        lines.append('dtypes: %s' % ', '.join(dtypes))
+        if memory_usage is None:
+            memory_usage = get_option('display.memory_usage')
+        if memory_usage:  # append memory usage of df to display
+            lines.append("memory usage: %s\n" %
+                            _sizeof_fmt(self.memory_usage(index=True).sum()))
         _put_lines(buf, lines)
 
+    def memory_usage(self, index=False):
+        """Memory usage of DataFrame columns.
+
+        Parameters
+        ----------
+        index : bool
+            Specifies whether to include memory usage of DataFrame's
+            index in returned Series. If `index=True` (default is False)
+            the first index of the Series is `Index`.
+
+        Returns
+        -------
+        sizes : Series
+            A series with column names as index and memory usage of
+            columns with units of bytes.
+
+        Notes
+        -----
+        Memory usage does not include memory consumed by elements that
+        are not components of the array.
+
+        See Also
+        --------
+        numpy.ndarray.nbytes
+        """
+        result = Series([ c.values.nbytes for col, c in self.iteritems() ],
+                        index=self.columns)
+        if index:
+             result = Series(self.index.values.nbytes,
+                        index=['Index']).append(result)
+        return result
+
     def transpose(self):
         """Transpose index and columns"""
         return super(DataFrame, self).transpose(1, 0)


Test patch:
diff --git a/pandas/tests/test_format.py b/pandas/tests/test_format.py
--- a/pandas/tests/test_format.py
+++ b/pandas/tests/test_format.py
@@ -43,7 +43,7 @@ def has_info_repr(df):
 def has_non_verbose_info_repr(df):
     has_info = has_info_repr(df)
     r = repr(df)
-    nv = len(r.split('\n')) == 5  # 1. <class>, 2. Index, 3. Columns, 4. dtype, 5. trailing newline
+    nv = len(r.split('\n')) == 6  # 1. <class>, 2. Index, 3. Columns, 4. dtype, 5. memory usage, 6. trailing newline
     return has_info and nv
 
 def has_horizontally_truncated_repr(df):
diff --git a/pandas/tests/test_frame.py b/pandas/tests/test_frame.py
--- a/pandas/tests/test_frame.py
+++ b/pandas/tests/test_frame.py
@@ -6553,7 +6553,7 @@ def test_info_max_cols(self):
                 buf = StringIO()
                 df.info(buf=buf, verbose=verbose)
                 res = buf.getvalue()
-                self.assertEqual(len(res.split('\n')), len_)
+                self.assertEqual(len(res.strip().split('\n')), len_)
 
         for len_, verbose in [(10, None), (5, False), (10, True)]:
 
@@ -6562,7 +6562,7 @@ def test_info_max_cols(self):
                 buf = StringIO()
                 df.info(buf=buf, verbose=verbose)
                 res = buf.getvalue()
-                self.assertEqual(len(res.split('\n')), len_)
+                self.assertEqual(len(res.strip().split('\n')), len_)
 
         for len_, max_cols in [(10, 5), (5, 4)]:
             # setting truncates
@@ -6570,15 +6570,49 @@ def test_info_max_cols(self):
                 buf = StringIO()
                 df.info(buf=buf, max_cols=max_cols)
                 res = buf.getvalue()
-                self.assertEqual(len(res.split('\n')), len_)
+                self.assertEqual(len(res.strip().split('\n')), len_)
 
             # setting wouldn't truncate
             with option_context('max_info_columns', 5):
                 buf = StringIO()
                 df.info(buf=buf, max_cols=max_cols)
                 res = buf.getvalue()
-                self.assertEqual(len(res.split('\n')), len_)
+                self.assertEqual(len(res.strip().split('\n')), len_)
 
+    def test_info_memory_usage(self):
+        # Ensure memory usage is displayed, when asserted, on the last line
+        dtypes = ['int64', 'float64', 'datetime64[ns]', 'timedelta64[ns]',
+                  'complex128', 'object', 'bool']
+        data = {}
+        n = 10
+        for i, dtype in enumerate(dtypes):
+            data[i] = np.random.randint(2, size=n).astype(dtype)
+        df = DataFrame(data)
+        buf = StringIO()
+        # display memory usage case
+        df.info(buf=buf, memory_usage=True)
+        res = buf.getvalue().splitlines()
+        self.assertTrue("memory usage: " in res[-1])
+        # do not display memory usage cas
+        df.info(buf=buf, memory_usage=False)
+        res = buf.getvalue().splitlines()
+        self.assertTrue("memory usage: " not in res[-1])
+
+        # Test a DataFrame with duplicate columns
+        dtypes = ['int64', 'int64', 'int64', 'float64']
+        data = {}
+        n = 100
+        for i, dtype in enumerate(dtypes):
+            data[i] = np.random.randint(2, size=n).astype(dtype)
+        df = DataFrame(data)
+        df.columns = dtypes
+        # Ensure df size is as expected
+        df_size = df.memory_usage().sum()
+        exp_size = len(dtypes) * n * 8  # cols * rows * bytes
+        self.assertEqual(df_size, exp_size)
+        # Ensure number of cols in memory_usage is the same as df
+        size_df = np.size(df.columns.values)  # index=False; default
+        self.assertEqual(size_df, np.size(df.memory_usage()))
 
     def test_dtypes(self):
         self.mixed_frame['bool'] = self.mixed_frame['A'] > 0