Repository: dask/dask

Issue:
how to get info (total memory usage) of a dask.dataframe?
I'm hoping to get an idea of the memory size of a dask.dataframe once I call .compute() on it

My current approach is

```
import dask.dataframe as dd
from dask.utils import format_bytes

ddf = dd.demo.make_timeseries(
    start="2000-01-01",
    end="2000-01-02",
    dtypes={"x": float, "y": float, "id": int},
    freq="10ms",
    partition_freq="24h",
)

format_bytes(ddf.memory_usage(deep=True).sum().compute())
```

<img width="481" alt="Screen Shot 2021-09-02 at 4 53 39 PM" src="https://user-images.githubusercontent.com/17162724/131914676-3f388dc0-6ab2-4509-b488-cd312d604cc0.png">


pandas has `.info` which gives the this as well as other info (and returns a `NoneType`)

I see dask.dataframe has an info() method but not sure if it works as expected:

```
ddf.info()
```

<img width="341" alt="Screen Shot 2021-09-02 at 4 57 01 PM" src="https://user-images.githubusercontent.com/17162724/131915093-ab1eedee-5552-486b-a983-6e909635d38f.png">

```
ddf.info.compute()
AttributeError: 'NoneType' object has no attribute 'compute'
```


Discussion:
Thanks for reporting @raybellwaves, I'm able to reproduce. It looks like if I pass `verbose=True`, then the Dask result is what I would expect based on the pandas `.info()` method:

```python
In [15]: df.info(verbose=True)
<class 'dask.dataframe.core.DataFrame'>
DatetimeIndex: 2592000 entries, 2000-01-01 00:00:00 to 2000-01-30 23:59:59
Data columns (total 4 columns):
 #   Column  Non-Null Count  Dtype
---  ------  --------------  -----
 0   id      2592000 non-null      int64
 1   name    2592000 non-null      object
 2   x       2592000 non-null      float64
 3   y       2592000 non-null      float64
dtypes: object(1), float64(2), int64(1)
```
So it seems like the task here is:
1. add the total memory usage to the output of `info(verbose=True)` 
2. change the default from `verbose=False` to `verbose=True`

1 seems like a good and straightforward addition to me, but 2 is a little more controversial since flipping the kwarg would trigger computation everytime `info` is called, which is obviously something that dask tries to avoid. 
Hi,
I'm a little new to Open Source. Could I take this up?
Sure! The work is to add the total_memory usage to the output of `info` when `verbose=True`. Just let me know if you run into any issues :)
Hi Julia! (@jsignell)

It was a pretty small change as most of it was already present. I was able to get it working but had a couple of small questions before I opened up a PR.

The enhancement to ```ddf.info(verbose=True)```
![image](https://user-images.githubusercontent.com/20301919/134048854-ad596ab4-3a84-415e-8d86-8ee286643169.png)

1. Considering the problem faced by @raybellwaves, wouldn't another already implement approach be using ```ddf.info(memory_usage=True)```? 
![image](https://user-images.githubusercontent.com/20301919/134049197-94172be9-edfe-473b-89ba-7e4d0e5b54a7.png)

2. Would you also suggest that I change the default from ```verbose=False``` to ```verbose=True```, as stated in the older content?

Thank you!
Yeah you are right! I should have gone to look at the method to check on what kwargs were already there. Given that there is already a `memory_usage=True` kwarg, I think we can either just close this issue, or we could make `verbose=True` also set `memory_usage=True`. 

For 2. I do not think we should flip the default from `verbose=False` to `verbose=True`. My thinking is that anytime dask triggers computation without the user explicitly calling `compute()` it is potentially surprising.

Resolving patch:
diff --git a/dask/dataframe/core.py b/dask/dataframe/core.py
--- a/dask/dataframe/core.py
+++ b/dask/dataframe/core.py
@@ -5031,6 +5031,7 @@ def info(self, buf=None, verbose=False, memory_usage=False):
         # Group and execute the required computations
         computations = {}
         if verbose:
+            memory_usage = True
             computations.update({"index": self.index, "count": self.count()})
         if memory_usage:
             computations.update(


Test patch:
diff --git a/dask/dataframe/tests/test_dataframe.py b/dask/dataframe/tests/test_dataframe.py
--- a/dask/dataframe/tests/test_dataframe.py
+++ b/dask/dataframe/tests/test_dataframe.py
@@ -16,7 +16,7 @@
 from dask.base import compute_as_if_collection
 from dask.blockwise import fuse_roots
 from dask.dataframe import _compat, methods
-from dask.dataframe._compat import PANDAS_GT_110, tm
+from dask.dataframe._compat import PANDAS_GT_110, PANDAS_GT_120, tm
 from dask.dataframe.core import (
     Scalar,
     _concat,
@@ -3484,7 +3484,7 @@ def test_groupby_multilevel_info():
 
     g = ddf.groupby(["A", "B"]).sum()
     # slight difference between memory repr (single additional space)
-    _assert_info(g.compute(), g, memory_usage=False)
+    _assert_info(g.compute(), g, memory_usage=True)
 
     buf = StringIO()
     g.info(buf, verbose=False)
@@ -3496,7 +3496,7 @@ def test_groupby_multilevel_info():
 
     # multilevel
     g = ddf.groupby(["A", "B"]).agg(["count", "sum"])
-    _assert_info(g.compute(), g, memory_usage=False)
+    _assert_info(g.compute(), g, memory_usage=True)
 
     buf = StringIO()
     g.info(buf, verbose=False)
@@ -3508,6 +3508,7 @@ def test_groupby_multilevel_info():
     assert buf.getvalue() == expected
 
 
+@pytest.mark.skipif(not PANDAS_GT_120, reason="need newer version of Pandas")
 def test_categorize_info():
     # assert that we can call info after categorize
     # workaround for: https://github.com/pydata/pandas/issues/14368
@@ -3533,7 +3534,8 @@ def test_categorize_info():
         " 0   x       4 non-null      int64\n"
         " 1   y       4 non-null      category\n"
         " 2   z       4 non-null      object\n"
-        "dtypes: category(1), object(1), int64(1)"
+        "dtypes: category(1), object(1), int64(1)\n"
+        "memory usage: 496.0 bytes\n"
     )
     assert buf.getvalue() == expected