DOC: update docs on direct plotting with matplotlib (GH8614)
State: closed

Closes #8614 
- this adds a warning that you now have to use `to_pydatetime` as direct plotting with a DatetimeIndex does not work anymore (see #8614)
- I also removed for now the note on speed and explanation of the registered formatters. @agijsberts could you shed some light on this?
  - "_The speed up for large data sets only applies to pandas 0.14.0 and later._" Why only for pandas 0.14 or later? And from where does this speed-up come from?
  - "_thereby extending date and time support to practically all plot types available in matplotlib_" -> but if you plot directly with matplotlib, I think you don't use the pandas registered formatters? So that sentence seems not fully correct, is that possible? And isn't that the reason for the possible speed-up (matplotlib defaults formatter being faster as pandas' formatter)?


Labels:
name: Docs
name: Visualization
description: plotting

Comments:
@jorisvandenbossche I wrote that documentation to reflect the changes in PR #6650 that I prepared for issue #6636. In short, pandas _does_ register its own formatters (see https://github.com/pydata/pandas/blob/master/pandas/tseries/converter.py#L27), so this behavior is entirely part of pandas. The PR obtained a speed-up by replacing a call to matplotlib's `date2num` (explicit for-loop) with `epoch2num` (vectorized), hence the statement that it's much faster since pandas 0.14.0.

It seems that the formatters registered by pandas are still correctly called:

```
In [1]: from pandas import DataFrame, date_range

In [2]: import matplotlib.units as units

In [3]: df = DataFrame(range(100), index = date_range('20130101', periods=100))

In [4]: units.registry.get_converter(df.index)
Out[4]: <pandas.tseries.converter.DatetimeConverter instance at 0x3f33f38>
```

So far I do not see any obvious cause for the new problems, but I'll dive deeper into this later on.

Ah, yes, I was confusing the 'converter' (which are registered in the matplotlib `units`) and the formatting of the labels (which is also nicer in pandas, but this is something you only have when plotting with pandas' `plot` and not when plotting directly with matplotlibs `plot`, while the unit converter works for both)

But, two points:
- your statement about the improved performance also holds true when plotting with pandas, so it is not specific to plotting directly with matplotlib, so it is seems a bit out of place in this section?
- The `units.registry.get_converter(df.index)` does indeed still work. But, the problem is, that when plotting with `plt.plot(df.index, df['col'])`, the `df.index` is first converted to a array of `datetime64`, and this is not recognized by matplotlib anymore: 
  
  ```
  In [35]: np.asarray(df.index)
  Out[35]:
  array(['2013-01-01T01:00:00.000000000+0100',
         ...
         '2013-04-10T02:00:00.000000000+0200'], dtype='datetime64[ns]')
  
  In [36]: units.registry.get_converter(np.asarray(df.index))
  
  -> None
  ```
  
  To make it fully complex, the `plt.fill_between` in the example does not do this (and does still convert it to datetimes), and because of that the example now crashed.

@jorisvandenbossche Re. your points:
- df.plot() did not benefit from the PR as it uses (at least back then) a different converter and formatter. See https://github.com/pydata/pandas/issues/6636#issuecomment-37870577 for details.
- You're right, `plt.plot` ends up looking for a `datetime64` formatter, which does not exist. The easiest workaround to make things work as they were is to register pandas' `DateTimeFormatter` for `datetime64`:

```
from numpy import datetime64
import pandas as pd
import matplotlib.units as units
import matplotlib.pyplot as plt
df = pd.DataFrame(range(100), index = pd.date_range('20130101', periods=100))
units.registry[datetime64] = pd.tseries.converter.DatetimeConverter()
plt.plot(df.index, df)
plt.show()
```
- The `DateTimeConverter` in matplotlib is limited to `datetime.datetime` and `datetime.date`. Of course you could use it if you first convert the index with `to_pydatetime()`, but it is wasteful to convert `datetime64` to `datetime` and then again to `float`. An example with a `DateTimeIndex` of length 100000:

```
In [22]: %timeit DatetimeConverter.convert(df.index, None, None)
10000 loops, best of 3: 74.4 us per loop

In [23]: %timeit DateConverter.convert(df.index.to_pydatetime(), None, None)
100 loops, best of 3: 9.35 ms per loop
```
- When not importing pandas, then I believe there two options to plot `datetime64` as a time-axis:
  1. convert to `datetime.datetime` and use matplotlib's `DateConverter`  (slow, see above)
  2. manually convert `datetime64` to matplotlib's time representation with `epoch2num(dt.asi8 / 1.0E9)`. Of course then you are still responsible for installing the date/time formatters. This conversion is however very fast and exactly what the PR implemented.

So our `DatetimeConverter` already works for `datetime64`? So we should just register it, and the initial problem is solved! (#8614)

Simply doing `units.registry[np.datetime64] = DatetimeConverter()` solves the issue

@agijsberts Thanks a lot for shedding your light on this!

@jorisvandenbossche I'm glad to help. And yes, `DatetimeConverter` should work for `datetime64`; it actually exploits the fact that `DatetimeIndex` is stored as `datetime64[ns]`.

OK, I will then close this PR as it is totally the wrong way :-) 
and open an new one to register the converter for `datetime64`. Or would there be people who rely on the fact that `datetime64` arrays are regarded as ints in matplotlib? Let's discuss further in #8655.