BUG: Fix plots showing multiple sets of  axis labels if index is a timeseries.

Fixes #3964


I could be confused here, but should this be checking `if isinstance(ax.yaxis...)` (i.e. **y**axis, not x?