Repository: sktime/sktime

Issue:
pandas-lag-plotting
Added pandas style lag plotting for a single lag as well as a list of lags

<!--
Thanks for contributing a pull request! Please ensure you have taken a look at our contribution guide: https://github.com/alan-turing-institute/sktime/blob/master/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.

Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Pandas-style lag plotting for sktime #588
https://github.com/alan-turing-institute/sktime/issues/588 

#### What does this implement/fix? Explain your changes.
<!--
A clear and concise description of what you have implemented.
-->

A utility for plotting lagged versions of a time series.
It allows plotting several such lagged versions together using 
the pandas lag_plot api which natively allows only one plot at
a particular lag.  


#### Does your contribution introduce a new dependency? If yes, which one?

<!--
If your contribution does add a new hard dependency, we may suggest to initially develop your contribution in a separate companion package in https://github.com/sktime/ to keep external dependencies of the core sktime package to a minimum.
-->

None, that I can think of.

#### What should a reviewer concentrate their feedback on?

<!-- This section is particularly useful if you have a pull request that is still in development. You can guide the reviews to focus on the parts that are ready for their comments. We suggest using bullets (indicated by * or -) and filled checkboxes [x] here -->

#### Any other comments?
<!--
Please be aware that we are a loose team of volunteers so patience is necessary; assistance handling other issues is very welcome. We value all user contributions, no matter how minor they are. If we are slow to review, either the pull request needs some benchmarking, tinkering, convincing, etc. or more likely the reviewers are simply busy. In either case, we ask for your understanding during the review process.
-->

Here is a demo of this utility in a collaboratory notebook: https://colab.research.google.com/drive/1tbHVqTZwGkeRjunvqGV-6Ql7NLjVJJvs?usp=sharing 

#### PR checklist
<!--
Please go through the checklist below. Please feel free to remove points if they are not applicable.
-->

##### For all contributions
- [x] I've added myself to the [list of contributors](https://github.com/alan-turing-institute/sktime/blob/master/.all-contributorsrc).
- [x] Optionally, I've updated sktime's [CODEOWNERS](https://github.com/alan-turing-institute/sktime/blob/master/CODEOWNERS) to receive notifications about future changes to these files.
- [x] I've added unit tests and made sure they pass locally.

##### For new estimators
- [x] I've added the estimator to the online documentation.
- [x] I've updated the existing example notebooks or provided a new one to showcase how my estimator works.


<!--
Thanks for contributing!
-->



Discussion:
As for testing the utility, I am not exactly sure whether unit tests are needed as its a wrapper around pandas with some additions to the display. 
As a test, you can look at this notebook here: https://colab.research.google.com/drive/1tbHVqTZwGkeRjunvqGV-6Ql7NLjVJJvs?usp=sharing 
Hi @hassan-jvd90, I suggest we create a new notebooks for time series EDA in `example/exploratory_data_analysis.ipynb` and include a simple example with your functions, that would be run automatically on CI so we have some sort of unit test for it. 

If you're interested in adding more functionality, here's another notebook with some functionality we could bring into sktime: https://github.com/mloning/intro-to-ml-with-time-series-DSSGx-2020/blob/master/notebooks/02_exploratory_data_analysis.ipynb
@hassan-jvd90 let me know if you need help with anything, would love to merge this PR!
> 
> 
> @hassan-jvd90 let me know if you need help with anything, would love to merge this PR!

@mloning Hi! sorry for this late response. I got busy with some work-related stuff. Thanks for your encouraging follow-up. I'd love to contribute further to this project. 

As suggested by you, I have added a notebook titled exploratory_data_analysis with an example of this lag plotting utility with some explanation as well. 

I have seen the exploratory data analysis notebook shared by you as well, and I think based on that we can pull in the stuff from there as well.

Let me know what you think. Thanks!


Check out this pull request on&nbsp; <a href="https://app.reviewnb.com/alan-turing-institute/sktime/pull/681"><img align="absmiddle"  alt="ReviewNB" height="28" class="BotMessageButtonImage" src="https://raw.githubusercontent.com/ReviewNB/support/master/images/button_reviewnb.png"/></a> 

 See visual diffs & provide feedback on Jupyter Notebooks. 

---

 <i>Powered by <a href='https://www.reviewnb.com/?utm_source=gh'>ReviewNB</a></i>
@hassan-jvd90 wanted to check-in with you on this PR. Let me know if there is anything I can do to help -- I think you are close and we can get this ready to go relatively quickly.
@hassan-jvd90 do you think you'll have time to wrap up this addition? If not, I can go ahead and finish up the suggested edits so we can merge this. 

Resolving patch:
diff --git a/sktime/utils/plotting.py b/sktime/utils/plotting.py
--- a/sktime/utils/plotting.py
+++ b/sktime/utils/plotting.py
@@ -1,29 +1,15 @@
 #!/usr/bin/env python3 -u
 # -*- coding: utf-8 -*-
 # copyright: sktime developers, BSD-3-Clause License (see LICENSE file)
-"""Common timeseries plotting functionality.
+"""Common timeseries plotting functionality."""
 
-Functions
----------
-plot_series(*series, labels=None, markers=None, ax=None)
-plot_correlations(
-    series,
-    lags=24,
-    alpha=0.05,
-    zero_lag=True,
-    acf_fft=False,
-    acf_adjusted=True,
-    pacf_method="ywadjusted",
-    suptitle=None,
-    series_title=None,
-    acf_title="Autocorrelation",
-    pacf_title="Partial Autocorrelation",
-)
-"""
 __all__ = ["plot_series", "plot_correlations"]
-__author__ = ["Markus Löning", "Ryan Kuhns", "Drishti Bhasin"]
+__author__ = ["mloning", "RNKuhns", "Drishti Bhasin"]
+
+import math
 
 import numpy as np
+import pandas as pd
 
 from sktime.utils.validation._dependencies import _check_soft_dependencies
 from sktime.utils.validation.forecasting import check_y
@@ -148,6 +134,79 @@ def format_fn(tick_val, tick_pos):
         return ax
 
 
+def plot_lags(series, lags=1, suptitle=None):
+    """Plot one or more lagged versions of a time series.
+
+    Parameters
+    ----------
+    series : pd.Series
+        Time series for plotting lags.
+    lags : int or array-like, default=1
+        The lag or lags to plot.
+
+        - int plots the specified lag
+        - array-like  plots specified lags in the array/list
+
+    suptitle : str, default=None
+        The text to use as the Figure's suptitle. If None, then the title
+        will be "Plot of series against lags {lags}"
+
+    Returns
+    -------
+    fig : matplotlib.figure.Figure
+
+    axes : np.ndarray
+        Array of the figure's Axe objects
+
+    Examples
+    --------
+    >>> from sktime.datasets import load_airline
+    >>> y = load_airline()
+    >>> fig, ax = plot_lags(y, lags=2) # plot of y(t) with y(t-2)
+    >>> fig, ax = plot_lags(y, lags=[1,2,3]) # plots of y(t) with y(t-1),y(t-2)..
+    """
+    _check_soft_dependencies("matplotlib")
+    import matplotlib.pyplot as plt
+
+    check_y(series)
+
+    if isinstance(lags, int):
+        single_lag = True
+        lags = [lags]
+    elif isinstance(lags, (tuple, list, np.ndarray)):
+        single_lag = False
+    else:
+        raise ValueError("`lags should be an integer, tuple, list, or np.ndarray.")
+
+    length = len(lags)
+    n_cols = min(3, length)
+    n_rows = math.ceil(length / n_cols)
+    fig, ax = plt.subplots(
+        nrows=n_rows,
+        ncols=n_cols,
+        figsize=(8, 6 * n_rows),
+        sharex=True,
+        sharey=True,
+    )
+    if single_lag:
+        axes = ax
+        pd.plotting.lag_plot(series, lag=lags[0], ax=axes)
+    else:
+        axes = ax.ravel()
+        for i, val in enumerate(lags):
+            pd.plotting.lag_plot(series, lag=val, ax=axes[i])
+
+    if suptitle is None:
+        fig.suptitle(
+            f"Plot of series against lags {', '.join([str(lag) for lag in lags])}",
+            size="xx-large",
+        )
+    else:
+        fig.suptitle(suptitle, size="xx-large")
+
+    return fig, np.array(fig.get_axes())
+
+
 def plot_correlations(
     series,
     lags=24,


Test patch:
diff --git a/sktime/utils/tests/test_plotting.py b/sktime/utils/tests/test_plotting.py
--- a/sktime/utils/tests/test_plotting.py
+++ b/sktime/utils/tests/test_plotting.py
@@ -1,12 +1,15 @@
 # -*- coding: utf-8 -*-
-"""Tests for series plotting functions."""
+# copyright: sktime developers, BSD-3-Clause License (see LICENSE file)
+"""Test functionality of time series plotting functions."""
+
+import re
 
 import numpy as np
 import pandas as pd
 import pytest
 
 from sktime.datasets import load_airline
-from sktime.utils.plotting import plot_series, plot_correlations
+from sktime.utils.plotting import plot_series, plot_correlations, plot_lags
 from sktime.utils.validation._dependencies import _check_soft_dependencies
 from sktime.utils.validation.series import VALID_DATA_TYPES
 
@@ -17,10 +20,14 @@
 series_to_test = [y_airline, (y_airline_true, y_airline_test)]
 invalid_input_types = [
     y_airline.values,
-    pd.DataFrame({"a": y_airline, "b": y_airline}),
+    pd.DataFrame({"y1": y_airline, "y2": y_airline}),
     "this_is_a_string",
 ]
 
+# can be used with pytest.mark.parametrize to check plots that accept
+# univariate series
+univariate_plots = [plot_correlations, plot_lags]
+
 
 # Need to use _plot_series to make it easy for test cases to pass either a
 # single series or a tuple of multiple series to be unpacked as argss
@@ -31,14 +38,14 @@ def _plot_series(series, ax=None, **kwargs):
         return plot_series(series, ax=ax, **kwargs)
 
 
+# Can be used with pytest.mark.parametrize to run a test on all plots
+all_plots = univariate_plots + [_plot_series]
+
+
 @pytest.fixture
 def valid_data_types():
-    """Return valid data types."""
-    valid_data_types = tuple(
-        filter(
-            lambda x: x is not np.ndarray and x is not pd.DataFrame, VALID_DATA_TYPES
-        )
-    )
+    """Filter valid data types for those that work with plotting functions."""
+    valid_data_types = tuple(filter(lambda x: x is not np.ndarray, VALID_DATA_TYPES))
     return valid_data_types
 
 
@@ -51,15 +58,31 @@ def test_plot_series_runs_without_error(series_to_plot):
     _plot_series(series_to_plot)
     plt.gcf().canvas.draw_idle()
 
+    # Test with labels specified
+    if isinstance(series_to_plot, pd.Series):
+        labels = ["Series 1"]
+    elif isinstance(series_to_plot, tuple):
+        labels = [f"Series {i+1}" for i in range(len(series_to_plot))]
+    _plot_series(series_to_plot, labels=labels)
+    plt.gcf().canvas.draw_idle()
+    plt.close()
+
 
 @pytest.mark.parametrize("series_to_plot", invalid_input_types)
 def test_plot_series_invalid_input_type_raises_error(series_to_plot, valid_data_types):
     """Tests whether plot_series raises error for invalid input types."""
-    # TODO: Is it possible to dynamically create the matching str if it includes
-    #       characters that need to be escaped (like .)
-    # match = f"Data must be a one of {valid_data_types}, but found type: {type(Z)}"
-    with pytest.raises((TypeError, ValueError)):
-        _plot_series(series_to_plot)
+    series_type = type(series_to_plot)
+
+    if not isinstance(series_to_plot, (pd.Series, pd.DataFrame)):
+        match = (
+            rf"input must be a one of {valid_data_types}, but found type: {series_type}"
+        )
+        with pytest.raises((TypeError), match=re.escape(match)):
+            _plot_series(series_to_plot)
+    else:
+        match = "input must be univariate, but found 2 variables."
+        with pytest.raises(ValueError, match=match):
+            _plot_series(series_to_plot)
 
 
 @pytest.mark.parametrize(
@@ -141,35 +164,65 @@ def test_plot_series_output_type(series_to_plot):
     )
 
 
+def test_plot_series_uniform_treatment_of_int64_range_index_types():
+    """Verify that plot_series treats Int64 and Range indices equally."""
+    # We test that int64 and range indices are treated uniformly and do not raise an
+    # error of inconsistent index types
+    _check_soft_dependencies("matplotlib")
+    import matplotlib.pyplot as plt
+
+    y1 = pd.Series(np.arange(10))
+    y2 = pd.Series(np.random.normal(size=10))
+    y1.index = pd.Int64Index(y1.index)
+    y2.index = pd.RangeIndex(y2.index)
+    plot_series(y1, y2)
+    plt.gcf().canvas.draw_idle()
+    plt.close()
+
+
+# Generically test whether plots only accepting univariate input run
 @pytest.mark.parametrize("series_to_plot", [y_airline])
-def test_plot_correlations_runs_without_error(series_to_plot):
-    """Tests whether plot_correlations runs without error."""
+@pytest.mark.parametrize("plot_func", univariate_plots)
+def test_univariate_plots_run_without_error(series_to_plot, plot_func):
+    """Tests whether plots that accept univariate series run without error."""
     _check_soft_dependencies("matplotlib")
     import matplotlib.pyplot as plt
 
-    plot_correlations(series_to_plot)
+    plot_func(series_to_plot)
     plt.gcf().canvas.draw_idle()
+    plt.close()
 
 
+# Generically test whether plots only accepting univariate input
+# raise an error when invalid input type is found
 @pytest.mark.parametrize("series_to_plot", invalid_input_types)
-def test_plot_correlations_invalid_input_type_raises_error(
-    series_to_plot, valid_data_types
+@pytest.mark.parametrize("plot_func", univariate_plots)
+def test_univariate_plots_invalid_input_type_raises_error(
+    series_to_plot, plot_func, valid_data_types
 ):
-    """Tests whether plot_correlations raises error for invalid input types."""
-    # TODO: Is it possible to dynamically create the matching str if it includes
-    #       characters that need to be escaped (like .)
-    # match = f"Data must be a one of {valid_data_types}, but found type: {type(Z)}"
-    with pytest.raises((TypeError, ValueError)):
-        plot_correlations(series_to_plot)
+    """Tests whether plots that accept univariate series run without error."""
+    if not isinstance(series_to_plot, (pd.Series, pd.DataFrame)):
+        series_type = type(series_to_plot)
+        match = (
+            rf"input must be a one of {valid_data_types}, but found type: {series_type}"
+        )
+        with pytest.raises(TypeError, match=re.escape(match)):
+            plot_func(series_to_plot)
+    else:
+        match = "input must be univariate, but found 2 variables."
+        with pytest.raises(ValueError, match=match):
+            plot_func(series_to_plot)
 
 
+# Generically test output of plots only accepting univariate input
 @pytest.mark.parametrize("series_to_plot", [y_airline])
-def test_plot_correlations_output_type(series_to_plot):
-    """Tests whether plot_correlations returns plt.fig and plt.ax."""
+@pytest.mark.parametrize("plot_func", univariate_plots)
+def test_univariate_plots_output_type(series_to_plot, plot_func):
+    """Tests whether plots accepting univariate series have correct output types."""
     _check_soft_dependencies("matplotlib")
     import matplotlib.pyplot as plt
 
-    fig, ax = plot_correlations(series_to_plot)
+    fig, ax = plot_func(series_to_plot)
 
     is_fig_figure = isinstance(fig, plt.Figure)
     is_ax_array = isinstance(ax, np.ndarray)
@@ -177,20 +230,38 @@ def test_plot_correlations_output_type(series_to_plot):
 
     assert is_fig_figure and is_ax_array and is_ax_array_axis, "".join(
         [
-            "plot_correlations should return plt.Figure and array of plt.Axes,",
+            f"{plot_func.__name__} should return plt.Figure and array of plt.Axes,",
             f"but returned: {type(fig)} and {type(ax)}",
         ]
     )
 
 
-def test_plot_series_uniform_treatment_of_int64_range_index_types():
-    """Test that int64 and range indices are treated the same without error."""
+# For plots that only accept univariate input, from here onwards are
+# tests specific to a given plot. E.g. to test specific arguments or functionality.
+@pytest.mark.parametrize("series_to_plot", [y_airline])
+@pytest.mark.parametrize("lags", [2, (1, 2, 3)])
+@pytest.mark.parametrize("suptitle", ["Lag Plot", None])
+def test_plot_lags_arguments(series_to_plot, lags, suptitle):
+    """Tests whether plot_lags run with different input arguments."""
     _check_soft_dependencies("matplotlib")
     import matplotlib.pyplot as plt
 
-    y1 = pd.Series(np.arange(10))
-    y2 = pd.Series(np.random.normal(size=10))
-    y1.index = pd.Int64Index(y1.index)
-    y2.index = pd.RangeIndex(y2.index)
-    plot_series(y1, y2)
+    plot_lags(series_to_plot, lags=lags, suptitle=suptitle)
+    plt.gcf().canvas.draw_idle()
+    plt.close()
+
+
+@pytest.mark.parametrize("series_to_plot", [y_airline])
+@pytest.mark.parametrize("lags", [6, 12, 24, 36])
+@pytest.mark.parametrize("suptitle", ["Correlation Plot", None])
+@pytest.mark.parametrize("series_title", ["Time Series", None])
+def test_plot_correlations_arguments(series_to_plot, lags, suptitle, series_title):
+    """Tests whether plot_lags run with different input arguments."""
+    _check_soft_dependencies("matplotlib")
+    import matplotlib.pyplot as plt
+
+    plot_correlations(
+        series_to_plot, lags=lags, suptitle=suptitle, series_title=series_title
+    )
     plt.gcf().canvas.draw_idle()
+    plt.close()