Repository: pymc-devs/pymc (Python)

Issue:
Remove Plots & Diagnostics from PyMC Code Base
Plots and diagnostics are still accessible via PyMC3 but are actually implemented by ArviZ now. Although this was a good call when ArviZ was still nascent, now this causes confusion for new comers and overhead for devs, who have to mirror the functions in ArviZ, which is more work when ArviZ adds or changes functions (e.g `pm.hpd` vs. `az.hdi`).

To make all this more robust and easier to maintain, _we should completely remove plots and diagnostics from the PyMC code base_ and rely on ArviZ, since the latter library is now much more known and flies on its own (yeah, I'm a part-time poet).

The coming 4.0 major release is an opportunity to make this breaking change -- plus, it'll make the code base cleaner and leaner 👌 Feel free to comment if you wanna take on this issue 🖖 


Discussion:
> yeah, I'm a part-time poet

Must come from all that listening to Baba Brinkman ;) 
+1  lol ^^^
____
Hey there @AlexAndorra, is this an Issue I can participate in? 

Also, when do you think the ETA/Due Date for the coming 4.0 major release would be?
> Hey there @AlexAndorra, is this an Issue I can participate in?

Of course @CloudChaoszero ! I just assigned it to you. Feel free to ask questions 😉 

> Also, when do you think the ETA/Due Date for the coming 4.0 major release would be?

We usually don't give hard deadlines but are all commited and working to make that happen as soon as possible 🙂 
Getting new contributions is definitely helpful in that regard, so thank you!
Great! I'll reach out if I have questions etc @AlexAndorra.
@AlexAndorra Hey there! 

I was looking looking at the codebase, particularly [Pymc's plots/__init__.py](https://github.com/pymc-devs/pymc3/blob/master/pymc3/plots/__init__.py), and I saw functions like `compareplot` and `traceplot` borrow functionality from [arviz](https://arviz-devs.github.io/arviz/api/generated/arviz.plot_compare.html), but do a couple of work arounds.

![Example](https://user-images.githubusercontent.com/19514362/103452313-0009c880-4c83-11eb-8779-2401965b67ee.png)




These workarounds tries to relabel `kwargs` parameters, and other related items like that.

I was wondering should the `kwargs` param renaming be copied to arviz repo, or should the expected breakchanges for v4.0 assume users rename the params themselves? 
Yes, you can just remove all this @CloudChaoszero, as the idea is to completely delegate to ArviZ.
As a sidenote, [we just switched](https://github.com/arviz-devs/arviz/pull/1473) `compact` to `True` by default in `az.plot_trace` anyway 😉 
> Yes, you can just remove all this @CloudChaoszero, as the idea is to completely delegate to ArviZ.
> As a sidenote, [we just switched](https://github.com/arviz-devs/arviz/pull/1473) `compact` to `True` by default in `az.plot_trace` anyway 😉

Sounds great, thanks!
@AlexAndorra  Quick Question: Should customer function `plot_posterior_predictive_glm `[Source](https://github.com/pymc-devs/pymc3/blob/master/pymc3/plots/posteriorplot.py) be removed as well, and users would try to adopt their function to arviz? I  only see arviz's `plot_posterior` [Source](https://arviz-devs.github.io/arviz/api/generated/arviz.plot_posterior.html) right now.
* Note:  I can try to make a migration PR for it into Arviz, if necessaary
Ugh, that's a good question -- we discussed it recently with @MarcoGorelli and decided to keep it around IIRC. But the 4.0 release is a good opportunity to remove it and completely switch to ArviZ. 

`plot_posterior_predictive_glm` can probably be easily replicated with `az.plot_ppc` 🤔 
Curious about what @OriolAbril think: is it worth it to migrate it to ArviZ, or are ArviZ's current methods enough?
I don't know how `plot_posterior_predictive_glm` works internally, but my guess is that is should be kept in PyMC3 until there are regression plots in ArviZ, that is when https://github.com/arviz-devs/arviz/issues/313 and https://github.com/arviz-devs/arviz/issues/512 are solved. Maybe in [this year's GSoc](https://github.com/arviz-devs/arviz/wiki/GSoC-2020-projects#increase-support-for-time-series-and-regressions)? 

How does that sound? It may not match the 3-4 timeline though
`plot_posterior_predictive_glm` is just a few lines of code - if you have e.g. a model of the form `y = Normal(a + bx, sigma)` then it takes samples from the trace and plots `a + bx` for the various samples. Granted my knowledge here is really limited, but given that in the above example it disregards the values of `sigma` and just plots the regression lines `a + bx`, is it fair to call it "posterior predictive"?
> should be kept in PyMC3 until there are regression plots in ArviZ

Agreed @OriolAbril, let's do that then -- if this function proves hard to maintain because of v4 changes though, we'll probably remove it before ArviZ has regression functionalities.

> is it fair to call it "posterior predictive"?

Good catch @MarcoGorelli ! I think these are indeed no posterior _predictive_ samples but rather predictive _pushforward_ samples. As the function is meant to disapear soon though (see my answer above), let's not change the name, as that would be a breaking change for just a few months
Sounds good, I'll update the PR accordingly!



Patch:
diff --git a/benchmarks/benchmarks/benchmarks.py b/benchmarks/benchmarks/benchmarks.py
index bbf8fbd5fc..489befbefc 100644
--- a/benchmarks/benchmarks/benchmarks.py
+++ b/benchmarks/benchmarks/benchmarks.py
@@ -11,10 +11,10 @@
 #   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 #   See the License for the specific language governing permissions and
 #   limitations under the License.
-
 import time
 import timeit
 
+import arviz as az
 import numpy as np
 import pandas as pd
 import theano
@@ -192,7 +192,7 @@ def track_glm_hierarchical_ess(self, init):
                 compute_convergence_checks=False,
             )
             tot = time.time() - t0
-        ess = float(pm.ess(trace, var_names=["mu_a"])["mu_a"].values)
+        ess = float(az.ess(trace, var_names=["mu_a"])["mu_a"].values)
         return ess / tot
 
     def track_marginal_mixture_model_ess(self, init):
@@ -214,7 +214,7 @@ def track_marginal_mixture_model_ess(self, init):
                 compute_convergence_checks=False,
             )
             tot = time.time() - t0
-        ess = pm.ess(trace, var_names=["mu"])["mu"].values.min()  # worst case
+        ess = az.ess(trace, var_names=["mu"])["mu"].values.min()  # worst case
         return ess / tot
 
 
@@ -245,7 +245,7 @@ def track_glm_hierarchical_ess(self, step):
                 compute_convergence_checks=False,
             )
             tot = time.time() - t0
-        ess = float(pm.ess(trace, var_names=["mu_a"])["mu_a"].values)
+        ess = float(az.ess(trace, var_names=["mu_a"])["mu_a"].values)
         return ess / tot
 
 
@@ -304,7 +304,7 @@ def freefall(y, t, p):
             t0 = time.time()
             trace = pm.sample(500, tune=1000, chains=2, cores=2, random_seed=0)
             tot = time.time() - t0
-        ess = pm.ess(trace)
+        ess = az.ess(trace)
         return np.mean([ess.sigma, ess.gamma]) / tot
 
 
diff --git a/docs/source/api.rst b/docs/source/api.rst
index 4c15139f91..50b20c8f05 100644
--- a/docs/source/api.rst
+++ b/docs/source/api.rst
@@ -28,7 +28,6 @@ API Reference
    api/shape_utils
    api/ode
 
-
 Indices and tables
 ===================
 
diff --git a/docs/source/api/plots.rst b/docs/source/api/plots.rst
index 70cf8719d2..0dec109b7f 100644
--- a/docs/source/api/plots.rst
+++ b/docs/source/api/plots.rst
@@ -8,16 +8,7 @@ Plots are delegated to the
 `ArviZ <https://arviz-devs.github.io/arviz/index.html>`_.
 library, a general purpose library for
 "exploratory analysis of Bayesian models."
-For plots, ``pymc3.<function>`` are now aliases
-for ArviZ functions. Thus, the links below will redirect you to
-ArviZ docs:
+Refer to its documentation to use the plotting functions directly.
 
-- :func:`pymc3.traceplot <arviz:arviz.plot_trace>`
-- :func:`pymc3.plot_posterior <arviz:arviz.plot_posterior>`
-- :func:`pymc3.forestplot <arviz:arviz.plot_forest>`
-- :func:`pymc3.compareplot <arviz:arviz.plot_compare>`
-- :func:`pymc3.autocorrplot <arviz:arviz.plot_autocorr>`
-- :func:`pymc3.energyplot <arviz:arviz.plot_energy>`
-- :func:`pymc3.kdeplot <arviz:arviz.plot_kde>`
-- :func:`pymc3.densityplot <arviz:arviz.plot_density>`
-- :func:`pymc3.pairplot <arviz:arviz.plot_pair>`
+.. automodule:: pymc3.plots.posteriorplot
+   :members:
diff --git a/docs/source/api/stats.rst b/docs/source/api/stats.rst
index 8d47e1adac..bf85bb596d 100644
--- a/docs/source/api/stats.rst
+++ b/docs/source/api/stats.rst
@@ -5,21 +5,4 @@ Statistics and diagnostics are delegated to the
 `ArviZ <https://arviz-devs.github.io/arviz/index.html>`_.
 library, a general purpose library for
 "exploratory analysis of Bayesian models."
-For statistics and diagnostics, ``pymc3.<function>`` are now aliases
-for ArviZ functions. Thus, the links below will redirect you to
-ArviZ docs:
-
-.. currentmodule:: pymc3.stats
-
-
-- :func:`pymc3.bfmi <arviz:arviz.bfmi>`
-- :func:`pymc3.compare <arviz:arviz.compare>`
-- :func:`pymc3.ess <arviz:arviz.ess>`
-- :data:`pymc3.geweke <arviz:arviz.geweke>`
-- :func:`pymc3.hpd <arviz:arviz.hpd>`
-- :func:`pymc3.loo <arviz:arviz.loo>`
-- :func:`pymc3.mcse <arviz:arviz.mcse>`
-- :func:`pymc3.r2_score <arviz:arviz.r2_score>`
-- :func:`pymc3.rhat <arviz:arviz.rhat>`
-- :func:`pymc3.summary <arviz:arviz.summary>`
-- :func:`pymc3.waic <arviz:arviz.waic>`
+Refer to its documentation to use the diagnostics functions directly.
diff --git a/pymc3/__init__.py b/pymc3/__init__.py
index 23751b518f..c42850ecf8 100644
--- a/pymc3/__init__.py
+++ b/pymc3/__init__.py
@@ -61,7 +61,6 @@ def __set_compiler_flags():
 from pymc3.plots import *
 from pymc3.sampling import *
 from pymc3.smc import *
-from pymc3.stats import *
 from pymc3.step_methods import *
 from pymc3.tests import test
 from pymc3.theanof import *
diff --git a/pymc3/plots/__init__.py b/pymc3/plots/__init__.py
index 15553e4570..5923114cee 100644
--- a/pymc3/plots/__init__.py
+++ b/pymc3/plots/__init__.py
@@ -14,9 +14,10 @@
 
 """PyMC3 Plotting.
 
-Plots are delegated to the ArviZ library, a general purpose library for
-"exploratory analysis of Bayesian models." See https://arviz-devs.github.io/arviz/
-for details on plots.
+Plots are delegated to the `ArviZ <https://arviz-devs.github.io/arviz/>`_ library, a general purpose library for
+exploratory analysis of Bayesian models. For more details, see https://arviz-devs.github.io/arviz/.
+
+Only `plot_posterior_predictive_glm` is kept in the PyMC code base for now, but it will move to ArviZ once the latter adds features for regression plots.
 """
 import functools
 import sys
@@ -24,98 +25,6 @@
 
 import arviz as az
 
-
-def map_args(func):
-    swaps = [("varnames", "var_names")]
-
-    @functools.wraps(func)
-    def wrapped(*args, **kwargs):
-        for (old, new) in swaps:
-            if old in kwargs and new not in kwargs:
-                warnings.warn(
-                    f"Keyword argument `{old}` renamed to `{new}`, and will be removed in pymc3 3.8"
-                )
-                kwargs[new] = kwargs.pop(old)
-            return func(*args, **kwargs)
-
-    return wrapped
-
-
-# pymc3 custom plots: override these names for custom behavior
-autocorrplot = map_args(az.plot_autocorr)
-forestplot = map_args(az.plot_forest)
-kdeplot = map_args(az.plot_kde)
-plot_posterior = map_args(az.plot_posterior)
-energyplot = map_args(az.plot_energy)
-densityplot = map_args(az.plot_density)
-pairplot = map_args(az.plot_pair)
-
-# Use compact traceplot by default
-@map_args
-@functools.wraps(az.plot_trace)
-def traceplot(*args, **kwargs):
-    try:
-        kwargs.setdefault("compact", True)
-        return az.plot_trace(*args, **kwargs)
-    except TypeError:
-        kwargs.pop("compact")
-        return az.plot_trace(*args, **kwargs)
-
-
-# addition arg mapping for compare plot
-@functools.wraps(az.plot_compare)
-def compareplot(*args, **kwargs):
-    if "comp_df" in kwargs:
-        comp_df = kwargs["comp_df"].copy()
-    else:
-        args = list(args)
-        comp_df = args[0].copy()
-    if "WAIC" in comp_df.columns:
-        comp_df = comp_df.rename(
-            index=str,
-            columns={
-                "WAIC": "waic",
-                "pWAIC": "p_waic",
-                "dWAIC": "d_waic",
-                "SE": "se",
-                "dSE": "dse",
-                "var_warn": "warning",
-            },
-        )
-    elif "LOO" in comp_df.columns:
-        comp_df = comp_df.rename(
-            index=str,
-            columns={
-                "LOO": "loo",
-                "pLOO": "p_loo",
-                "dLOO": "d_loo",
-                "SE": "se",
-                "dSE": "dse",
-                "shape_warn": "warning",
-            },
-        )
-    if "comp_df" in kwargs:
-        kwargs["comp_df"] = comp_df
-    else:
-        args[0] = comp_df
-    return az.plot_compare(*args, **kwargs)
-
-
 from pymc3.plots.posteriorplot import plot_posterior_predictive_glm
 
-# Access to arviz plots: base plots provided by arviz
-for plot in az.plots.__all__:
-    setattr(sys.modules[__name__], plot, map_args(getattr(az.plots, plot)))
-
-__all__ = tuple(az.plots.__all__) + (
-    "autocorrplot",
-    "compareplot",
-    "forestplot",
-    "kdeplot",
-    "plot_posterior",
-    "traceplot",
-    "energyplot",
-    "densityplot",
-    "pairplot",
-    "plot_posterior_predictive_glm",
-)
+__all__ = ["plot_posterior_predictive_glm"]
diff --git a/pymc3/plots/posteriorplot.py b/pymc3/plots/posteriorplot.py
index 08ba7f0487..92b341d9cc 100644
--- a/pymc3/plots/posteriorplot.py
+++ b/pymc3/plots/posteriorplot.py
@@ -14,6 +14,8 @@
 
 from __future__ import annotations
 
+import warnings
+
 from typing import TYPE_CHECKING, Any, Callable, Optional, Union
 
 import matplotlib.pyplot as plt
@@ -33,20 +35,33 @@ def plot_posterior_predictive_glm(
     **kwargs: Any
 ) -> None:
     """Plot posterior predictive of a linear model.
-    :Arguments:
-        trace: InferenceData or MultiTrace
-            Output of pm.sample()
-        eval: <array>
-            Array over which to evaluate lm
-        lm: function <default: linear function>
-            Function mapping parameters at different points
-            to their respective outputs.
-            input: point, sample
-            output: estimated value
-        samples: int <default=30>
-            How many posterior samples to draw.
-    Additional keyword arguments are passed to pylab.plot().
+
+    Parameters
+    ----------
+    trace: InferenceData or MultiTrace
+        Output of pm.sample()
+    eval: <array>
+        Array over which to evaluate lm
+    lm: function <default: linear function>
+        Function mapping parameters at different points
+        to their respective outputs.
+        input: point, sample
+        output: estimated value
+    samples: int <default=30>
+        How many posterior samples to draw.
+    kwargs : mapping, optional
+        Additional keyword arguments are passed to ``matplotlib.pyplot.plot()``.
+
+    Warnings
+    --------
+    The `plot_posterior_predictive_glm` function will be removed in a future PyMC3 release.
     """
+    warnings.warn(
+        "The `plot_posterior_predictive_glm` function will migrate to Arviz in a future release. "
+        "\nKeep up to date with `ArviZ <https://arviz-devs.github.io/arviz/>`_ for future updates.",
+        DeprecationWarning,
+    )
+
     if lm is None:
         lm = lambda x, sample: sample["Intercept"] + sample["x"] * x
 
diff --git a/pymc3/sampling.py b/pymc3/sampling.py
index 0dedb87723..2699d7c6d2 100644
--- a/pymc3/sampling.py
+++ b/pymc3/sampling.py
@@ -416,7 +416,7 @@ def sample(
            ...:     y = pm.Binomial("y", n=n, p=p, observed=h)
            ...:     trace = pm.sample()
 
-        In [3]: pm.summary(trace, kind="stats")
+        In [3]: az.summary(trace, kind="stats")
 
         Out[3]:
             mean     sd  hdi_3%  hdi_97%
diff --git a/pymc3/stats/__init__.py b/pymc3/stats/__init__.py
deleted file mode 100644
index d4670d67ce..0000000000
--- a/pymc3/stats/__init__.py
+++ /dev/null
@@ -1,69 +0,0 @@
-#   Copyright 2020 The PyMC Developers
-#
-#   Licensed under the Apache License, Version 2.0 (the "License");
-#   you may not use this file except in compliance with the License.
-#   You may obtain a copy of the License at
-#
-#       http://www.apache.org/licenses/LICENSE-2.0
-#
-#   Unless required by applicable law or agreed to in writing, software
-#   distributed under the License is distributed on an "AS IS" BASIS,
-#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-#   See the License for the specific language governing permissions and
-#   limitations under the License.
-
-"""Statistical utility functions for PyMC3
-
-Diagnostics and auxiliary statistical functions are delegated to the ArviZ library, a general
-purpose library for "exploratory analysis of Bayesian models." See
-https://arviz-devs.github.io/arviz/ for details.
-"""
-import functools
-import warnings
-
-import arviz as az
-
-
-def map_args(func):
-    swaps = [("varnames", "var_names")]
-
-    @functools.wraps(func)
-    def wrapped(*args, **kwargs):
-        for (old, new) in swaps:
-            if old in kwargs and new not in kwargs:
-                warnings.warn(
-                    "Keyword argument `{old}` renamed to `{new}`, and will be removed in "
-                    "pymc3 3.9".format(old=old, new=new)
-                )
-                kwargs[new] = kwargs.pop(old)
-            return func(*args, **kwargs)
-
-    return wrapped
-
-
-bfmi = map_args(az.bfmi)
-compare = map_args(az.compare)
-ess = map_args(az.ess)
-geweke = map_args(az.geweke)
-hpd = map_args(az.hpd)
-loo = map_args(az.loo)
-mcse = map_args(az.mcse)
-r2_score = map_args(az.r2_score)
-rhat = map_args(az.rhat)
-summary = map_args(az.summary)
-waic = map_args(az.waic)
-
-
-__all__ = [
-    "bfmi",
-    "compare",
-    "ess",
-    "geweke",
-    "hpd",
-    "loo",
-    "mcse",
-    "r2_score",
-    "rhat",
-    "summary",
-    "waic",
-]
diff --git a/pymc3/step_methods/mlda.py b/pymc3/step_methods/mlda.py
index f99af28252..559f894f30 100644
--- a/pymc3/step_methods/mlda.py
+++ b/pymc3/step_methods/mlda.py
@@ -333,7 +333,7 @@ class MLDA(ArrayStepShared):
         ...                       tune=100, step=step_method,
         ...                       random_seed=123)
         ...
-        ... pm.summary(trace, kind="stats")
+        ... az.summary(trace, kind="stats")
            mean     sd  hdi_3%  hdi_97%
         x  0.99  0.987  -0.734    2.992
 


Test patch:
diff --git a/pymc3/tests/sampler_fixtures.py b/pymc3/tests/sampler_fixtures.py
index 318165c697..fcf66f1556 100644
--- a/pymc3/tests/sampler_fixtures.py
+++ b/pymc3/tests/sampler_fixtures.py
@@ -12,6 +12,7 @@
 #   See the License for the specific language governing permissions and
 #   limitations under the License.
 
+import arviz as az
 import numpy as np
 import numpy.testing as npt
 import theano.tensor as tt
@@ -146,12 +147,12 @@ def setup_class(cls):
 
     def test_neff(self):
         if hasattr(self, "min_n_eff"):
-            n_eff = pm.ess(self.trace[self.burn :])
+            n_eff = az.ess(self.trace[self.burn :])
             for var in n_eff:
                 npt.assert_array_less(self.min_n_eff, n_eff[var])
 
     def test_Rhat(self):
-        rhat = pm.rhat(self.trace[self.burn :])
+        rhat = az.rhat(self.trace[self.burn :])
         for var in rhat:
             npt.assert_allclose(rhat[var], 1, rtol=0.01)
 
diff --git a/pymc3/tests/test_examples.py b/pymc3/tests/test_examples.py
index 9b741a9674..d79093b392 100644
--- a/pymc3/tests/test_examples.py
+++ b/pymc3/tests/test_examples.py
@@ -12,6 +12,7 @@
 #   See the License for the specific language governing permissions and
 #   limitations under the License.
 
+import arviz as az
 import matplotlib
 import numpy as np
 import pandas as pd
@@ -202,7 +203,7 @@ def test_disaster_model(self):
             # Use slice sampler for means (other variables auto-selected)
             step = pm.Slice([model.early_mean_log__, model.late_mean_log__])
             tr = pm.sample(500, tune=50, start=start, step=step, chains=2)
-            pm.summary(tr)
+            az.summary(tr)
 
     def test_disaster_model_missing(self):
         model = build_disaster_model(masked=True)
@@ -212,7 +213,7 @@ def test_disaster_model_missing(self):
             # Use slice sampler for means (other variables auto-selected)
             step = pm.Slice([model.early_mean_log__, model.late_mean_log__])
             tr = pm.sample(500, tune=50, start=start, step=step, chains=2)
-            pm.summary(tr)
+            az.summary(tr)
 
 
 class TestGLMLinear(SeededTest):