Repository: arviz-devs/arviz

Issue:
Add skip to animation tests if ffmpeg is not installed
**Describe the bug**
Running test suite without ffmpeg in environment will raise numerous failures

**To Reproduce**
Steps to reproduce the behavior. Ideally a self-contained snippet of code, or link to a notebook or external code. Please include screenshots/images produced with ArviZ here, or the stacktrace including `arviz` code to help.

<img width="1038" alt="image" src="https://user-images.githubusercontent.com/7213793/59773837-2dd71a00-9263-11e9-863c-329798656979.png">

> 
> FWARNING:matplotlib.animation:MovieWriter ffmpeg unavailable. Trying to use html instead.
> INFO:matplotlib.animation:Animation.save using <class 'matplotlib.animation.HTMLWriter'>
> 
> test_plots.py:559 (test_plot_ppc_save_animation[cumulative])
> models = <arviz.tests.test_plots.models.<locals>.Models object at 0x19a9999b0>
> kind = 'cumulative'
> 
>     @pytest.mark.parametrize("kind", ["density", "cumulative", "scatter"])
>     def test_plot_ppc_save_animation(models, kind):
>         animation_kwargs = {"blit": False}
>         axes, anim = plot_ppc(
>             models.model_1,
>             kind=kind,
>             animated=True,
>             animation_kwargs=animation_kwargs,
>             num_pp_samples=5,
>             random_seed=3,
>         )
>         assert axes
>         assert anim
>         animations_folder = "saved_animations"
>         os.makedirs(animations_folder, exist_ok=True)
>         path = os.path.join(animations_folder, "ppc_{}_animation.mp4".format(kind))
> >       anim.save(path)
> 
> test_plots.py:576: 
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
> ../../../../.miniconda3/envs/arviz/lib/python3.7/site-packages/matplotlib/animation.py:1166: in save
>     with writer.saving(self._fig, filename, dpi):
> ../../../../.miniconda3/envs/arviz/lib/python3.7/contextlib.py:112: in __enter__
>     return next(self.gen)
> ../../../../.miniconda3/envs/arviz/lib/python3.7/site-packages/matplotlib/animation.py:228: in saving
>     self.setup(fig, outfile, dpi, *args, **kwargs)
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
> 
> self = <matplotlib.animation.HTMLWriter object at 0x1ca9415f8>
> fig = <Figure size 640x480 with 1 Axes>
> outfile = 'saved_animations/ppc_cumulative_animation.mp4', dpi = 100.0
> frame_dir = None
> 
>     def setup(self, fig, outfile, dpi, frame_dir=None):
>         root, ext = os.path.splitext(outfile)
>         if ext not in ['.html', '.htm']:
> >           raise ValueError("outfile must be *.htm or *.html")
> E           ValueError: outfile must be *.htm or *.html
> 
> ../../../../.miniconda3/envs/arviz/lib/python3.7/site-packages/matplotlib/animation.py:857: ValueError

**Expected behavior**
Tests should be skipped if ffmpeg is not installed to avoid erroneous failures

**Additional context**
Versions of `arviz` and other libraries used, operating system used, and anything else that may be useful.

Cannot get log-likelihood from Logistic Regression in Pyro 
I'm trying to get log-likelihood estimates via arviz from my logistic regression model

My model looks like this:

```
def logistic_regression(X, obs=None):
   alpha = pyro.sample("alpha", dist.Normal(0, 5.)) # Prior for the bias/intercept
   beta  = pyro.sample("beta", dist.Normal(torch.zeros(X.shape[1]), 
                                            5.*torch.ones(X.shape[1]))) # Priors for the regression coeffcients
    
   with pyro.plate("data"):
       logits = alpha + X.matmul(beta)
       y = pyro.sample("y", dist.Bernoulli(logits=logits), obs=obs)
        
   return y
```

I train it via standard MCMC, like this:

```
nuts_kernel = NUTS(logistic_regression,jit_compile=True, ignore_jit_warnings=True)
mcmc_logistic = MCMC(nuts_kernel, num_samples=1000, warmup_steps=500, num_chains=2)
mcmc_logistic.run(X_train, y_train)
mcmc_logistic.summary()
```

And then I would like to convert it to an arviz module.

```
pyro_data_logit = az.from_pyro(
    mcmc_logistic,
    prior=prior_logit,
    posterior_predictive=posterior_predictive_logit
)
```

The problem is that this object does not contain a log-likelihood, which seems weird. The missing log-likehood is problematic as I can use other functionalities such as .loo   

I'm able to calculate the log-likelihood by hand, like

```
alpha_hat = samples_logistic["alpha"].mean(axis=0).detach().numpy()
beta_hat = samples_logistic["beta"].mean(axis=0).detach().numpy()
y_hat = sigmoid(alpha_hat + np.dot(X_test, beta_hat))
log_likelihood_logit = stats.binom.logpmf(1, n = 1, p =y_hat)
```

By arviz.frompyro does not accept manually specified log-likelihood as argument (arviz.frompystan accept this argument).

I'm using arviz version '0.7.0'  and pyro version '1.2.1'





Discussion:
Maybe a skipif after having checked [`is_available`](https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.animation.MovieWriterRegistry.html#matplotlib.animation.MovieWriterRegistry.is_available)?
Hi! Thanks for reporting.

`az.from_pyro` uses [`pyro.Predictive.get_vectorized_trace()`](http://docs.pyro.ai/en/stable/inference_algos.html#pyro.infer.predictive.Predictive.get_vectorized_trace) to automatically retrieve and store pointwise log likelihood samples. To get the `log_likelihood` group populated, ArviZ must be able to execute this function, which as stated in Pyro docs: 

> ... requires that the model has all batch dims correctly annotated via `plate`.

In your case, I think you need to annotate the dimension of `beta` `X.shape[1]` with `plate` to allow ArviZ to automatically populate `log_likelihood` group. 

If this were not to work (I am not a Pyro expert), you can always combine `from_pyro` with `from_dict` and do something like:

    pyro_data_logit = az.concat(
        az.from_pyro(...), 
        az.from_dict(log_likelihood={"y": log_likelihood_logit}
    )

Note that this requires `log_likelihood_logit` to follow ArviZ dimension order: `chain, draw, *shape`, otherwise they will not be annotated correctly.

Resolving patch:
diff --git a/arviz/data/io_pymc3.py b/arviz/data/io_pymc3.py
--- a/arviz/data/io_pymc3.py
+++ b/arviz/data/io_pymc3.py
@@ -86,7 +86,7 @@ def __init__(
                 "Using `from_pymc3` without the model will be deprecated in a future release. "
                 "Not using the model will return less accurate and less useful results. "
                 "Make sure you use the model argument or call from_pymc3 within a model context.",
-                PendingDeprecationWarning,
+                FutureWarning,
             )
 
         # This next line is brittle and may not work forever, but is a secret
diff --git a/arviz/data/io_pyro.py b/arviz/data/io_pyro.py
--- a/arviz/data/io_pyro.py
+++ b/arviz/data/io_pyro.py
@@ -1,5 +1,6 @@
 """Pyro-specific conversion code."""
 import logging
+import warnings
 import numpy as np
 from packaging import version
 import xarray as xr
@@ -26,6 +27,7 @@ def __init__(
         posterior=None,
         prior=None,
         posterior_predictive=None,
+        log_likelihood=True,
         predictions=None,
         constant_data=None,
         predictions_constant_data=None,
@@ -62,6 +64,7 @@ def __init__(
         self.posterior = posterior
         self.prior = prior
         self.posterior_predictive = posterior_predictive
+        self.log_likelihood = log_likelihood
         self.predictions = predictions
         self.constant_data = constant_data
         self.predictions_constant_data = predictions_constant_data
@@ -130,6 +133,8 @@ def sample_stats_to_xarray(self):
     @requires("model")
     def log_likelihood_to_xarray(self):
         """Extract log likelihood from Pyro posterior."""
+        if not self.log_likelihood:
+            return None
         data = {}
         if self.observations is not None:
             try:
@@ -143,6 +148,10 @@ def log_likelihood_to_xarray(self):
                     data[obs_name] = np.reshape(log_like, shape)
             except:  # pylint: disable=bare-except
                 # cannot get vectorized trace
+                warnings.warn(
+                    "Could not get vectorized trace, log_likelihood group will be omitted. "
+                    "Check your model vectorization or set log_likelihood=False"
+                )
                 return None
         return dict_to_dataset(data, library=self.pyro, coords=self.coords, dims=self.dims)
 
@@ -273,6 +282,7 @@ def from_pyro(
     *,
     prior=None,
     posterior_predictive=None,
+    log_likelihood=True,
     predictions=None,
     constant_data=None,
     predictions_constant_data=None,
@@ -294,6 +304,8 @@ def from_pyro(
         Prior samples from a Pyro model
     posterior_predictive : dict
         Posterior predictive samples for the posterior
+    log_likelihood : bool, optional
+        Calculate and store pointwise log likelihood values.
     predictions: dict
         Out of sample predictions
     constant_data: dict
@@ -313,6 +325,7 @@ def from_pyro(
         posterior=posterior,
         prior=prior,
         posterior_predictive=posterior_predictive,
+        log_likelihood=log_likelihood,
         predictions=predictions,
         constant_data=constant_data,
         predictions_constant_data=predictions_constant_data,


Test patch:
diff --git a/arviz/tests/base_tests/test_plots_matplotlib.py b/arviz/tests/base_tests/test_plots_matplotlib.py
--- a/arviz/tests/base_tests/test_plots_matplotlib.py
+++ b/arviz/tests/base_tests/test_plots_matplotlib.py
@@ -3,6 +3,7 @@
 import os
 from copy import deepcopy
 import matplotlib.pyplot as plt
+from matplotlib import animation
 from pandas import DataFrame
 from scipy.stats import gaussian_kde
 import numpy as np
@@ -497,6 +498,8 @@ def test_plot_pair_shapes(marginals, max_subplots):
 @pytest.mark.parametrize("alpha", [None, 0.2, 1])
 @pytest.mark.parametrize("animated", [False, True])
 def test_plot_ppc(models, kind, alpha, animated):
+    if animation and not animation.writers.is_available("ffmpeg"):
+        pytest.skip("matplotlib animations within ArviZ require ffmpeg")
     animation_kwargs = {"blit": False}
     axes = plot_ppc(
         models.model_1,
@@ -516,6 +519,8 @@ def test_plot_ppc(models, kind, alpha, animated):
 @pytest.mark.parametrize("jitter", [None, 0, 0.1, 1, 3])
 @pytest.mark.parametrize("animated", [False, True])
 def test_plot_ppc_multichain(kind, jitter, animated):
+    if animation and not animation.writers.is_available("ffmpeg"):
+        pytest.skip("matplotlib animations within ArviZ require ffmpeg")
     data = from_dict(
         posterior_predictive={
             "x": np.random.randn(4, 100, 30),
@@ -543,6 +548,8 @@ def test_plot_ppc_multichain(kind, jitter, animated):
 @pytest.mark.parametrize("kind", ["kde", "cumulative", "scatter"])
 @pytest.mark.parametrize("animated", [False, True])
 def test_plot_ppc_discrete(kind, animated):
+    if animation and not animation.writers.is_available("ffmpeg"):
+        pytest.skip("matplotlib animations within ArviZ require ffmpeg")
     data = from_dict(
         observed_data={"obs": np.random.randint(1, 100, 15)},
         posterior_predictive={"obs": np.random.randint(1, 300, (1, 20, 15))},
@@ -556,6 +563,10 @@ def test_plot_ppc_discrete(kind, animated):
     assert axes
 
 
+@pytest.mark.skipif(
+    not animation.writers.is_available("ffmpeg"),
+    reason="matplotlib animations within ArviZ require ffmpeg",
+)
 @pytest.mark.parametrize("kind", ["kde", "cumulative", "scatter"])
 def test_plot_ppc_save_animation(models, kind):
     animation_kwargs = {"blit": False}
@@ -577,6 +588,10 @@ def test_plot_ppc_save_animation(models, kind):
     assert os.path.getsize(path)
 
 
+@pytest.mark.skipif(
+    not animation.writers.is_available("ffmpeg"),
+    reason="matplotlib animations within ArviZ require ffmpeg",
+)
 @pytest.mark.parametrize("kind", ["kde", "cumulative", "scatter"])
 def test_plot_ppc_discrete_save_animation(kind):
     data = from_dict(
@@ -602,6 +617,10 @@ def test_plot_ppc_discrete_save_animation(kind):
     assert os.path.getsize(path)
 
 
+@pytest.mark.skipif(
+    not animation.writers.is_available("ffmpeg"),
+    reason="matplotlib animations within ArviZ require ffmpeg",
+)
 @pytest.mark.parametrize("system", ["Windows", "Darwin"])
 def test_non_linux_blit(models, monkeypatch, system, caplog):
     import platform
@@ -657,6 +676,10 @@ def test_plot_ppc_ax(models, kind, fig_ax):
     assert axes[0] is ax
 
 
+@pytest.mark.skipif(
+    not animation.writers.is_available("ffmpeg"),
+    reason="matplotlib animations within ArviZ require ffmpeg",
+)
 def test_plot_ppc_bad_ax(models, fig_ax):
     _, ax = fig_ax
     _, ax2 = plt.subplots(1, 2)
diff --git a/arviz/tests/external_tests/test_data_pymc.py b/arviz/tests/external_tests/test_data_pymc.py
--- a/arviz/tests/external_tests/test_data_pymc.py
+++ b/arviz/tests/external_tests/test_data_pymc.py
@@ -425,7 +425,7 @@ def test_no_model_deprecation(self):
             obs = pm.Normal("obs", x * beta, 1, observed=y)  # pylint: disable=unused-variable
             prior = pm.sample_prior_predictive()
 
-        with pytest.warns(PendingDeprecationWarning, match="without the model"):
+        with pytest.warns(FutureWarning, match="without the model"):
             inference_data = from_pymc3(prior=prior)
         test_dict = {
             "prior": ["beta", "obs"],
diff --git a/arviz/tests/external_tests/test_data_pyro.py b/arviz/tests/external_tests/test_data_pyro.py
--- a/arviz/tests/external_tests/test_data_pyro.py
+++ b/arviz/tests/external_tests/test_data_pyro.py
@@ -17,6 +17,7 @@
 torch = importorskip("torch")
 pyro = importorskip("pyro")
 Predictive = pyro.infer.Predictive
+dist = pyro.distributions
 
 
 class TestDataPyro:
@@ -164,9 +165,6 @@ def test_inference_data_only_posterior_has_log_likelihood(self, data):
         assert not fails
 
     def test_multiple_observed_rv(self):
-        import pyro.distributions as dist
-        from pyro.infer import MCMC, NUTS
-
         y1 = torch.randn(10)
         y2 = torch.randn(10)
 
@@ -175,8 +173,8 @@ def model_example_multiple_obs(y1=None, y2=None):
             pyro.sample("y1", dist.Normal(x, 1), obs=y1)
             pyro.sample("y2", dist.Normal(x, 1), obs=y2)
 
-        nuts_kernel = NUTS(model_example_multiple_obs)
-        mcmc = MCMC(nuts_kernel, num_samples=10)
+        nuts_kernel = pyro.infer.NUTS(model_example_multiple_obs)
+        mcmc = pyro.infer.MCMC(nuts_kernel, num_samples=10)
         mcmc.run(y1=y1, y2=y2)
         inference_data = from_pyro(mcmc)
         test_dict = {
@@ -190,9 +188,6 @@ def model_example_multiple_obs(y1=None, y2=None):
         assert not hasattr(inference_data.sample_stats, "log_likelihood")
 
     def test_inference_data_constant_data(self):
-        import pyro.distributions as dist
-        from pyro.infer import MCMC, NUTS
-
         x1 = 10
         x2 = 12
         y1 = torch.randn(10)
@@ -201,8 +196,8 @@ def model_constant_data(x, y1=None):
             _x = pyro.sample("x", dist.Normal(1, 3))
             pyro.sample("y1", dist.Normal(x * _x, 1), obs=y1)
 
-        nuts_kernel = NUTS(model_constant_data)
-        mcmc = MCMC(nuts_kernel, num_samples=10)
+        nuts_kernel = pyro.infer.NUTS(model_constant_data)
+        mcmc = pyro.infer.MCMC(nuts_kernel, num_samples=10)
         mcmc.run(x=x1, y1=y1)
         posterior = mcmc.get_samples()
         posterior_predictive = Predictive(model_constant_data, posterior)(x1)
@@ -232,3 +227,34 @@ def test_inference_data_num_chains(self, predictions_data, chains):
         inference_data = from_pyro(predictions=predictions, num_chains=chains)
         nchains = inference_data.predictions.dims["chain"]
         assert nchains == chains
+
+    @pytest.mark.parametrize("log_likelihood", [True, False])
+    def test_log_likelihood(self, log_likelihood):
+        """Test behaviour when log likelihood cannot be retrieved.
+
+        If log_likelihood=True there is a warning to say log_likelihood group is skipped,
+        if log_likelihood=False there is no warning and log_likelihood is skipped.
+        """
+        x = torch.randn((10, 2))
+        y = torch.randn(10)
+
+        def model_constant_data(x, y=None):
+            beta = pyro.sample("beta", dist.Normal(torch.ones(2), 3))
+            pyro.sample("y", dist.Normal(x.matmul(beta), 1), obs=y)
+
+        nuts_kernel = pyro.infer.NUTS(model_constant_data)
+        mcmc = pyro.infer.MCMC(nuts_kernel, num_samples=10)
+        mcmc.run(x=x, y=y)
+        if log_likelihood:
+            with pytest.warns(UserWarning, match="Could not get vectorized trace"):
+                inference_data = from_pyro(mcmc, log_likelihood=log_likelihood)
+        else:
+            inference_data = from_pyro(mcmc, log_likelihood=log_likelihood)
+        test_dict = {
+            "posterior": ["beta"],
+            "sample_stats": ["diverging"],
+            "~log_likelihood": [],
+            "observed_data": ["y"],
+        }
+        fails = check_multiple_attrs(test_dict, inference_data)
+        assert not fails