Repository: pyro-ppl/pyro

Issue:
Maximum a posteriori inference
I am considering using pyro for my problem, but I don't quite grok the library yet.  Part of my confusion comes from SVI being the main well documented inference engine in pyro, and I am a beginner to variational inference.

For me, the simplest "inference" methods to understand are maximum a posteriori (MAP) and maximum likelihood, and it would be nice if the `pyro` had some examples of these. I assume `pyro` has the optimization tools necessary to handle MAP, but the documentation is not very clear about how to do it.


Discussion:
Great question!  You can do MAP inference by using SVI with `Delta` distributions in your guide:
```python
def model():
    mu = pyro.sample("mu", dist.normal, Variable(torch.zeros(1)), Variable(torch.ones(1)))
    return pyro.sample("x", dist.normal, mu, Variable(torch.ones(1)))

def guide():
    mu_param = pyro.param("mu_param", Variable(torch.randn(1), requires_grad=True))
    return pyro.sample("mu", dist.delta, mu_param)
```

For maximum likelihood, you can either put very wide `Uniform` priors on your latent variables in the model and proceed with deltas in the guide as above (not recommended), or you can change the `sample` statements in your model into `param` statements and use an empty guide:
```python
def model():
    mu = pyro.param("mu", Variable(torch.randn(1), requires_grad=True))
    return pyro.sample("x", dist.normal, mu, Variable(torch.ones(1)))

def guide():
    return None
```
Thanks so much for that description. I am still a little confused about how variational inference with delta functions as the approximating distribution reduces to MAP.  This is something that `edward` also glosses over in its [MAP documentation](http://edwardlib.org/api/ed/MAP). I guess the ELBO equals the log of joint probability in the case when q(z) is a delta function. Would it be worth adding an inference class for this special case?
:+1: for a helper along the lines of
```py
def MAP(model, *args, **kwargs):
    guide = poutine.lower(model)  # poutine.lower is opposite of poutine.lift
    return SVI(model, guide, loss="ELBO", *args, **kwargs)
```
> I guess the ELBO equals the log of joint probability in the case when q(z) is a delta function

Yep, this is correct.

> Would it be worth adding an inference class for this special case?

We could do something like what @fritzo suggested but I'm partial to the manual `Delta` guide method because it doesn't require executing the model twice.
At least for beginners, I think it would be nice to have a MAP convenience function which someone who knows nothing about variational inference can use. 
Great use case of poutine.lower!
@nbren12 I believe #573 should work for you. The purpose of SVI and guides (over simple MAP) is to be able to deal with latent variables whose value you want to integrate over, rather than optimize. The purpose of the guide is to provide a good random guess for each latent variable.
@eb8680 
1) In your MAP and MLE examples, where does the data come into play? In particular, you didn't 'observe' anything. If data is not involved, how can pyro estimate anything? 
2) Pyro documentation insists that the guide 'samples' the same latents as the model. However,  this is not the case in any of your examples. Is there a bug in the examples?
3) It would be great if you could provide an actual minimal running example -- e.g. learn the mean of an actual dataset of 100 numbers sampled from a Gaussian distribution.

Please help - I'm also trying to learn pyro starting from these basic cases and am finding it very difficult -- which is a shame, there seems to be great potential here!

@kjchalup This should MAP estimate the mean `mu_est` of numbers packed into `data`:
```py
def model(data):
    pyro.sample('mu', normal, Variable(torch.zeros(1)), Variable(torch.ones(1)))
    for i, x in enumerate(data):
        pyro.sample('obs_{}'.format(i), normal, mu, Variable(torch.ones(1)), obs=x)

def guide(data):
    mu_map = pyro.param('mu_map', Variable(torch.randn(1), requires_grad=True))
    pyro.sample('mu', delta, mu_map)
```
@kjchalup are your questions specifically about MAP, or about Pyro more generally?  If it's the latter, I recommend our [examples and tutorials](http://pyro.ai/examples/index.html), especially the [introductory inference tutorial](pyro.ai/examples/intro_part_ii.html) and the [stochastic variational inference (SVI) tutorial](http://pyro.ai/examples/svi_part_i.html).  To specifically address your questions:

1. Data enters a model through `pyro.condition` or the `obs` kwarg of `pyro.sample`, whose semantics are described in more detail in the [inference tutorial](pyro.ai/examples/intro_part_ii.html) and the [documentation](http://docs.pyro.ai/primitives.html#pyro.__init__.sample) respectively.  In my original snippets, the use of `pyro.condition` was not obvious or explicit - I've fleshed it out below.
2. The guide must sample the same *unconstrained* latents as the model.  In this case, since sample site "x" will be constrained by observed data, the guide should not contain a sample site "x".
3. See below for a more complete example:

```python
def model():
    mu = pyro.sample("mu", dist.normal, Variable(torch.zeros(1)), Variable(torch.ones(1)))
    return pyro.sample("x", dist.normal, mu, Variable(torch.ones(1)))

def guide():
    mu_param = pyro.param("mu_param", Variable(torch.randn(1), requires_grad=True))
    return pyro.sample("mu", dist.delta, mu_param)

# use pyro.condition to constrain model sample site "x" to be equal to some fixed observed data
conditioned_model = pyro.condition(model, data={"x": data})

svi = pyro.infer.SVI(model=conditioned_model,
                     guide=guide,
                     optim=pyro.optim.SGD({"lr": 0.001}),
                     loss="ELBO")

losses = []
for t in range(1000):
    # svi.step is called with whatever arguments model and guide take - in this case, nothing
    losses.append(svi.step())

# to get the value of mu, just call guide
mu_final = guide()
```

Resolving patch:
diff --git a/pyro/contrib/autoguide/__init__.py b/pyro/contrib/autoguide/__init__.py
--- a/pyro/contrib/autoguide/__init__.py
+++ b/pyro/contrib/autoguide/__init__.py
@@ -15,6 +15,7 @@ def model():
 
 from __future__ import absolute_import, division, print_function
 
+import numbers
 import weakref
 
 import torch
@@ -33,12 +34,14 @@ def model():
     from contextlib2 import ExitStack  # python 2
 
 __all__ = [
-    'AutoGuide',
-    'AutoGuideList',
     'AutoContinuous',
-    'AutoMultivariateNormal',
+    'AutoDelta',
     'AutoDiagonalNormal',
     'AutoDiscreteParallel',
+    'AutoGuide',
+    'AutoGuideList',
+    'AutoLowRankMultivariateNormal',
+    'AutoMultivariateNormal',
 ]
 
 
@@ -65,6 +68,7 @@ def __init__(self, model):
         self.master = None
         self.model = model
         self.prototype_trace = None
+        self._iaranges = {}
 
     def __call__(self, *args, **kwargs):
         """
@@ -88,6 +92,23 @@ def _create_iaranges(self):
         return {frame.name: pyro.iarange(frame.name, frame.size, dim=frame.dim)
                 for frame in sorted(self._iaranges.values())}
 
+    def _setup_prototype(self, *args, **kwargs):
+        # run the model so we can inspect its structure
+        self.prototype_trace = poutine.block(poutine.trace(self.model).get_trace)(*args, **kwargs)
+        self.prototype_trace = prune_subsample_sites(self.prototype_trace)
+        if self.master is not None:
+            self.master()._check_prototype(self.prototype_trace)
+
+        self._iaranges = {}
+        for name, site in self.prototype_trace.nodes.items():
+            if site["type"] != "sample" or site["is_observed"]:
+                continue
+            for frame in site["cond_indep_stack"]:
+                if frame.vectorized:
+                    self._iaranges[frame.name] = frame
+                else:
+                    raise NotImplementedError("AutoGuideList does not support pyro.irange")
+
 
 class AutoGuideList(AutoGuide):
     """
@@ -105,7 +126,6 @@ class AutoGuideList(AutoGuide):
     def __init__(self, model):
         super(AutoGuideList, self).__init__(model)
         self.parts = []
-        self._iaranges = {}
         self.iaranges = {}
 
     def _check_prototype(self, part_trace):
@@ -151,20 +171,61 @@ def __call__(self, *args, **kwargs):
             result.update(part(*args, **kwargs))
         return result
 
-    def _setup_prototype(self, *args, **kwargs):
-        # run the model so we can inspect its structure
-        self.prototype_trace = poutine.block(poutine.trace(self.model).get_trace)(*args, **kwargs)
-        self.prototype_trace = prune_subsample_sites(self.prototype_trace)
 
-        self._iaranges = {}
+class AutoDelta(AutoGuide):
+    """
+    This implementation of :class:`AutoGuide` uses Delta distributions to
+    construct a MAP guide over the entire latent space. The guide does not
+    depend on the model's ``*args, **kwargs``.
+
+    Usage::
+
+        guide = AutoDelta(model)
+        svi = SVI(model, guide, ...)
+
+    By default latent variables are randomly initialized by the model.  To
+    change this default behavior the user should call :func:`pyro.param` before
+    beginning inference, with ``"auto_"`` prefixed to the targetd sample site
+    names e.g. for sample sites named "level" and "concentration", initialize
+    via::
+
+        pyro.param("auto_level", torch.tensor([-1., 0., 1.]))
+        pyro.param("auto_concentration", torch.ones(k),
+                   constraint=constraints.positive)
+    """
+    def __call__(self, *args, **kwargs):
+        """
+        An automatic guide with the same ``*args, **kwargs`` as the base ``model``.
+
+        :return: A dict mapping sample site name to sampled value.
+        :rtype: dict
+        """
+        # if we've never run the model before, do so now so we can inspect the model structure
+        if self.prototype_trace is None:
+            self._setup_prototype(*args, **kwargs)
+
+        iaranges = self._create_iaranges()
+        result = {}
         for name, site in self.prototype_trace.nodes.items():
             if site["type"] != "sample" or site["is_observed"]:
                 continue
-            for frame in site["cond_indep_stack"]:
-                if frame.vectorized:
-                    self._iaranges[frame.name] = frame
-                else:
-                    raise NotImplementedError("AutoGuideList does not support pyro.irange")
+            with ExitStack() as stack:
+                for frame in site["cond_indep_stack"]:
+                    if frame.vectorized:
+                        stack.enter_context(iaranges[frame.name])
+                value = pyro.param("auto_{}".format(name), site["value"].detach(),
+                                   constraint=site["fn"].support)
+                result[name] = pyro.sample(name, dist.Delta(value, event_dim=site["fn"].event_dim))
+        return result
+
+    def median(self, *args, **kwargs):
+        """
+        Returns the posterior median value of each latent variable.
+
+        :return: A dict mapping sample site name to median tensor.
+        :rtype: dict
+        """
+        return self(*args, **kwargs)
 
 
 class AutoContinuous(AutoGuide):
@@ -186,29 +247,19 @@ class AutoContinuous(AutoGuide):
         Blei
     """
     def _setup_prototype(self, *args, **kwargs):
-        # run the model so we can inspect its structure
-        self.prototype_trace = poutine.block(poutine.trace(self.model).get_trace)(*args, **kwargs)
-        self.prototype_trace = prune_subsample_sites(self.prototype_trace)
-        if self.master is not None:
-            self.master()._check_prototype(self.prototype_trace)
-
+        super(AutoContinuous, self)._setup_prototype(*args, **kwargs)
         self._unconstrained_shapes = {}
         self._cond_indep_stacks = {}
-        self._iaranges = {}
         for name, site in self.prototype_trace.nodes.items():
             if site["type"] != "sample" or site["is_observed"]:
                 continue
 
-            # collect the shapes of unconstrained values, which may differ from the shapes of constrained values
+            # Collect the shapes of unconstrained values.
+            # These may differ from the shapes of constrained values.
             self._unconstrained_shapes[name] = biject_to(site["fn"].support).inv(site["value"]).shape
 
-            # collect independence contexts
+            # Collect independence contexts.
             self._cond_indep_stacks[name] = site["cond_indep_stack"]
-            for frame in site["cond_indep_stack"]:
-                if frame.vectorized:
-                    self._iaranges[frame.name] = frame
-                else:
-                    raise NotImplementedError("AutoContinuous does not support pyro.irange")
 
         self.latent_dim = sum(_product(shape) for shape in self._unconstrained_shapes.values())
 
@@ -266,6 +317,44 @@ def __call__(self, *args, **kwargs):
 
         return result
 
+    def _loc_scale(self, *args, **kwargs):
+        """
+        :returns: a tuple ``(loc, scale)`` used by :meth:`median` and
+            :meth:`quantiles`
+        """
+        raise NotImplementedError
+
+    def median(self, *args, **kwargs):
+        """
+        Returns the posterior median value of each latent variable.
+
+        :return: A dict mapping sample site name to median tensor.
+        :rtype: dict
+        """
+        loc, scale = self._loc_scale(*args, **kwargs)
+        return {site["name"]: biject_to(site["fn"].support)(unconstrained_value)
+                for site, unconstrained_value in self._unpack_latent(loc)}
+
+    def quantiles(self, quantiles, *args, **kwargs):
+        """
+        Returns posterior quantiles each latent variable. Example::
+
+            print(guide.quantiles([0.05, 0.5, 0.95]))
+
+        :param quantiles: A list of requested quantiles between 0 and 1.
+        :type quantiles: torch.Tensor or list
+        :return: A dict mapping sample site name to a list of quantile values.
+        :rtype: dict
+        """
+        loc, scale = self._loc_scale(*args, **kwargs)
+        quantiles = loc.new_tensor(quantiles).unsqueeze(-1)
+        latents = dist.Normal(loc, scale).icdf(quantiles)
+        result = {}
+        for latent in latents:
+            for site, unconstrained_value in self._unpack_latent(latent):
+                result.setdefault(site["name"], []).append(biject_to(site["fn"].support)(unconstrained_value))
+        return result
+
 
 class AutoMultivariateNormal(AutoContinuous):
     """
@@ -292,44 +381,16 @@ def sample_latent(self, *args, **kwargs):
         """
         Samples the (single) multivariate normal latent used in the auto guide.
         """
-        loc = pyro.param("auto_loc", torch.zeros(self.latent_dim))
-        scale_tril = pyro.param("auto_scale_tril", torch.eye(self.latent_dim),
+        loc = pyro.param("auto_loc", lambda: torch.zeros(self.latent_dim))
+        scale_tril = pyro.param("auto_scale_tril", lambda: torch.eye(self.latent_dim),
                                 constraint=constraints.lower_cholesky)
         return pyro.sample("_auto_latent", dist.MultivariateNormal(loc, scale_tril=scale_tril),
                            infer={"is_auxiliary": True})
 
-    def median(self, *args, **kwargs):
-        """
-        Returns the posterior median value of each latent variable.
-
-        :return: A dict mapping sample site name to median tensor.
-        :rtype: dict
-        """
-        latent = pyro.param("auto_loc")
-        return {site["name"]: biject_to(site["fn"].support)(unconstrained_value)
-                for site, unconstrained_value in self._unpack_latent(latent)}
-
-    def quantiles(self, quantiles, *args, **kwargs):
-        """
-        Returns posterior quantiles each latent variable. Example::
-
-            print(guide.quantiles([0.05, 0.5, 0.95]))
-
-        :param quantiles: A list of requested quantiles between 0 and 1.
-        :type quantiles: torch.Tensor or list
-        :return: A dict mapping sample site name to a list of quantile values.
-        :rtype: dict
-        """
+    def _loc_scale(self, *args, **kwargs):
         loc = pyro.param("auto_loc")
         scale = pyro.param("auto_scale_tril").diag()
-        quantiles = loc.new_tensor(quantiles).unsqueeze(-1)
-        latents = dist.Normal(loc, scale).icdf(quantiles)
-
-        result = {}
-        for latent in latents:
-            for site, unconstrained_value in self._unpack_latent(latent):
-                result.setdefault(site["name"], []).append(biject_to(site["fn"].support)(unconstrained_value))
-        return result
+        return loc, scale
 
 
 class AutoDiagonalNormal(AutoContinuous):
@@ -356,44 +417,68 @@ def sample_latent(self, *args, **kwargs):
         """
         Samples the (single) diagnoal normal latent used in the auto guide.
         """
-        loc = pyro.param("auto_loc", torch.zeros(self.latent_dim))
-        scale = pyro.param("auto_scale", torch.ones(self.latent_dim),
+        loc = pyro.param("auto_loc", lambda: torch.zeros(self.latent_dim))
+        scale = pyro.param("auto_scale", lambda: torch.ones(self.latent_dim),
                            constraint=constraints.positive)
         return pyro.sample("_auto_latent", dist.Normal(loc, scale).independent(1),
                            infer={"is_auxiliary": True})
 
-    def median(self, *args, **kwargs):
-        """
-        Returns the posterior median value of each latent variable.
+    def _loc_scale(self, *args, **kwargs):
+        loc = pyro.param("auto_loc")
+        scale = pyro.param("auto_scale")
+        return loc, scale
 
-        :return: A dict mapping sample site name to median tensor.
-        :rtype: dict
-        """
-        latent = pyro.param("auto_loc")
-        return {site["name"]: biject_to(site["fn"].support)(unconstrained_value)
-                for site, unconstrained_value in self._unpack_latent(latent)}
 
-    def quantiles(self, quantiles, *args, **kwargs):
-        """
-        Returns posterior quantiles each latent variable. Example::
+class AutoLowRankMultivariateNormal(AutoContinuous):
+    """
+    This implementation of :class:`AutoContinuous` uses a low rank plus
+    diagonal Multivariate Normal distribution to construct a guide
+    over the entire latent space. The guide does not depend on the model's
+    ``*args, **kwargs``.
 
-            print(guide.quantiles([0.05, 0.5, 0.95]))
+    Usage::
 
-        :param quantiles: A list of requested quantiles between 0 and 1.
-        :type quantiles: torch.Tensor or list
-        :return: A dict mapping sample site name to a list of quantile values.
-        :rtype: dict
+        guide = AutoLowRankMultivariateNormal(model, rank=10)
+        svi = SVI(model, guide, ...)
+
+    By default the ``D_term`` is initialized to 1/2 and the ``W_term`` is
+    intialized randomly such that ``W_term.t().matmul(W_term)`` is half the
+    identity matrix. To change this default behavior the user
+    should call :func:`pyro.param` before beginning inference, e.g.::
+
+        latent_dim = 10
+        pyro.param("auto_loc", torch.randn(latent_dim))
+        pyro.param("auto_W_term", torch.randn(latent_dim)))
+        pyro.param("auto_D_term", torch.randn(latent_dim).exp()),
+                   constraint=constraints.positive)
+
+    :param callable model: a generative model
+    :param int rank: the rank of the low-rank part of the covariance matrix
+    """
+    def __init__(self, model, rank=1):
+        if not isinstance(rank, numbers.Number) or not rank > 0:
+            raise ValueError("Expected rank >= 0 but got {}".format(rank))
+        self.rank = rank
+        super(AutoLowRankMultivariateNormal, self).__init__(model)
+
+    def sample_latent(self, *args, **kwargs):
         """
-        loc = pyro.param("auto_loc")
-        scale = pyro.param("auto_scale")
-        quantiles = loc.new_tensor(quantiles).unsqueeze(-1)
-        latents = dist.Normal(loc, scale).icdf(quantiles)
+        Samples the (single) multivariate normal latent used in the auto guide.
+        """
+        loc = pyro.param("auto_loc", lambda: torch.zeros(self.latent_dim))
+        W_term = pyro.param("auto_W_term",
+                            lambda: torch.randn(self.rank, self.latent_dim) * (0.5 / self.rank) ** 0.5)
+        D_term = pyro.param("auto_D_term", lambda: torch.ones(self.latent_dim) * 0.5,
+                            constraint=constraints.positive)
+        return pyro.sample("_auto_latent", dist.LowRankMultivariateNormal(loc, W_term, D_term),
+                           infer={"is_auxiliary": True})
 
-        result = {}
-        for latent in latents:
-            for site, unconstrained_value in self._unpack_latent(latent):
-                result.setdefault(site["name"], []).append(biject_to(site["fn"].support)(unconstrained_value))
-        return result
+    def _loc_scale(self, *args, **kwargs):
+        loc = pyro.param("auto_loc")
+        W_term = pyro.param("auto_W_term")
+        D_term = pyro.param("auto_D_term")
+        scale = (W_term.pow(2).sum(0) + D_term).sqrt()
+        return loc, scale
 
 
 class AutoDiscreteParallel(AutoGuide):


Test patch:
diff --git a/tests/contrib/autoguide/test_advi.py b/tests/contrib/autoguide/test_advi.py
--- a/tests/contrib/autoguide/test_advi.py
+++ b/tests/contrib/autoguide/test_advi.py
@@ -7,13 +7,18 @@
 import pyro
 import pyro.distributions as dist
 import pyro.poutine as poutine
-from pyro.contrib.autoguide import AutoDiagonalNormal, AutoDiscreteParallel, AutoGuideList, AutoMultivariateNormal
+from pyro.contrib.autoguide import (AutoDelta, AutoDiagonalNormal, AutoDiscreteParallel, AutoGuideList,
+                                    AutoLowRankMultivariateNormal, AutoMultivariateNormal)
 from pyro.infer import SVI, Trace_ELBO, TraceEnum_ELBO, TraceGraph_ELBO
 from pyro.optim import Adam
 from tests.common import assert_equal
 
 
-@pytest.mark.parametrize("auto_class", [AutoMultivariateNormal, AutoDiagonalNormal])
+@pytest.mark.parametrize("auto_class", [
+    AutoDiagonalNormal,
+    AutoMultivariateNormal,
+    AutoLowRankMultivariateNormal,
+])
 def test_scores(auto_class):
     def model():
         pyro.sample("z", dist.Normal(0.0, 1.0))
@@ -32,7 +37,12 @@ def model():
 
 
 @pytest.mark.parametrize("Elbo", [Trace_ELBO, TraceGraph_ELBO, TraceEnum_ELBO])
-@pytest.mark.parametrize("auto_class", [AutoMultivariateNormal, AutoDiagonalNormal])
+@pytest.mark.parametrize("auto_class", [
+    AutoDelta,
+    AutoDiagonalNormal,
+    AutoMultivariateNormal,
+    AutoLowRankMultivariateNormal,
+])
 def test_shapes(auto_class, Elbo):
 
     def model():
@@ -48,7 +58,12 @@ def model():
 
 
 @pytest.mark.xfail(reason="irange is not yet supported")
-@pytest.mark.parametrize('auto_class', [AutoDiagonalNormal, AutoMultivariateNormal])
+@pytest.mark.parametrize('auto_class', [
+    AutoDelta,
+    AutoDiagonalNormal,
+    AutoMultivariateNormal,
+    AutoLowRankMultivariateNormal,
+])
 @pytest.mark.parametrize("Elbo", [Trace_ELBO, TraceGraph_ELBO])
 def test_irange_smoke(auto_class, Elbo):
 
@@ -70,7 +85,12 @@ def model():
     infer.step()
 
 
-@pytest.mark.parametrize("auto_class", [AutoMultivariateNormal, AutoDiagonalNormal])
+@pytest.mark.parametrize("auto_class", [
+    AutoDelta,
+    AutoDiagonalNormal,
+    AutoMultivariateNormal,
+    AutoLowRankMultivariateNormal,
+])
 @pytest.mark.parametrize("Elbo", [Trace_ELBO, TraceGraph_ELBO, TraceEnum_ELBO])
 def test_median(auto_class, Elbo):
 
@@ -80,17 +100,20 @@ def model():
         pyro.sample("z", dist.Beta(2.0, 2.0))
 
     guide = auto_class(model)
-    infer = SVI(model, guide, Adam({'lr': 0.01}), Elbo(strict_enumeration_warning=False))
+    infer = SVI(model, guide, Adam({'lr': 0.05}), Elbo(strict_enumeration_warning=False))
     for _ in range(100):
         infer.step()
 
     median = guide.median()
     assert_equal(median["x"], torch.tensor(0.0), prec=0.1)
-    assert_equal(median["y"], torch.tensor(1.0), prec=0.1)
+    if auto_class is AutoDelta:
+        assert_equal(median["y"], torch.tensor(-1.0).exp(), prec=0.1)
+    else:
+        assert_equal(median["y"], torch.tensor(1.0), prec=0.1)
     assert_equal(median["z"], torch.tensor(0.5), prec=0.1)
 
 
-@pytest.mark.parametrize("auto_class", [AutoMultivariateNormal, AutoDiagonalNormal])
+@pytest.mark.parametrize("auto_class", [AutoDiagonalNormal, AutoMultivariateNormal, AutoLowRankMultivariateNormal])
 @pytest.mark.parametrize("Elbo", [Trace_ELBO, TraceGraph_ELBO, TraceEnum_ELBO])
 def test_quantiles(auto_class, Elbo):
 
@@ -126,7 +149,12 @@ def model():
     assert quantiles["z"][2] < 0.99
 
 
-@pytest.mark.parametrize("continuous_class", [AutoMultivariateNormal, AutoDiagonalNormal])
+@pytest.mark.parametrize("continuous_class", [
+    AutoDelta,
+    AutoDiagonalNormal,
+    AutoMultivariateNormal,
+    AutoLowRankMultivariateNormal,
+])
 def test_discrete_parallel(continuous_class):
     K = 2
     data = torch.tensor([0., 1., 10., 11., 12.])
diff --git a/tests/contrib/autoguide/test_inference.py b/tests/contrib/autoguide/test_inference.py
--- a/tests/contrib/autoguide/test_inference.py
+++ b/tests/contrib/autoguide/test_inference.py
@@ -10,7 +10,7 @@
 import pyro
 import pyro.distributions as dist
 import pyro.optim as optim
-from pyro.contrib.autoguide import AutoDiagonalNormal, AutoMultivariateNormal
+from pyro.contrib.autoguide import AutoDiagonalNormal, AutoLowRankMultivariateNormal, AutoMultivariateNormal
 from pyro.infer import SVI, Trace_ELBO
 from tests.common import assert_equal
 from tests.integration_tests.test_conjugate_gaussian_models import GaussianChain
@@ -71,7 +71,7 @@ def do_test_auto(self, N, reparameterized, n_steps):
                      msg="guide covariance off")
 
 
-@pytest.mark.parametrize('auto_class', [AutoDiagonalNormal, AutoMultivariateNormal])
+@pytest.mark.parametrize('auto_class', [AutoDiagonalNormal, AutoMultivariateNormal, AutoLowRankMultivariateNormal])
 def test_auto_diagonal_gaussians(auto_class):
     n_steps = 3501 if auto_class == AutoDiagonalNormal else 6001
 
@@ -79,7 +79,10 @@ def model():
         pyro.sample("x", dist.Normal(-0.2, 1.2))
         pyro.sample("y", dist.Normal(0.2, 0.7))
 
-    guide = auto_class(model)
+    if auto_class is AutoLowRankMultivariateNormal:
+        guide = auto_class(model, rank=1)
+    else:
+        guide = auto_class(model)
     adam = optim.Adam({"lr": .001, "betas": (0.95, 0.999)})
     svi = SVI(model, guide, adam, loss=Trace_ELBO())
 
@@ -87,26 +90,24 @@ def model():
         loss = svi.step()
         assert np.isfinite(loss), loss
 
-    if auto_class == AutoMultivariateNormal:
-        L = pyro.param("auto_scale_tril")
-        diag_cov = torch.mm(L, L.t()).diag()
-    else:
-        diag_cov = torch.pow(pyro.param("auto_scale"), 2.0)
-
-    assert_equal(pyro.param("auto_loc"), torch.tensor([-0.2, 0.2]), prec=0.05,
+    loc, scale = guide._loc_scale()
+    assert_equal(loc, torch.tensor([-0.2, 0.2]), prec=0.05,
                  msg="guide mean off")
-    assert_equal(diag_cov, torch.tensor([1.44, 0.49]), prec=0.08,
+    assert_equal(scale, torch.tensor([1.21, 0.71]), prec=0.08,
                  msg="guide covariance off")
 
 
-@pytest.mark.parametrize('auto_class', [AutoDiagonalNormal, AutoMultivariateNormal])
+@pytest.mark.parametrize('auto_class', [AutoDiagonalNormal, AutoMultivariateNormal, AutoLowRankMultivariateNormal])
 def test_auto_transform(auto_class):
     n_steps = 3500
 
     def model():
         pyro.sample("x", dist.LogNormal(0.2, 0.7))
 
-    guide = auto_class(model)
+    if auto_class is AutoLowRankMultivariateNormal:
+        guide = auto_class(model, rank=1)
+    else:
+        guide = auto_class(model)
     adam = optim.Adam({"lr": .001, "betas": (0.90, 0.999)})
     svi = SVI(model, guide, adam, loss=Trace_ELBO())
 
@@ -114,19 +115,14 @@ def model():
         loss = svi.step()
         assert np.isfinite(loss), loss
 
-    if auto_class == AutoMultivariateNormal:
-        L = pyro.param("auto_scale_tril")
-        diag_cov = torch.mm(L, L.t()).diag()
-    else:
-        diag_cov = torch.pow(pyro.param("auto_scale"), 2.0)
-
-    assert_equal(pyro.param("auto_loc"), torch.tensor([0.2]), prec=0.04,
+    loc, scale = guide._loc_scale()
+    assert_equal(loc, torch.tensor([0.2]), prec=0.04,
                  msg="guide mean off")
-    assert_equal(diag_cov, torch.tensor([0.49]), prec=0.04,
+    assert_equal(scale, torch.tensor([0.71]), prec=0.04,
                  msg="guide covariance off")
 
 
-@pytest.mark.parametrize('auto_class', [AutoDiagonalNormal, AutoMultivariateNormal])
+@pytest.mark.parametrize('auto_class', [AutoDiagonalNormal, AutoMultivariateNormal, AutoLowRankMultivariateNormal])
 def test_auto_dirichlet(auto_class):
     num_steps = 2000
     prior = torch.tensor([0.5, 1.0, 1.5, 3.0])