internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: pyro-ppl/pyro

Issue:
HMC gives non deterministic results with parallel chaining
Even though the `ParallelSampler` yields the same set of traces (each worker gives deterministic results), the order in which the traces are consumed might be different, which gives different results when we use `TracePredictive`. 

One way to get deterministic results from HMC/NUTS with `num_chains > 1` would be to store intermediate results, and yield results from each of the workers in a round-robin fashion. 


Discussion:

Resolving patch:
diff --git a/pyro/infer/mcmc/mcmc.py b/pyro/infer/mcmc/mcmc.py
--- a/pyro/infer/mcmc/mcmc.py
+++ b/pyro/infer/mcmc/mcmc.py
@@ -8,7 +8,7 @@
 import sys
 import threading
 import warnings
-from collections import OrderedDict
+from collections import OrderedDict, deque
 
 import six
 from six.moves import queue
@@ -144,6 +144,11 @@ def _traces(self, *args, **kwargs):
         # restore original handler
         signal.signal(signal.SIGINT, sigint_handler)
         active_workers = self.num_chains
+        # To yield a deterministic ordering, we hold intermediate traces
+        # from each of the workers in its own queue in `results_buffer`
+        # and yield these in a round robin fashion.
+        buffer_idx = 0
+        results_buffer = [deque() for _ in range(self.num_chains)]
         try:
             for w in self.workers:
                 w.start()
@@ -162,10 +167,22 @@ def _traces(self, *args, **kwargs):
                 if isinstance(val, Exception):
                     # Exception trace is already logged by worker.
                     raise val
-                elif val is not None:
-                    yield val
-                else:
+                if val is None:
                     active_workers -= 1
+                else:
+                    results_buffer[chain_id - 1].append(val)
+                while results_buffer[buffer_idx]:
+                    yield results_buffer[buffer_idx].popleft()
+                    buffer_idx = (buffer_idx + 1) % self.num_chains
+            # empty out the results buffer
+            non_empty_buffers = set(range(self.num_chains))
+            while non_empty_buffers:
+                if results_buffer[buffer_idx]:
+                    yield results_buffer[buffer_idx].popleft()
+                else:
+                    if buffer_idx in non_empty_buffers:
+                        non_empty_buffers.remove(buffer_idx)
+                buffer_idx = (buffer_idx + 1) % self.num_chains
         finally:
             self.terminate()
 


Test patch:
Repository: pyro-ppl/pyro

Issue:
[FR] Add tutorial on implementing new effects
Users at PROBPROG 2018 requested a tutorial on high-level Pyro architecture including advanced features like poutines.  This issue proposes adding a tutorial on implementing a new effect handler.

Whereas #1553 should aim to explain Pyro's architecture in a simplified way, this tutorial should prepare developers to make changes to Pyro.


Discussion:
Hi guys, has there been any progress with this issue? Would been keen to see any current drafts of tutorials and provide feedback. This sort of tutorial really is necessary I think to get the scientific community on board to implement new methodology they are working on (or in general, to encourage and make contributions easier). 
@robsalomone sorry, I've been busy with other things and haven't had time to get to this.  It'll definitely be included with the [0.3.0 release](https://github.com/uber/pyro/milestone/5), which will hopefully be out in a few weeks.  I agree it's important for encouraging contributors and I'll cc you when I open a PR with a draft.

Resolving patch:
diff --git a/pyro/poutine/runtime.py b/pyro/poutine/runtime.py
--- a/pyro/poutine/runtime.py
+++ b/pyro/poutine/runtime.py
@@ -149,7 +149,7 @@ def default_process_message(msg):
     :param msg: a message to be processed
     :returns: None
     """
-    if msg["done"] or msg["is_observed"]:
+    if msg["done"] or msg["is_observed"] or msg["value"] is not None:
         msg["done"] = True
         return msg
 


Test patch:
Repository: pyro-ppl/pyro

Issue:
Silent error when pyro.markov history is too small
## Problem

Higher-order markov processes require setting `pyro.markov(..., history=t)` for some `t>1`. However when setting `t=1`, Pyro does not notice the enumeration dim conflict.

## Proposed solution

During enumeration, if the newly allocated enum dimension conflicts with a `.batch_shape` dim of the distribution, raise an error.

cc @martinjankowiak who experienced this silent error.


Discussion:

Resolving patch:
diff --git a/pyro/util.py b/pyro/util.py
--- a/pyro/util.py
+++ b/pyro/util.py
@@ -261,7 +261,13 @@ def check_site_shape(site, max_plate_nesting):
                 '- .to_event(...) the distribution being sampled',
                 '- .permute() data dimensions']))
 
-    # TODO Check parallel dimensions on the left of max_plate_nesting.
+    # Check parallel dimensions on the left of max_plate_nesting.
+    enum_dim = site["infer"].get("_enumerate_dim")
+    if enum_dim is not None:
+        if len(site["fn"].batch_shape) >= -enum_dim and site["fn"].batch_shape[enum_dim] != 1:
+            raise ValueError('\n  '.join([
+                'Enumeration dim conflict at site "{}"'.format(site["name"]),
+                'Try increasing pyro.markov history size']))
 
 
 def _are_independent(counters1, counters2):


Test patch:
diff --git a/tests/infer/test_valid_models.py b/tests/infer/test_valid_models.py
--- a/tests/infer/test_valid_models.py
+++ b/tests/infer/test_valid_models.py
@@ -1556,6 +1556,31 @@ def guide():
     assert_ok(model, guide, TraceEnum_ELBO(max_plate_nesting=2))
 
 
+@pytest.mark.parametrize('history', [0, 1, 2, 3])
+def test_markov_history(history):
+
+    @config_enumerate
+    def model():
+        p = pyro.param("p", 0.25 * torch.ones(2, 2))
+        q = pyro.param("q", 0.25 * torch.ones(2))
+        x_prev = torch.tensor(0)
+        x_curr = torch.tensor(0)
+        for t in pyro.markov(range(10), history=history):
+            probs = p[x_prev, x_curr]
+            x_prev, x_curr = x_curr, pyro.sample("x_{}".format(t), dist.Bernoulli(probs)).long()
+            pyro.sample("y_{}".format(t), dist.Bernoulli(q[x_curr]),
+                        obs=torch.tensor(0.))
+
+    def guide():
+        pass
+
+    if history < 2:
+        assert_error(model, guide, TraceEnum_ELBO(max_plate_nesting=0),
+                     match="Enumeration dim conflict")
+    else:
+        assert_ok(model, guide, TraceEnum_ELBO(max_plate_nesting=0))
+
+
 def test_mean_field_ok():
 
     def model():
Repository: pyro-ppl/pyro

Issue:
Get error when running lda.py with “pyro.enable_validation(True)” 

As title described, I add "pyro.enable_validation(True)" at the head of script (examples/lda.py), and run with "python lda.py".

I get an error, saying "ValueError: Error while computing log_prob at site 'topic_words':".

However, the original script without "pyro.enable_validation(True)" can run smoothly.

Thanks for suggestion. 

### Environment
 - Centos 7,python 3.6.
 - PyTorch: 1.0.0
 - Pyro version: 0.3.0




Discussion:

Resolving patch:
diff --git a/examples/lda.py b/examples/lda.py
--- a/examples/lda.py
+++ b/examples/lda.py
@@ -71,6 +71,7 @@ def make_predictor(args):
         layer.bias.data.normal_(0, 0.001)
         layers.append(layer)
         layers.append(nn.Sigmoid())
+    layers.append(nn.Softmax(dim=-1))
     return nn.Sequential(*layers)
 
 
@@ -78,12 +79,12 @@ def parametrized_guide(predictor, data, args, batch_size=None):
     # Use a conjugate guide for global variables.
     topic_weights_posterior = pyro.param(
             "topic_weights_posterior",
-            lambda: torch.ones(args.num_topics) / args.num_topics,
+            lambda: torch.ones(args.num_topics),
             constraint=constraints.positive)
     topic_words_posterior = pyro.param(
             "topic_words_posterior",
-            lambda: torch.ones(args.num_topics, args.num_words) / args.num_words,
-            constraint=constraints.positive)
+            lambda: torch.ones(args.num_topics, args.num_words),
+            constraint=constraints.greater_than(0.5))
     with pyro.plate("topics", args.num_topics):
         pyro.sample("topic_weights", dist.Gamma(topic_weights_posterior, 1.))
         pyro.sample("topic_words", dist.Dirichlet(topic_words_posterior))
@@ -105,6 +106,9 @@ def parametrized_guide(predictor, data, args, batch_size=None):
 def main(args):
     logging.info('Generating data')
     pyro.set_rng_seed(0)
+    pyro.clear_param_store()
+    pyro.enable_validation(True)
+
     # We can generate synthetic data directly by calling the model.
     true_topic_weights, true_topic_words, data = model(args=args)
 


Test patch:
Repository: pyro-ppl/pyro

Issue:
make test: no attribute 'optim' error in 'examples/contrib/oed/ab_test.py' 
### Issue Description
On the latest dev branch, `make test` gives the following error:

_
examples/contrib/oed/ab_test.py:12: in <module>
    from gp_bayes_opt import GPBayesOptimizer
examples/contrib/oed/gp_bayes_opt.py:11: in <module>
    class GPBayesOptimizer(pyro.optim.multi.MultiOptimizer):
E   AttributeError: module 'pyro' has no attribute 'optim'

### Environment
For any bugs, please provide the following:
 - OS and python version: CentOS Linux 7 (Core); Python 3.7.1
 - PyTorch version, or if relevant, output of `pip freeze`: PyTorch 1.0.0
 - Pyro version: output of `python -c 'import pyro; print pyro.__version__'`: pyro 0.3.0+9adbdb7

### Code Snippet

```
make install
make format
make test
```



Discussion:
That's weird, my guess is that `make` isn't picking the right conda env / virtual env or something, so that `make test` picks up an outdated pyro version. Can you try to repeatedly
```sh
pip uninstall pyro-ppl
```
You can test locally more quickly using
```sh
pytest -v tests/test_examples.py -k ab
```

Resolving patch:
diff --git a/examples/contrib/oed/gp_bayes_opt.py b/examples/contrib/oed/gp_bayes_opt.py
--- a/examples/contrib/oed/gp_bayes_opt.py
+++ b/examples/contrib/oed/gp_bayes_opt.py
@@ -3,9 +3,9 @@
 import torch.optim as optim
 from torch.distributions import transform_to
 
-import pyro
 import pyro.contrib.gp as gp
 from pyro.infer import TraceEnum_ELBO
+import pyro.optim
 
 
 class GPBayesOptimizer(pyro.optim.multi.MultiOptimizer):


Test patch:
Repository: pyro-ppl/pyro

Issue:
pip install pyro-ppl failing in Python 3
```
$ conda create -n test3 python=3
...
$ source activate test3
$ pip install pyro-ppl
Collecting pyro-ppl
  Downloading https://files.pythonhosted.org/packages/75/0e/e6013ad748349e447c6da7ba0b3d66950e834a5b45a0ff426c131b00cde5/pyro-ppl-0.3.1.tar.gz (207kB)
    100% |████████████████████████████████| 215kB 4.9MB/s
    Complete output from command python setup.py egg_info:
    fatal: Not a git repository (or any of the parent directories): .git
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/c5/cqn89jt900v5tmm2y7p0tf4r0000gn/T/pip-install-6feboagj/pyro-ppl/setup.py", line 25, in <module>
        cwd=PROJECT_PATH).decode('ascii').strip()
      File "/Users/fritzobermeyer/miniconda2/envs/test3/lib/python3.7/subprocess.py", line 395, in check_output
        **kwargs).stdout
      File "/Users/fritzobermeyer/miniconda2/envs/test3/lib/python3.7/subprocess.py", line 487, in run
        output=stdout, stderr=stderr)
    subprocess.CalledProcessError: Command '['git', 'tag', '--points-at', 'HEAD']' returned non-zero exit status 128.

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/c5/cqn89jt900v5tmm2y7p0tf4r0000gn/T/pip-install-6feboagj/pyro-ppl/
```


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -27,8 +27,9 @@
     if not current_tag == version:
         commit_sha = subprocess.check_output(['git', 'rev-parse', '--short', 'HEAD'],
                                              cwd=PROJECT_PATH).decode('ascii').strip()
-except OSError:
-    pass
+# catch all exception to be safe
+except Exception:
+    pass  # probably not a git repo
 
 # Write version to _version.py
 if commit_sha:
@@ -44,7 +45,7 @@
     import pypandoc
     long_description = pypandoc.convert('README.md', 'rst')
     print(long_description)
-except (IOError, ImportError, OSError) as e:
+except Exception as e:
     sys.stderr.write('Failed to convert README.md to rst:\n  {}\n'.format(e))
     sys.stderr.flush()
     long_description = open('README.md').read()


Test patch:
Repository: pyro-ppl/pyro

Issue:
LDA tutorial: incorrect tensor shape when document size is larger than the vocabulary size
https://github.com/pyro-ppl/pyro/blob/56c06176763f1676bd586c89b2dc503c64315f0a/examples/lda.py#L101

The filler array of ones should have expanded into the size of the sliced data. This is problematic when the number of words per document is larger than the vocabulary size
Try 
`counts.scatter_add_(0, data[:, ind], torch.tensor(1.).expand(data[:,ind].shape))`
instead


Discussion:
Thanks @curtis999 for the report and fix! Do you have a simple example we can add as a regression test? It can be totally random data.
@fritzo the error can be replicated simply by passing `-w 63` or whatever that sets the vocabulary size to be smaller than the total number of words in a document.

In the example, the data is from the generative model, or we can explicatly generate it with 
```
theta = dist.Dirichlet(torch.zeros([args.num_docs, args.num_topics]) + 0.2).sample()
phi = dist.Dirichlet(torch.zeros([args.num_topics, args.num_words]) + 0.1).sample()
z = dist.Categorical(theta).expand([args.num_words_per_doc, args.num_docs]).sample()
data = dist.Categorical(phi[z]).sample()
```
 

Resolving patch:
diff --git a/examples/lda.py b/examples/lda.py
--- a/examples/lda.py
+++ b/examples/lda.py
@@ -92,13 +92,11 @@ def parametrized_guide(predictor, data, args, batch_size=None):
     # Use an amortized guide for local variables.
     pyro.module("predictor", predictor)
     with pyro.plate("documents", args.num_docs, batch_size) as ind:
+        data = data[:, ind]
         # The neural network will operate on histograms rather than word
         # index vectors, so we'll convert the raw data to a histogram.
-        if torch._C._get_tracing_state():
-            counts = torch.eye(1024)[data[:, ind]].sum(0).t()
-        else:
-            counts = torch.zeros(args.num_words, ind.size(0))
-            counts.scatter_add_(0, data[:, ind], torch.tensor(1.).expand(counts.shape))
+        counts = (torch.zeros(args.num_words, ind.size(0))
+                       .scatter_add(0, data, torch.ones(data.shape)))
         doc_topics = predictor(counts.transpose(0, 1))
         pyro.sample("doc_topics", dist.Delta(doc_topics, event_dim=1))
 


Test patch:
diff --git a/tests/test_examples.py b/tests/test_examples.py
--- a/tests/test_examples.py
+++ b/tests/test_examples.py
@@ -116,7 +116,7 @@ def xfail_jit(*args):
     'hmm.py --num-steps=1 --truncate=10 --model=3 --jit',
     'hmm.py --num-steps=1 --truncate=10 --model=4 --jit',
     'hmm.py --num-steps=1 --truncate=10 --model=5 --jit',
-    xfail_jit('lda.py --num-steps=2 --num-words=100 --num-docs=100 --num-words-per-doc=8 --jit'),
+    'lda.py --num-steps=2 --num-words=100 --num-docs=100 --num-words-per-doc=8 --jit',
     'minipyro.py --backend=pyro --jit',
     'minipyro.py --jit',
     xfail_jit('vae/ss_vae_M2.py --num-epochs=1 --aux-loss --jit'),
Repository: pyro-ppl/pyro

Issue:
Cannot delete constrained parameter [bug]
### **Issue Description**
Deleting a constrained parameter throws an error.
In the function `param_store.__delitem__()`, the line 
`unconstrained_value = constrained_value.unconstrained()`
throws
`AttributeError: 'Tensor' object has no attribute 'unconstrained'`

### **Environment**
OS: Windows 8.1
Python Version: 3.6.8
Pytorch Version: 1.1.0
Pyro Version: 0.3.3

This error was also present using Pytorch 1.0 and Pyro 0.3.1.
### **Code Snippet**
```py
import torch
import pyro
from torch.distributions import constraints

param_store = pyro.get_param_store()
a = pyro.param('a', torch.ones(3))
print(param_store.keys())  #dict_keys(['a'])
param_store.__delitem__('a')  #Works fine
print(param_store.keys())  #dict_keys([])
b = pyro.param('b', torch.ones(3), constraint=constraints.positive)
print(param_store.keys())  #dict_keys(['b'])
param_store.__delitem__('b')  #AttributeError: 'Tensor' object has no attribute 'unconstrained'
```


Discussion:
Thanks for the clear failing example @NoAxioms ! I'll take a look.

Resolving patch:
diff --git a/pyro/params/param_store.py b/pyro/params/param_store.py
--- a/pyro/params/param_store.py
+++ b/pyro/params/param_store.py
@@ -89,8 +89,7 @@ def __delitem__(self, name):
         """
         Remove a parameter from the param store.
         """
-        constrained_value = self._params.pop(name)
-        unconstrained_value = constrained_value.unconstrained()
+        unconstrained_value = self._params.pop(name)
         self._param_to_name.pop(unconstrained_value)
         self._constraints.pop(name)
 


Test patch:
diff --git a/tests/params/test_param.py b/tests/params/test_param.py
--- a/tests/params/test_param.py
+++ b/tests/params/test_param.py
@@ -123,3 +123,13 @@ def test_dict_interface():
     assert param_store['y'].shape == (4, 5)
     assert_equal(param_store.setdefault('y', torch.zeros(4, 5)), torch.ones(4, 5))
     assert_equal(param_store['y'].unconstrained(), torch.zeros(4, 5))
+
+    # remove y
+    del param_store['y']
+    assert not param_store
+    assert len(param_store) == 0
+    assert 'x' not in param_store
+    assert 'y' not in param_store
+    assert list(param_store.keys()) == []
+    assert list(key for key, value in param_store.items()) == []
+    assert len(list(param_store.values())) == 0
Repository: pyro-ppl/pyro

Issue:
Cholesky issue in HMC
In PyTorch, if Cholesky issue happens, a Runtime Error will be raised (instead of returning `nan`) regarding singular matrix. So it will be difficult to run MCMC with gaussian process kernels (which involves Cholesky decomposition of covariance matrices). This issue is raised in https://github.com/pyro-ppl/pyro/issues/1863 and #1973. Because `nan` is a valid potential energy in HMC, we should add a mechanism to catch this error.

My proposed solution is to add a try, catch exception to potential function so that when Cholesky issue happens, it will return `nan`.


Discussion:

Resolving patch:
diff --git a/pyro/ops/integrator.py b/pyro/ops/integrator.py
--- a/pyro/ops/integrator.py
+++ b/pyro/ops/integrator.py
@@ -70,7 +70,16 @@ def potential_grad(potential_fn, z):
     z_keys, z_nodes = zip(*z.items())
     for node in z_nodes:
         node.requires_grad_(True)
-    potential_energy = potential_fn(z)
+    try:
+        potential_energy = potential_fn(z)
+    # deal with singular matrices
+    except RuntimeError as e:
+        if "singular U" in str(e):
+            grads = {k: v.new_zeros(v.shape) for k, v in z.items()}
+            return grads, z_nodes[0].new_tensor(float('nan'))
+        else:
+            raise e
+
     grads = grad(potential_energy, z_nodes)
     for node in z_nodes:
         node.requires_grad_(False)


Test patch:
diff --git a/tests/infer/mcmc/test_hmc.py b/tests/infer/mcmc/test_hmc.py
--- a/tests/infer/mcmc/test_hmc.py
+++ b/tests/infer/mcmc/test_hmc.py
@@ -297,3 +297,23 @@ def potential_energy(params):
     posterior = torch.stack([sample["z"] for sample in posterior])
     assert_close(torch.mean(posterior), true_mean, rtol=0.05)
     assert_close(torch.std(posterior), true_std, rtol=0.05)
+
+
+@pytest.mark.parametrize('jit', [False, mark_jit(True)], ids=jit_idfn)
+@pytest.mark.parametrize('op', [torch.inverse, torch.cholesky])
+def test_singular_matrix_catch(jit, op):
+    def potential_energy(z):
+        return op(z['cov']).sum()
+
+    init_params = {'cov': torch.eye(3)}
+    potential_fn = potential_energy if not jit else torch.jit.trace(potential_energy, init_params)
+    hmc_kernel = HMC(potential_fn=potential_fn, adapt_step_size=False,
+                     num_steps=10, step_size=1e-20)
+    hmc_kernel.initial_params = init_params
+    hmc_kernel.setup(warmup_steps=0)
+    # setup an invalid cache to trigger singular error for torch.inverse
+    hmc_kernel._cache({'cov': torch.ones(3, 3)}, torch.tensor(0.), {'cov': torch.zeros(3, 3)})
+
+    samples = init_params
+    for i in range(10):
+        samples = hmc_kernel.sample(samples)
Repository: pyro-ppl/pyro

Issue:
Student T tests fail on GPU [bug]
```self = MultivariateStudentT(df: 1.5, loc: torch.Size([2]), scale_tril: torch.Size([2, 2]))
sample_shape = torch.Size([])

    def rsample(self, sample_shape=torch.Size()):
        shape = self._extended_shape(sample_shape)
        X = torch.empty(shape, dtype=self.df.dtype, device=self.df.device).normal_()
        Z = self._chi2.rsample(sample_shape)
        Y = X * torch.rsqrt(Z / self.df).unsqueeze(-1)
>       return self.loc + self.scale_tril.matmul(Y.unsqueeze(-1)).squeeze(-1)
E       RuntimeError: Expected object of device type cuda but got device type cpu for argument #2 'mat2' in call to _th_mm

pyro/distributions/multivariate_studentt.py:74: RuntimeError
```

This issue was discussed in #2226 - running `make test` on the dev branch errors out for me if running on a machine with cuda. I am guessing this hasn't shown up in the CI because it uses a cpu only machine. 

I think this bug is pretty simple - it happens because, as we can see in the above snippet, y inherits its device from self.df, and in the fixture, self.df is set to a scalar value. This is not converted into a tensor by the tensors_default_to context manager, and so isn't sent to the gpu. 

I fixed this in #2226 by changing the fixture, but @fritzo suggested that it might suggest a missing coercion rather than a change to the fixture, so that change in the PR was reverted and I am opening this issue instead.



Discussion:
We really do want to test with Python scalars as inputs. This appears to be a bug in `MultivariateStudentT` which incorrectly coerces `df` without examining the device of other tenors:
```py
df, = broadcast_all(df)  # this line is buggy
```

Resolving patch:
diff --git a/pyro/distributions/multivariate_studentt.py b/pyro/distributions/multivariate_studentt.py
--- a/pyro/distributions/multivariate_studentt.py
+++ b/pyro/distributions/multivariate_studentt.py
@@ -2,7 +2,7 @@
 
 import torch
 from torch.distributions import constraints
-from torch.distributions.utils import broadcast_all, lazy_property
+from torch.distributions.utils import lazy_property
 
 from pyro.distributions.torch import Chi2
 from pyro.distributions.torch_distribution import TorchDistribution
@@ -28,7 +28,8 @@ class MultivariateStudentT(TorchDistribution):
     def __init__(self, df, loc, scale_tril, validate_args=None):
         dim = loc.size(-1)
         assert scale_tril.shape[-2:] == (dim, dim)
-        df, = broadcast_all(df)
+        if not isinstance(df, torch.Tensor):
+            df = loc.new_tensor(df)
         batch_shape = broadcast_shape(df.shape, loc.shape[:-1], scale_tril.shape[:-2])
         event_shape = (dim,)
         self.df = df.expand(batch_shape)


Test patch: