Repository: pyro-ppl/pyro

Issue:
How to find cause of nan error in ELBO function
I am trying to modify the `vae.py` example, and the ELBO loss function is returning `nan` before I even do any training steps. My `model` and `guide` functions give sensible output, so I have no way to guess where the error is.  Thanks for your help.

I wouldn't typically open a github issue for this kind of question, but I am not sure what the best place to get support for `pyro` is. Thanks!


Discussion:
I'm not sure the best place to debug NaNs, but here's what I'd try:
1. Insert checks like `assert not np.isnan(my_tensor.sum())` into Pyro code.
2. Try using advance Pyro features like
  ```py
  tr = poutine.trace(guide).get_trace()
  tr.log_pdf()
  print('\n'.join('{}: {}'.format(name, site['log_pdf']) for name, site in tr.nodes.items()))
  ```

If you do find a good debugging trick, we'd be happy to add it to Pyro for error checking.

(And yes this is the right place for questions. We're in the process of seting up a gitter and a discuss.)
to add to some of the suggestions:
1) revert your changes to get a working state of the vae, then modify one change at a time and put debugger statements in your guide and model to see exactly where the occurrence of the nan is. 
2) you can inspect the parameters directly in the param store via `pyro.get_param_store()` - you can step through and inspect the computed gradients before a step is taken.  the `TraceELBO` class also provides a `loss()` method you can call to inspect the ELBo without taking a step. make sure those values are what youd expect
unfortunately, as @fritzo alludes to, debugging NaNs is a very ad-hoc process that even the most experienced researchers often get plagued with.
@jpchen I am trying to debug a case when a single evaluation of `svi.loss` gives an error. I have been carefully stepping through with `pdb`, and just discovered that the problem is probably with my `model` . A `nan` warning at  `pyro/infer/trace_elbo.py(85)_get_traces()` would have saved me a bunch of time, and could at least have told me whether the `guide` or `model` is wrong. 
I recommend checking if you have softmax or sigmoids in the loop for
categoricals. They can do that. Also, are you using a Gaussian? Is your std
constrained to be positive?

On Tue, Nov 14, 2017, 9:18 AM Noah D Brenowitz <notifications@github.com>
wrote:

> @jpchen <https://github.com/jpchen> I am trying to debug a case when a
> single evaluation of svi.loss gives an error. I have been carefully
> stepping through with pdb, and just discovered that the problem is
> probably with my model . A nan warning at
> pyro/infer/trace_elbo.py(85)_get_traces() would have saved me a bunch of
> time, and could at least have told me whether the guide or model is wrong.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/uber/pyro/issues/571#issuecomment-344331471>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/ABVhL1wJONnGENwWl8XxxFoSUnC-Qhjnks5s2ctzgaJpZM4Qcgrb>
> .
>


Resolving patch:
diff --git a/pyro/infer/trace_elbo.py b/pyro/infer/trace_elbo.py
--- a/pyro/infer/trace_elbo.py
+++ b/pyro/infer/trace_elbo.py
@@ -1,6 +1,9 @@
 from __future__ import absolute_import, division, print_function
 
 import numbers
+import warnings
+
+import numpy as np
 
 import pyro
 import pyro.poutine as poutine
@@ -118,6 +121,8 @@ def loss(self, model, guide, *args, **kwargs):
             elbo += torch_data_sum(weight * elbo_particle)
 
         loss = -elbo
+        if np.isnan(loss):
+            warnings.warn('Encountered NAN loss')
         return loss
 
     def loss_and_grads(self, model, guide, *args, **kwargs):
@@ -176,5 +181,6 @@ def loss_and_grads(self, model, guide, *args, **kwargs):
                 pyro.get_param_store().mark_params_active(trainable_params)
 
         loss = -elbo
-
+        if np.isnan(loss):
+            warnings.warn('Encountered NAN loss')
         return loss
diff --git a/pyro/infer/tracegraph_elbo.py b/pyro/infer/tracegraph_elbo.py
--- a/pyro/infer/tracegraph_elbo.py
+++ b/pyro/infer/tracegraph_elbo.py
@@ -4,6 +4,7 @@
 from collections import namedtuple
 
 import networkx
+import numpy as np
 import torch
 
 import pyro
@@ -237,9 +238,11 @@ def loss(self, model, guide, *args, **kwargs):
                         elbo_particle += model_trace.nodes[name]["log_pdf"]
                         elbo_particle -= guide_trace.nodes[name]["log_pdf"]
 
-            elbo += weight * elbo_particle.data[0]
+            elbo += torch_data_sum(weight * elbo_particle)
 
         loss = -elbo
+        if np.isnan(loss):
+            warnings.warn('Encountered NAN loss')
         return loss
 
     def loss_and_grads(self, model, guide, *args, **kwargs):
@@ -304,4 +307,6 @@ def _loss_and_grads_particle(self, weight, model_trace, guide_trace):
             pyro.get_param_store().mark_params_active(trainable_params)
 
         loss = -elbo
+        if np.isnan(loss):
+            warnings.warn('Encountered NAN loss')
         return weight * loss
diff --git a/pyro/poutine/trace.py b/pyro/poutine/trace.py
--- a/pyro/poutine/trace.py
+++ b/pyro/poutine/trace.py
@@ -1,8 +1,19 @@
 from __future__ import absolute_import, division, print_function
 
 import collections
+import warnings
 
 import networkx
+import numpy as np
+
+
+def _warn_if_nan(name, variable):
+    value = variable.data[0]
+    if np.isnan(value):
+        warnings.warn("Encountered NAN log_pdf at site '{}'".format(name))
+    if np.isinf(value) and value > 0:
+        warnings.warn("Encountered +inf log_pdf at site '{}'".format(name))
+    # Note that -inf log_pdf is fine: it is merely a zero-probability event.
 
 
 class Trace(networkx.DiGraph):
@@ -70,6 +81,7 @@ def log_pdf(self, site_filter=lambda name, site: True):
                     site_log_p = site["fn"].log_pdf(
                         site["value"], *args, **kwargs) * site["scale"]
                     site["log_pdf"] = site_log_p
+                    _warn_if_nan(name, site_log_p)
                 log_p += site_log_p
         return log_p
 
@@ -91,6 +103,7 @@ def batch_log_pdf(self, site_filter=lambda name, site: True):
                         site["value"], *args, **kwargs) * site["scale"]
                     site["batch_log_pdf"] = site_log_p
                     site["log_pdf"] = site_log_p.sum()
+                    _warn_if_nan(name, site["log_pdf"])
                 # Here log_p may be broadcast to a larger tensor:
                 log_p = log_p + site_log_p
         return log_p
@@ -111,6 +124,7 @@ def compute_batch_log_pdf(self, site_filter=lambda name, site: True):
                         site["value"], *args, **kwargs) * site["scale"]
                     site["batch_log_pdf"] = site_log_p
                     site["log_pdf"] = site_log_p.sum()
+                    _warn_if_nan(name, site["log_pdf"])
 
     @property
     def observation_nodes(self):


Test patch: