Bayesian models for CNN using Pyro

username: Maryam_Osama
cooked: <p>Hi<br>
I am applying a Bayesian model for a CNN that has many layers (more than 3), using Stochastic Variational Inference in Pyro Package.<br>
However after defining the NN, Model and Guide functions and running the training loop I found that the loss stops decreasing on loss ~8000 (which is extremely high). I tried different learning rates and different optimization functions but non of them reaches a loss lower than 8000.<br>
At last I changed the autoguide function (I tried AutoNormal, AutoGaussian, AutoBeta) they all stopped dicreasing the loss at the same point.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/a/1af5ec28fe73dd1259f21d96c87d8a9183e10112.png" data-download-href="https://discuss.pytorch.org/uploads/default/1af5ec28fe73dd1259f21d96c87d8a9183e10112" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1af5ec28fe73dd1259f21d96c87d8a9183e10112_2_690x496.png" alt="image" data-base62-sha1="3QviE7iWaJJlAQCgDiFnJxKZo0q" width="690" height="496" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1af5ec28fe73dd1259f21d96c87d8a9183e10112_2_690x496.png, https://discuss.pytorch.org/uploads/default/original/3X/1/a/1af5ec28fe73dd1259f21d96c87d8a9183e10112.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/a/1af5ec28fe73dd1259f21d96c87d8a9183e10112.png 2x" data-dominant-color="F6F8FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">720×518 36.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The last thing I did is trying the AutoMultivatiateNormal and for this it reached negative values (it reached -1) but when I looked at the weight matrices I found that they are all turned into scalers!!<br>
The following graph represent the loss pattern for the   AutoMultivatiateNormal<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/3/733bed1ddc87d6430a7c40619f89c43a5cba180b.png" data-download-href="https://discuss.pytorch.org/uploads/default/733bed1ddc87d6430a7c40619f89c43a5cba180b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/3/733bed1ddc87d6430a7c40619f89c43a5cba180b_2_669x500.png" alt="image" data-base62-sha1="grpiD0UaDqahNEqOucbj7MKuGxR" width="669" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/3/733bed1ddc87d6430a7c40619f89c43a5cba180b_2_669x500.png, https://discuss.pytorch.org/uploads/default/original/3X/7/3/733bed1ddc87d6430a7c40619f89c43a5cba180b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/3/733bed1ddc87d6430a7c40619f89c43a5cba180b.png 2x" data-dominant-color="C0D7E8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">694×518 40.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>