Parametrized Gaussian shared same trainable parameter

username: iamctR
cooked: <p>Hi all, I want to parametrize two Gaussian distributions, but their parameters are related. But it seems like that the variable will be freed during the training. What can I do to fix this problem?<br>
Here is the minimum code. Thank you!</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch.distributions as D
class GaussianModel(nn.Module):
    def __init__(self):
        super(GaussianModel, self).__init__()
        self.v1 = nn.Parameter(torch.zeros(1))
        self.v2 = self.v1*2
        self.g1=D.Normal(torch.tensor([4.0]), self.v1)
        self.g2=D.Normal(torch.tensor([4.0]), self.v2)
    def forward(self,x):
        return torch.mean(self.g1.log_prob(x)+self.g2.log_prob(x))
model=GaussianModel()
opt=torch.optim.SGD(params=model.parameters(), lr=1e-3)
for i in range(10):
    opt.zero_grad()
    x=torch.randn(10)
    loss=model(x)
    loss.backward()
    opt.step()
</code></pre>
<p>And this is the message from terminal:</p>
<pre><code class="lang-auto">RuntimeErrorTraceback (most recent call last)
&lt;ipython-input-92-46e3175cfeff&gt; in &lt;module&gt;
      3     x=torch.randn(10)
      4     loss=model(x)
----&gt; 5     loss.backward()
      6     opt.step()
      7 #     model.p1.detach()

~/.virtualenvs/ddp-pytorch/lib/python3.6/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    219                 retain_graph=retain_graph,
    220                 create_graph=create_graph)
--&gt; 221         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    222 
    223     def register_hook(self, hook):

~/.virtualenvs/ddp-pytorch/lib/python3.6/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
    130     Variable._execution_engine.run_backward(
    131         tensors, grad_tensors_, retain_graph, create_graph,
--&gt; 132         allow_unreachable=True)  # allow_unreachable flag
    133 
    134 

RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time.

</code></pre>

username: KFrank
cooked: <p>Hi Iam!</p>
<aside class="quote no-group" data-username="iamctR" data-post="1" data-topic="127102" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/iamctr/48/40412_2.png" class="avatar"> iamctR:</div>
<blockquote>
<pre><code class="lang-auto">class GaussianModel(nn.Module):
    def __init__(self):
        super(GaussianModel, self).__init__()
        self.v1 = nn.Parameter(torch.zeros(1))
        self.v2 = self.v1*2
        self.g1=D.Normal(torch.tensor([4.0]), self.v1)
        self.g2=D.Normal(torch.tensor([4.0]), self.v2)
    def forward(self,x):
        return torch.mean(self.g1.log_prob(x)+self.g2.log_prob(x))
</code></pre>
</blockquote>
</aside>
<p>The problem is that <code>self.v2 = self.v1*2</code> creates a new tensor<br>
for <code>v2</code>, and does so only once at <code>__init__()</code> time.  Subsequent<br>
updates to <code>v1</code> (made by the optimizer) never get reflected in <code>v2</code><br>
(nor in <code>g2</code>).</p>
<p>(I don’t really understand how this leads to the specific autograd<br>
<code>RuntimeError</code> you see, but this is the key error in your code.)</p>
<p>I would probably just do:</p>
<pre data-code-wrap="python"><code class="lang-python">class GaussianModel (nn.Module):
    def __init__ (self):
        super (GaussianModel, self).__init__()
        self.v1 = nn.Parameter (torch.zeros (1))
    def forward (self, x):
        return torch.mean (D.Normal (torch.tensor ([4.0]), self.v1).log_prob (x) + D.Normal (torch.tensor ([4.0]), 2 * self.v1).log_prob (x))
</code></pre>
<p>I believe you could also implement this as:</p>
<pre data-code-wrap="python"><code class="lang-python">class GaussianModel (nn.Module):
    def __init__ (self):
        super (GaussianModel, self).__init__()
        self.v1 = nn.Parameter (torch.zeros (1))
        self.v2 = self.v1 * 2   # these initial values will be overwritten
        self.g1=D.Normal (torch.tensor ([4.0]), self.v1)
        self.g2=D.Normal (torch.tensor ([4.0]), self.v2)
    def forward (self, x):
        self.v2.copy_ (self.v1 * 2)
        return torch.mean (self.g1.log_prob (x)+self.g2.log_prob (x))
</code></pre>
<p>but, to me, my first version is more readable.</p>
<p>As an aside, you are initializing <code>g1</code> and <code>g2</code> with a <code>scale</code> (standard<br>
deviation) of <code>0.0</code>.  I would expect this to lead to <code>nan</code>s in your first<br>
forward pass.  Up-to-date versions of pytorch will flag this:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; d = torch.distributions.Normal (torch.tensor ([4.0]), torch.zeros (1), validate_args = False)
&gt;&gt;&gt; d
Normal(loc: tensor([4.]), scale: tensor([0.]))
&gt;&gt;&gt; d.log_prob (torch.tensor ([1.0]))
tensor([nan])
&gt;&gt;&gt; d = torch.distributions.Normal (torch.tensor ([4.0]), torch.zeros (1))
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "C:\Users\LisaBrown\Documents\admin\programs\Miniconda3\lib\site-packages\torch\distributions\normal.py", line 50, in __init__
    super(Normal, self).__init__(batch_shape, validate_args=validate_args)
  File "C:\Users\LisaBrown\Documents\admin\programs\Miniconda3\lib\site-packages\torch\distributions\distribution.py", line 53, in __init__
    raise ValueError("The parameter {} has invalid values".format(param))
ValueError: The parameter scale has invalid values
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: iamctR
cooked: <p>Thank you Frank! It works!</p>