Autograd for variance in normal distribution not working

username: nierth
cooked: <p>Hi,</p>
<p>I am currently fiddling around with a PPO implementation where the loss depends on the entropy of a multivariate normal distribution. The entropy is calculated as follows</p>
<pre><code class="lang-auto">class Actor(nn.Module):
    def __init__(self, state_dim, action_dim, config):
        super(Actor, self).__init__()

        self.net = nn.Sequential(
                        nn.Linear(state_dim, 64),
                        nn.ReLU(),
                        nn.Linear(64, 64),
                        nn.ReLU(),
                        nn.Linear(64, action_dim),
                        nn.Tanh())

        self.action_std = torch.nn.Parameter(torch.ones(action_dim)*action_std).to(device)

    def evaluate(self, state, action):
        action_mean = self.net(state)
        dist = torch.distributions.normal.Normal(action_mean, self.action_std)
        action_logprobs = torch.sum(dist.log_prob(action), dim=1)
        dist_entropy = torch.sum(dist.entropy(), dim=1)
        return action_logprobs, dist_entropy
</code></pre>
<p>and the loss also depends on the return value “dist_entropy”. However during the backward pass the parameter “self.action_std” does not change at all. Any clue about the reason?</p>

username: albanD
cooked: <p>Hi,</p>
<p>The problem is that <code>self.action_std</code> is not the Paramter, the <code>.to()</code> op creates a new Tensor.<br>
You should do</p>
<pre><code class="lang-auto">        self.action_std = torch.nn.Parameter(torch.ones(action_dim, device=device)*action_std)
</code></pre>

username: nierth
cooked: <p>Hi,</p>
<p>thanks for the quick reply. Indeed, this modification solved the issue but leaves me a little bit puzzled. Out of curiosity I tried the following:</p>
<pre><code class="lang-auto">a = torch.nn.Parameter(torch.ones([1], device='cpu'))
b = torch.nn.Parameter(torch.ones([1], device='cuda'))
c = torch.nn.Parameter(torch.ones([1])).to('cpu')
d = torch.nn.Parameter(torch.ones([1])).to('cuda')

print(a, end="\n\n")
print(b, end="\n\n")
print(c, end="\n\n")
print(d, end="\n\n")
</code></pre>
<p>which gave the following output</p>
<pre><code class="lang-auto">Parameter containing:
tensor([1.], requires_grad=True)

Parameter containing:
tensor([1.], device='cuda:0', requires_grad=True)

Parameter containing:
tensor([1.], requires_grad=True)

tensor([1.], device='cuda:0', grad_fn=&lt;CopyBackwards&gt;)
</code></pre>
<p>a, b, and c make sense, but what happens to d? From my basic pytorch understanding (and the help of stackoverflow) a and b are straightforward. c would be copied to cpu but as it is already stored there, the .to(‘cpu’) has no effect.</p>
<p>Now to d: My understanding until now was, that  <code>torch.ones([1]) </code> is a “normal” pytorch tensor that becomes a parameter (and thus differentiable) and gets moved to the GPU by the  <code>to('cuda')</code> in the last step. Finally, d points to the address on the GPU. But the print output indicates that something different is going on… Btw, what does  <code>grad_fn=&lt;CopyBackwards&gt; </code> mean?</p>

username: albanD
cooked: <blockquote>
<p>a, b, and c make sense, but what happens to d?</p>
</blockquote>
<p>Actually the special case is <code>c</code> here.<br>
a and b are Parameters because you created them.<br>
d is  just a Tensor with a grad_fn: it is not a leaf anymore because it was created by a differentiable op<br>
c is the special case because the Tensor is already on the cpu, then <code>.to('cpu')</code> is doing nothing and you still get the Parameter back.</p>