Manually applied gradient to neural network raises Error: Can't detach views in-place

username: ElleryL
cooked: <p>Hi; I’m working on gradient estimator; so I need to implement model that requires manually compute and apply gradients</p>
<p>here is my current script</p>
<pre><code class="lang-auto">from itertools import chain
net1 = Net1() # neural network
net2 = Net2() # neural network

opt = Adam(list(net1.parameters()) + list(net2.parameters()), lr=1e-3)

params = chain(net1.parameters(), net2.parameters())

for _ in range(epoches):
    opt.zero_grad()  # at second iterations, line raises error
    grad = torch.autograd.grad(loss, params)

    for i,p in enumerate(params):
        p.grad = grad[I]
   
    opt.step()
    
</code></pre>
<p>At second iterations,</p>
<p><code>opt.zero_grad()</code></p>
<p>leads to</p>
<pre><code class="lang-auto">/anaconda3/lib/python3.7/site-packages/torch/optim/optimizer.py in zero_grad(self)
    161             for p in group['params']:
    162                 if p.grad is not None:
--&gt; 163                     p.grad.detach_()
    164                     p.grad.zero_()
    165 
</code></pre>
<p><code>RuntimeError: Can't detach views in-place. Use detach() instead</code></p>
<p>I couldn’t know where does my code go wrong here. Any help ?</p>
<p>Here is a short script of simple model</p>
<pre><code class="lang-auto">class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(1,4)
        self.fc2 = nn.Linear(4,1)
    def forward(self,x):
        h = self.fc1(x)
        h = self.fc2(nn.ReLU()(h))
        return h


# create data
x = torch.torch.randn(20,1)
y = 1.2 * x ** 2 - 1
loss_fn = lambda y_hat,y: (y_hat - y).pow(2).sum()


net = Net()
optimizer = optim.SGD(net.parameters(), lr=1e-3)

Loss = []
for _ in range(1000):
    optimizer.zero_grad()
    y_hat = net(x)
    loss = loss_fn(y_hat,y)
    grad = torch.autograd.grad(loss, net.parameters())
    for i,p in enumerate(net.parameters()):
        p.grad = grad[i]
        print(p.grad)
    optimizer.step()
    Loss.append(loss.item())
</code></pre>
<p>Thanks in advance</p>

username: ptrblck
cooked: <p>Could you post the definition of the models so that we can reproduce this issue?</p>

username: ElleryL
cooked: <p>Thanks for the reply; I’ve attached a simple example that raises the error and my pytorch version is <code>1.1.0</code></p>

username: ptrblck
cooked: <p>Thanks for the code.<br>
Could you <code>clone</code> the <code>grad</code>, so that you won’t create a view of this tensor?</p>
<pre><code class="lang-python">p.grad = grad[i].clone()
</code></pre>

username: ElleryL
cooked: <p>Thanks ! It solves the problem!</p>

username: Ashima_Garg
cooked: <p>Hi, I was also facing the problem of  <code>RuntimeError: Can't detach views in-place. Use detach() instead</code>  with  <code>x.detach_()</code><br>
It got resolved using both  <code>x = x.detach()</code>  and  <code>x = x.contiguous().detach_()</code> .</p>
<p>But still, I don’t exactly understand the differences between these three lines? And when is the  <code>view</code>  created of a variable?</p>
<p>Thanks. Any help would really be appreciated!</p>

username: hughperkins
cooked: <p>wow, thanks Patrick. Would never have figured that out… Awesome <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>