Inplace operation I cannot find

lelouedec: <p>hi,<br>
I am implementing a network using resnet an some fancy stuff upon it, but I have an inplce operation on a variable in it, but can’t put my hand on it. Any idea where it is ?</p>
<blockquote>
<p>class refinet(nn.Module):<br>
def <strong>init</strong>(self,num_classes):<br>
super(refinet, self).<strong>init</strong>()<br>
resnet = models.resnet18(pretrained=True)</p>
<pre><code>    self.layer1 = nn.Sequential(
        resnet.conv1,
        resnet.bn1,
        resnet.relu,
        resnet.maxpool,
        resnet.layer1
    )

    self.layer2 = resnet.layer2
    self.layer3 = resnet.layer3
    self.layer4 = resnet.layer4
    
    self.rcu1 = torch.nn.Sequential(
        nn.ReLU(inplace=True),
        conv3x3(512, 512, stride=1),
        nn.ReLU(inplace=True),
        conv3x3(512, 512, stride=1)
    )
    self.rcu2 = torch.nn.Sequential(
        nn.ReLU(inplace=True),
        conv3x3(256, 256, stride=1),
        nn.ReLU(inplace=True),
        conv3x3(256, 256, stride=1)
    )
    self.rcu3 = torch.nn.Sequential(
        nn.ReLU(inplace=True),
        conv3x3(128, 128, stride=1),
        nn.ReLU(inplace=True),
        conv3x3(128, 128, stride=1)
    )
    self.rcu4 = torch.nn.Sequential(
        nn.ReLU(inplace=True),
        conv3x3(64, 64, stride=1),
        nn.ReLU(inplace=True),
        conv3x3(64, 64, stride=1)
    )
    self.multires = torch.nn.Sequential(
        nn.Conv2d(512, 256, kernel_size=3,stride=1, padding=1, bias=False),
        nn.Upsample(scale_factor=2, mode='bilinear')
    )
    self.multires2 = torch.nn.Sequential(
        nn.Conv2d(256, 256, kernel_size=3,stride=1, padding=1, bias=False),
        nn.Upsample(scale_factor=2, mode='bilinear')
    )
    self.multires3 = torch.nn.Sequential(
        nn.Conv2d(256, 128, kernel_size=3,stride=1, padding=1, bias=False),
        nn.Upsample(scale_factor=2, mode='bilinear')
    )
    self.multires4 = torch.nn.Sequential(
        nn.Conv2d(128, 64, kernel_size=3,stride=1, padding=1, bias=False),
        nn.Upsample(scale_factor=2, mode='bilinear')
    )
    self.multires_end = torch.nn.Sequential(
        nn.Conv2d(64, 1,  kernel_size=3, stride=1, padding=1,bias=False),
        nn.Upsample(scale_factor=4, mode='bilinear')
    )
    self.res1 = torch.nn.Sequential(
        nn.MaxPool2d(kernel_size=5, stride=1, padding=2),
        nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1, bias=False)
    )
    self.res2 = torch.nn.Sequential(
        nn.MaxPool2d(kernel_size=5, stride=1, padding=2),
        nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1, bias=False)
    )
    self.res3 = torch.nn.Sequential(
        nn.MaxPool2d(kernel_size=5, stride=1, padding=2),
        nn.Conv2d(128, 128, kernel_size=3, stride=1, padding=1, bias=False)
    )
    self.res4 = torch.nn.Sequential(
        nn.MaxPool2d(kernel_size=5, stride=1, padding=2),
        nn.Conv2d(64, 64, kernel_size=3, stride=1, padding=1, bias=False)
    )
    self.relu = nn.ReLU(inplace=True)
    self.m = nn.Sigmoid()
    
def respool1(self,inputs):
    inputs = self.relu(inputs)
    for i in range(0,4):
        output = self.res1(inputs)
        inputs = inputs + output
    return inputs

def respool2(self,inputs):
    inputs = self.relu(inputs)
    for i in range(0,4):
        output = self.res2(inputs)
        inputs = inputs + output
    return inputs

def respool3(self,inputs):
    inputs = self.relu(inputs)
    for i in range(0,4):
        output = self.res3(inputs)
        inputs = inputs + output
    return inputs

def respool4(self,inputs):
    inputs = self.relu(inputs)
    for i in range(0,4):
        output = self.res4(inputs)
        inputs = inputs + output
    return inputs
    
    
def forward(self,x):
    x1 = self.layer1(x)
    x2 = self.layer2(x1)
    x3 = self.layer3(x2)
    x4 = self.layer4(x3)
    
    out = x4 + self.rcu1(x4) #firststep
    out = out + self.rcu1(out)
    out = self.respool1(out)
    out = out + self.rcu1(out)
    
    out2 = x3 + self.rcu2(x3)#secondstep
    out2 = out2 + self.rcu2(out2)
    out = self.multires(out) + out2
    out = self.respool2(out)
    out = out + self.rcu2(out)
    
    out2 = x2 + self.rcu3(x2)#thirdstep
    out2 = out2 + self.rcu3(out2)
    out = self.multires3(out) + out2
    out = self.respool3(out)
    out = out + self.rcu3(out)
    
    out2 = x1 + self.rcu4(x1)#thirdstep
    out2 = out2 + self.rcu4(out2)
    out = self.multires4(out) + out2
    out = self.respool4(out)
    out = out + self.rcu4(out)
    
    out = self.multires_end(out)
    out = self.m(out)
    
    return out
</code></pre>
</blockquote>

albanD: <p>You have a lot of <code>nn.ReLU(inplace=True)</code> <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"> I don’t see any other inplace operation.</p>

lelouedec: <p>I thought about that but even without any relu inplace still the same error <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation</p>
</blockquote>

albanD: <p>There are no other inplace operations here.<br>
I would double check that I didn’t missed any inplace relu or some other code that does inplace operation, If that doesn’t solve it, a good way to debug this is to remove part of the net, and add a <code>.sum().backward()</code> in the middle, to check that everything above is correct, or use <code>.detach()</code> in the middle (that will prevent backprop from going above) to check that the bottom part of you net is correct. (Here top means where the input is, and bottom is where the output is <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=5" title=":smiley:" class="emoji" alt=":smiley:"> ).</p>

lelouedec: <p>mhh yeah good idea <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"> thanks a lot <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

lelouedec: <p>I tried to detach, lets say this is my network:</p>
<blockquote>
<p>out4 = x1 + self.rcu4(x1)<span class="hashtag-raw">#fourthstep</span><br>
out4 += self.rcu4(out4)<br>
out4 += out3<br>
out4 = self.respool4(out4)<br>
out4 += self.rcu4(out4)<br>
out4 = self.multires_end(out4)<br>
out4.detach()</p>
<pre><code>    out4 = self.m(out4)
    return out4
</code></pre>
</blockquote>
<p>I still have an error :</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation</p>
</blockquote>

albanD: <p>Then it is possibly in your loss function? Or whatever you do with the output of this network?</p>

lelouedec: <p>I found at least one reason for the moment  :</p>
<blockquote>
<p>out4+= something</p>
</blockquote>
<p>breaks the computational graph<br>
while:</p>
<blockquote>
<p>out4 = out4 + something</p>
</blockquote>
<p>doesn’t</p>

albanD: <p>It depends where <code>out4</code> is used. Changing data inplace is tricky because the original value is often needed to compute the backward pass for other operations.<br>
In python <code>+=</code> is an inplace operation, <code>=</code> is not.</p>

lelouedec: <p>this explains that <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"> thanks for the help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

raz1313: <p>it’s the last Sigmoid rectification…</p>
<pre><code class="lang-auto">self.m = nn.Sigmoid()
: 
: 
out = self.m(out)
</code></pre>
<p>I have not found a solution to this. It seems that using a Sigmoid activation on the outpout layer (so that the output remains between 0 and 1) causes a problem with the auto differentiation during backward propagation.<br>
Sigmoid() works just fine before the output layer!  And there is no “inplace=False” flag!<br>
Did you ever find a solution??</p>

dlzzk: <p>Did you find a solution to this? <a class="mention" href="/u/raz1313">@raz1313</a></p>