One Error: inplace operation

username: Cverlpeng
cooked: <p>I get one error:<br>
one of the variables needed for gradient computation has been modified by an inplace operation<br>
thanks<br>
my model:</p>
<pre><code class="lang-python">class SCNN(nn.Module):
    def __init__(self, in_ch, out_ch):
        super(SCNN, self).__init__()
        self.down = nn.ModuleList(
            [nn.Conv2d(in_ch, out_ch, kernel_size=(1, 9), stride=1, padding=(0, 4)) for i in range(35)])
        self.up = nn.ModuleList(
            [nn.Conv2d(in_ch, out_ch, kernel_size=(1, 9), stride=1, padding=(0, 4)) for i in range(36)])
        self.left = nn.ModuleList(
            [nn.Conv2d(in_ch, out_ch, kernel_size=(9, 1), stride=1, padding=(4, 0)) for i in range(99)])
        self.right = nn.ModuleList(
            [nn.Conv2d(in_ch, out_ch, kernel_size=(9, 1), stride=1, padding=(4, 0)) for i in range(100)])

    def forward(self, x):
        h = x.shape[2]
        w = x.shape[3]
        for i in range(h - 1):
            x_h = F.relu(self.down[i](x[:, :, [i], :]), inplace=True)
            x[:, :, [i + 1], :].add_(x_h)
        for i in range(h - 1, 0, -1):
            x_h = F.relu(self.up[i](x[:, :, [i], :]), inplace=True)
            x[:, :, [i - 1], :].add_(x_h)
        x[:, :, [0], :] = F.relu(self.up[0](x[:, :, [0], :]), inplace=True)
        for i in range(w - 1):
            x_h = F.relu(self.left[i](x[:, :, :, [i]]), inplace=True)
            x[:, :, :, [i + 1]].add_(x_h)
        for i in range(w - 1, 0, -1):
            x_h = F.relu(self.right[i](x[:, :, :, [i]]), inplace=True)
            x[:, :, :, [i - 1]].add_(x_h)
        x[:, :, :, [0]] = F.relu(self.up[0](x[:, :, :, [0]]), inplace=True)
        return x
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>You cannot always do inplace operations as some values are needed to compute gradients and so the backward cannot happen if you changed these values with an inplace operation.</p>
<p>To find which one is the problem, you should remove all the inplace operations and then add them back one by one.</p>

username: Cverlpeng
cooked: <p>In my model, those are inplace operations?<br>
thanks</p>

username: albanD
cooked: <p>Hi,</p>
<p>all the operations with the argument <code>inplace=True</code> are inplace operations. You can set the argument to False to disable it.<br>
Also operations with trailing underscores (like <code>add_</code>) are inplace operation. The out of place operations are without the underscore (like <code>add</code>).<br>
Finally operations like <code>x[smth] = foo</code> are inplace operations as well.</p>

username: Cverlpeng
cooked: <p>HI,<br>
Replace them, will it affect the performance of the model?<br>
thanks</p>

username: albanD
cooked: <p>I won’t affect the result, only the memory comsumption.</p>

username: Adex
cooked: <aside class="quote no-group" data-username="albanD" data-post="4" data-topic="52588">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>Finally operations like <code>x[smth] = foo</code> are inplace operations as well.</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/alband">@albanD</a> is there any way to avoid <code>x[smth] = foo </code> inplace operations in pytorch?</p>

username: tom
cooked: <p>The canonical thing to do is <code>x_new = where(smth, foo, x)</code> will do, but then <code>foo</code> has to be constructed differently (to be of the same shape as <code>x</code>, so this is in fact similar to <code>x[smth] = foo[smth]</code> out of place).</p>