What’s the difference between nn.ReLU() and nn.ReLU(inplace=True)?

yunjey: 

apaszke: 

laoreja: 

cdancette: is not an in place operation, because you use the same variable name, but it’s not the same variable underneath. You just point your x name to a new variable, the old one is still in memory (because it’s referenced by the pytorch graph)

laoreja: 

harryhan618: I’m a newbie of pytorch. So I wonder whether nn.ReLI(inplace=True) would do harm to backprop? And what about F.ReLU(inplace)?

rasbt: 

yxchng: 

streaver91: For relu, when input is negative, both the grad and output should be zero, grads should stop propagating from there, so inplace doesn’t hurt anything while saves memory.

Jay_Timbadia: Is this an in-place operation? Thanks!

cdancette: 

cdancette: You can check on pytorch: <SCODE>>>> a = torch.tensor(1.0)
>>> torch.sigmoid(a)
tensor(0.7311)
>>> print(a)
tensor(1.0)
<ECODE> <SCODE>>>> a = torch.tensor(1.0)
>>> torch.sigmoid_(a)
tensor(0.7311)
>>> print(a)
tensor(0.7311)
<ECODE> In the first case, a still has its original value, while in the second case, a is different.

helson73: In case y = F.relu(x, inplace=True), it won’t hurt anything if value of x should  always be positive in your computational graph. However, some other node that shares x as input while it requires x has both positive and negative value, then your network may malfunction. For example, in the following situation, If (1) is declared first and execuated first, then value of x is changed, then z may have incorrect expected value.

cwinkler: Hi all, Even though there are multiple answers, I will explain my problem here. I am facing the following error message: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [4, 64, 3, 3]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck! <SCODE>    def forward(self, z, xlr=None, logdet=0, logpz=0, eps=None, reverse=False,
                use_stored=False):

        self.h_new, self.c_next = self.conv_lstm(z, (self.h, self.c))

        # Encode
        if not reverse:
            for i in range(self.L):
                print("Level", i)
                for layer in self.level_modules[i]:

                    if isinstance(layer, modules.Squeeze):
                        z = layer(z, reverse=False)
                        self.h_new = layer(self.h_new, reverse=False)

                    elif isinstance(layer, FlowStep):
                        z, logdet = layer(z, lr_feat_map=self.h_new, # lr_downsampled_feats[i + 1], # TODO: change this part
                                          x_lr=xlr, logdet=logdet, reverse=False)

                    elif isinstance(layer, modules.GaussianPrior):
                        z, logdet, logpz = layer(z, logdet=logdet, logpz=logpz,
                                                 lr_feat_map=self.h_new, #lr_downsampled_feats[i + 1],
                                                 eps=eps, reverse=False)
                        self.h = self.last_squeezer(self.h_new, reverse=True)
                        self.c = self.c_next
<ECODE> Do you think it would be the best to just pass on the hidden and context states through the function outputs? I have already set loss.mean().backward(retain_graph=True) and skimmed the code for other inplace operations. Any help would be much appreciated !! Please let me know if further code snippets are required.