RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [128, 4096]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detec

nour: <p>Hi, I’m new to pytorch and I’m trying to train Alexnet on 128x128 images. I modified my kernel sizes with respect to <a href="https://miro.medium.com/max/1400/1*P1p6ubyxUATVoI_fceandQ.webp" rel="noopener nofollow ugc">here</a>. But I get the error:</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [128, 4096]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</p>
</blockquote>
<p>I don’t get any sense from this error and have 0 ideas to overcome. Any help is appreciated.</p>
<pre><code class="lang-auto">

# define model parameters
NUM_EPOCHS = 90  # original paper
BATCH_SIZE = 128
MOMENTUM = 0.9
LR_DECAY = 0.0005
LR_INIT = 0.01
IMAGE_DIM = 128  # pixels
NUM_CLASSES = 2  # 1000 classes for imagenet 2012 dataset
DEVICE_IDS = [0, 1, 2, 3]  # GPUs to use
# modify this to point to your data directory
INPUT_ROOT_DIR = 'alexnet_data_in'
TRAIN_IMG_DIR = '/content/train'
OUTPUT_DIR = 'alexnet_data_out'
LOG_DIR = OUTPUT_DIR + '/tblogs'  # tensorboard logs
CHECKPOINT_DIR = OUTPUT_DIR + '/models'  # model checkpoints

# make checkpoint path directory
os.makedirs(CHECKPOINT_DIR, exist_ok=True)


class AlexNet(nn.Module):
    """
    Neural network model consisting of layers propsed by AlexNet paper.
    """
    def __init__(self, num_classes=2):
        """
        Define and allocate layers for this neural net.
        Args:
            num_classes (int): number of classes to predict with this model
        """
        super().__init__()
        # input size should be : (b x 3 x 227 x 227)
        # The image in the original paper states that width and height are 224 pixels, but
        # the dimensions after first convolution layer do not lead to 55 x 55.
        self.net = nn.Sequential(
            nn.Conv2d(in_channels=3, out_channels=96, kernel_size=8, stride=4),  # (b x 96 x 55 x 55) (b x 96 x 31 x 31)
            nn.ReLU(),
            nn.LocalResponseNorm(size=5, alpha=0.0001, beta=0.75, k=2),  # section 3.3
            nn.MaxPool2d(kernel_size=3, stride=2),  # (b x 96 x 27 x 27) (b x 96 x 15 x 15)
            nn.Conv2d(96, 256, 5, padding=2),  # (b x 256 x 27 x 27) (b x 256 x 15 x 15)
            nn.ReLU(),
            nn.LocalResponseNorm(size=5, alpha=0.0001, beta=0.75, k=2),
            nn.MaxPool2d(kernel_size=3, stride=2),  # (b x 256 x 13 x 13) (b x 256 x 7 x 7)
            nn.Conv2d(256, 384, 3, padding=1),  # (b x 384 x 13 x 13) (b x 384 x 7 x 7)
            nn.ReLU(),
            nn.Conv2d(384, 384, 3, padding=1),  # (b x 384 x 13 x 13)
            nn.ReLU(),
            nn.Conv2d(384, 256, 3, padding=1),  # (b x 256 x 13 x 13)
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=3, stride=2),  # (b x 256 x 6 x 6) (b x 256 x 3 x 3)
        )
        # classifier is just a name for linear layers
        self.classifier = nn.Sequential(
            nn.Dropout(p=0.5, inplace=True),
            nn.Linear(in_features=(256 * 3 * 3), out_features=4096),
            nn.ReLU(),
            nn.Dropout(p=0.5, inplace=True),
            nn.Linear(in_features=4096, out_features=4096),
            nn.ReLU(),
            nn.Linear(in_features=4096, out_features=num_classes),
        )
        self.init_bias()  # initialize bias

    def init_bias(self):
        for layer in self.net:
            if isinstance(layer, nn.Conv2d):
                nn.init.normal_(layer.weight, mean=0, std=0.01)
                nn.init.constant_(layer.bias, 0)
        # original paper = 1 for Conv2d layers 2nd, 4th, and 5th conv layers
        nn.init.constant_(self.net[4].bias, 1)
        nn.init.constant_(self.net[10].bias, 1)
        nn.init.constant_(self.net[12].bias, 1)

    def forward(self, x):
        """
        Pass the input through the net.
        Args:
            x (Tensor): input tensor
        Returns:
            output (Tensor): output tensor
        """
        x = self.net(x)
        x = x.view(-1, 256 * 3 * 3)  # reduce the dimensions for linear layer input
        return self.classifier(x)
</code></pre>

ptrblck: <p>The error is raised by a disallowed inplace operation on a tensor which is needed for the gradient computation.<br>
Based on your code snippet you are explicitly using <code>inplace=True</code> in <code>Dropout</code> layers. Set this argument to <code>False</code> for these layers and the code should work again. I would also guess the second dropout layer is raising the issue as it’s manipulating the <code>ReLU</code> output inplace, which would fit the error message.</p>

nour: <p>Sir, you are a great person… But this raises a new question, I only adjusted the model for image size, why did I get the error ?</p>

ptrblck: <p>I don’t think the error is related to the input shape and I can reproduce it via a spatial input shape of <code>224x224</code> after adjusting the <code>in_features</code> of the first linear layer. Setting <code>inplace=False</code> of the second dropout layer fixes the issue, too.</p>