Issue 1:
PyTorch checks if tensors have been modified in-place to ensure correctness of the backward operation, with ctx.save_for_backward and ctx.saved_tensors in the forward and backward methods of subclasses of torch.autograd.Function. This check does not appear to function when the saved tensor is not the 'owner' of its storage, e.g. when it is the result of a view or transpose operation.

To reproduce:

import torch

class f(torch.autograd.Function):
@staticmethod
def forward(ctx, tensor1):
# make sure we don't share storage with our input
# (just to be clear that that's not what's going wrong)
tensor2 = tensor1.clone()
# tensor3 shares storage with tensor2
# tensor3 = tensor2.view(-1) etc. would also demonstrate the problem
tensor3 = tensor2.t()
ctx.save_for_backward(tensor3)
return tensor3
@staticmethod
def backward(ctx, grad):
# trigger correctness check
_ = ctx.saved_tensors
return grad.t().clone()

x = torch.rand(2, 3, requires_grad=True)
y = f.apply(x)
y += 1
# Does not raise an error; this is the bug.
y.backward(torch.rand_like(y))

# In constrast, the following code correctly raises an error.
class ff(torch.autograd.Function):
@staticmethod
def forward(ctx, tensor1):
tensor2 = tensor1.clone()
ctx.save_for_backward(tensor2)
return tensor2
@staticmethod
def backward(ctx, grad):
_ = ctx.saved_tensors
return grad.clone()

xx = torch.rand(2, 3, requires_grad=True)
yy = ff.apply(xx)
yy += 1
yy.backward(torch.rand_like(yy))

Expected behaviour is for an exception to be raised on the line y.backward(torch.rand_like(y)), in the same manner as an exception is raised on the line yy.backward(torch.rand_like(yy)).

PyTorch Version (e.g., 1.0): 1.2

OS (e.g., Linux): Linux

How you installed PyTorch (conda, pip, source): conda

Python version: 3.7.4

cc @ezyang @ssnl @albanD

Issue 2:
🐛 Bug

Gradients aren't backpropagated when all three conditions are met:

Use torch.utils.checkpoint.

Place nn.InstanceNorm?d right before any in-place module such as nn.ReLU(inplace=True).

The checkpoint returns the result of the in-place module.

To Reproduce

import torch
from torch import nn
from torch.utils.checkpoint import checkpoint

x = torch.ones(3, 2, 1, requires_grad=True)

model = nn.Sequential(
nn.InstanceNorm1d(2),
nn.ReLU(inplace=True),
)
y = checkpoint(model, x)

y.norm().backward()

print(x.grad)
# Output: None
# Expected: tensor([[[0.],[0.]], ...])

x the input tensor doesn't receive gradients. x.grad is still None, which is not expected.

Expected behavior

x.grad should be torch.zeros(3, 2, 1). If we remove any of the conditions, the gradients are backpropagated correctly:

# No in-place operation
model = nn.Sequential(
nn.InstanceNorm1d(2),
nn.ReLU(inplace=False),
)
y = checkpoint(model, x)

# No checkpoint
model = nn.Sequential(
nn.InstanceNorm1d(2),
nn.ReLU(inplace=True),
)
y = model(x)

# Not at the end of checkpoint
model = nn.Sequential(
nn.InstanceNorm1d(2),
nn.ReLU(inplace=True),
nn.Conv1d(2, 2, 1),
)
y = checkpoint(model, x)

# Other norm
model = nn.Sequential(
nn.BatchNorm1d(2),
nn.ReLU(inplace=True),
)
y = checkpoint(model, x)

Environment

Collecting environment information...
PyTorch version: 1.2.0
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Ubuntu 18.04.2 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: version 3.10.2

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration:
GPU 0: Tesla P40
GPU 1: Tesla P40
GPU 2: Tesla P40
GPU 3: Tesla P40
GPU 4: Tesla P40
GPU 5: Tesla P40
GPU 6: Tesla P40
GPU 7: Tesla P40

Nvidia driver version: 410.104
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.2

Versions of relevant libraries:
[pip3] numpy==1.16.4
[pip3] torch==1.2.0
[pip3] torchgpipe==0.0.3
[pip3] torchvision==0.4.0
[conda] blas 1.0 mkl
[conda] mkl 2019.4 243
[conda] mkl-include 2019.4 243
[conda] mkl-service 2.0.2 py36h7b6447c_0
[conda] mkl_fft 1.0.12 py36ha843d7b_0
[conda] mkl_random 1.0.2 py36hd81dba3_0
[conda] torch 1.2.0 pypi_0 pypi
[conda] torchgpipe 0.0.3 dev_0 <develop>
[conda] torchvision 0.4.0 pypi_0 pypi

cc @ezyang @ssnl @albanD @zou3519 @gqchen