Issue 1:
If the output of a python autograd function is a view and gets modified in-place with a torch.no_grad block later, the python autograd function's backward function doesn't get called and instead an autograd backward pass (through the forward function) runs. This results in incorrect behavior, especially when the forward pass contains non-differentiable things.

Expected behavior: we should either throw an error message about something or compute the correct gradients. Neither of those happen right now.

import torch
from torch.autograd import Function

class Mul3ThenFlatten(Function):
@staticmethod
def forward(ctx, x):
ctx.save_for_backward(x)
z = x.detach().numpy() * 3
# If I add a .contiguous() to the end of this statement, other things happen.
return torch.tensor(z).view(-1)

@staticmethod
def backward(ctx, dx):
x = ctx.saved_tensors[0]
print("Some side effect")
return dx.view(x.shape) * 3

x = torch.randn(2, 2, requires_grad=True)
output = Mul3ThenFlatten.apply(x)

# Modify the counter in no_grad mode.
with torch.no_grad():
output.fill_(1)

# Doesn't print "Some side effect" :/
output.sum().backward()

x.grad # is None, should really be something or at least an assert

cc @colesbury

Issue 2:
🐛 Bug

Gradients aren't backpropagated when all three conditions are met:

Use torch.utils.checkpoint.

Place nn.InstanceNorm?d right before any in-place module such as nn.ReLU(inplace=True).

The checkpoint returns the result of the in-place module.

To Reproduce

import torch
from torch import nn
from torch.utils.checkpoint import checkpoint

x = torch.ones(3, 2, 1, requires_grad=True)

model = nn.Sequential(
nn.InstanceNorm1d(2),
nn.ReLU(inplace=True),
)
y = checkpoint(model, x)

y.norm().backward()

print(x.grad)
# Output: None
# Expected: tensor([[[0.],[0.]], ...])

x the input tensor doesn't receive gradients. x.grad is still None, which is not expected.

Expected behavior

x.grad should be torch.zeros(3, 2, 1). If we remove any of the conditions, the gradients are backpropagated correctly:

# No in-place operation
model = nn.Sequential(
nn.InstanceNorm1d(2),
nn.ReLU(inplace=False),
)
y = checkpoint(model, x)

# No checkpoint
model = nn.Sequential(
nn.InstanceNorm1d(2),
nn.ReLU(inplace=True),
)
y = model(x)

# Not at the end of checkpoint
model = nn.Sequential(
nn.InstanceNorm1d(2),
nn.ReLU(inplace=True),
nn.Conv1d(2, 2, 1),
)
y = checkpoint(model, x)

# Other norm
model = nn.Sequential(
nn.BatchNorm1d(2),
nn.ReLU(inplace=True),
)
y = checkpoint(model, x)

Environment

Collecting environment information...
PyTorch version: 1.2.0
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Ubuntu 18.04.2 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: version 3.10.2

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration:
GPU 0: Tesla P40
GPU 1: Tesla P40
GPU 2: Tesla P40
GPU 3: Tesla P40
GPU 4: Tesla P40
GPU 5: Tesla P40
GPU 6: Tesla P40
GPU 7: Tesla P40

Nvidia driver version: 410.104
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.2

Versions of relevant libraries:
[pip3] numpy==1.16.4
[pip3] torch==1.2.0
[pip3] torchgpipe==0.0.3
[pip3] torchvision==0.4.0
[conda] blas 1.0 mkl
[conda] mkl 2019.4 243
[conda] mkl-include 2019.4 243
[conda] mkl-service 2.0.2 py36h7b6447c_0
[conda] mkl_fft 1.0.12 py36ha843d7b_0
[conda] mkl_random 1.0.2 py36hd81dba3_0
[conda] torch 1.2.0 pypi_0 pypi
[conda] torchgpipe 0.0.3 dev_0 <develop>
[conda] torchvision 0.4.0 pypi_0 pypi

cc @ezyang @ssnl @albanD @zou3519 @gqchen