Issue 1:
🐛 Bug

I'm using the nightly build: 1.0.0.dev20181123. This issue is very similar to #13569 . When I instantiate two nn.Embedding, with DataParallel and with max_norm=1.0, I get the following assert

RuntimeError: output_nr_ == 0 ASSERT FAILED at /pytorch/torch/csrc/autograd/variable.cpp:196, please report a bug to PyTorch.

If I remove the self.lut_dummy, the issue disappears.

To Reproduce

import torch
import os

print torch.__version__
class Moda(torch.nn.Module):
def __init__(self):
super(Moda, self).__init__()
self.lut_dummy = torch.nn.Embedding(1, 1, max_norm=1.0).to("cuda")
self.lut_a = torch.nn.Embedding(22, 256, max_norm=1.0).to("cuda")

def forward(self, src):
print "ok1"
ebd = self.lut_a(src)
print "ok2"
return ebd

def main():
os.environ['CUDA_VISIBLE_DEVICES'] = "0,1"
model = Moda()
model = torch.nn.DataParallel(model)
src = torch.randint(4,(2,)).to("cuda")
output = model(src)

if __name__ == '__main__':
main()

Output

1.0.0.dev20181123

/home/software/LM_stash/amitoj/pytorch1.0/local/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py:25: UserWarning:

There is an imbalance between your GPUs. You may want to exclude GPU 0 which

has less than 75% of the memory or cores of GPU 1. You can do so by setting

the device_ids argument to DataParallel, or by setting the CUDA_VISIBLE_DEVICES

environment variable.

warnings.warn(imbalance_warn.format(device_ids[min_pos], device_ids[max_pos]))

ok1

ok1

ok2

Traceback (most recent call last):

File "error_train.py", line 25, in

main()

File "error_train.py", line 22, in main

output = model(src)

File "/home/software/LM_stash/amitoj/pytorch1.0/local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 479, in call

result = self.forward(*input, **kwargs)

File "/home/software/LM_stash/amitoj/pytorch1.0/local/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 143, in forward

outputs = self.parallel_apply(replicas, inputs, kwargs)

File "/home/software/LM_stash/amitoj/pytorch1.0/local/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 153, in parallel_apply

return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])

File "/home/software/LM_stash/amitoj/pytorch1.0/local/lib/python2.7/site-packages/torch/nn/parallel/parallel_apply.py", line 83, in parallel_apply

raise output

RuntimeError: output_nr_ == 0 ASSERT FAILED at /pytorch/torch/csrc/autograd/variable.cpp:196, please report a bug to PyTorch.

Additional context

Observations

If I comment out the self.lut_dummy, the issue disappears. Though in my real model, I am using the lut_dummy.

If max_norm is removed, the issue disappears again.

If I set os.environ["CUDA_VISIBLE_DEVICES"] to "0" or "1", again, the code works just fine.

cc @ezyang @gchanan @ssnl @albanD

Issue 2:
🐛 Bug

Gradients aren't backpropagated when all three conditions are met:

Use torch.utils.checkpoint.

Place nn.InstanceNorm?d right before any in-place module such as nn.ReLU(inplace=True).

The checkpoint returns the result of the in-place module.

To Reproduce

import torch
from torch import nn
from torch.utils.checkpoint import checkpoint

x = torch.ones(3, 2, 1, requires_grad=True)

model = nn.Sequential(
nn.InstanceNorm1d(2),
nn.ReLU(inplace=True),
)
y = checkpoint(model, x)

y.norm().backward()

print(x.grad)
# Output: None
# Expected: tensor([[[0.],[0.]], ...])

x the input tensor doesn't receive gradients. x.grad is still None, which is not expected.

Expected behavior

x.grad should be torch.zeros(3, 2, 1). If we remove any of the conditions, the gradients are backpropagated correctly:

# No in-place operation
model = nn.Sequential(
nn.InstanceNorm1d(2),
nn.ReLU(inplace=False),
)
y = checkpoint(model, x)

# No checkpoint
model = nn.Sequential(
nn.InstanceNorm1d(2),
nn.ReLU(inplace=True),
)
y = model(x)

# Not at the end of checkpoint
model = nn.Sequential(
nn.InstanceNorm1d(2),
nn.ReLU(inplace=True),
nn.Conv1d(2, 2, 1),
)
y = checkpoint(model, x)

# Other norm
model = nn.Sequential(
nn.BatchNorm1d(2),
nn.ReLU(inplace=True),
)
y = checkpoint(model, x)

Environment

Collecting environment information...
PyTorch version: 1.2.0
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Ubuntu 18.04.2 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: version 3.10.2

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration:
GPU 0: Tesla P40
GPU 1: Tesla P40
GPU 2: Tesla P40
GPU 3: Tesla P40
GPU 4: Tesla P40
GPU 5: Tesla P40
GPU 6: Tesla P40
GPU 7: Tesla P40

Nvidia driver version: 410.104
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.2

Versions of relevant libraries:
[pip3] numpy==1.16.4
[pip3] torch==1.2.0
[pip3] torchgpipe==0.0.3
[pip3] torchvision==0.4.0
[conda] blas 1.0 mkl
[conda] mkl 2019.4 243
[conda] mkl-include 2019.4 243
[conda] mkl-service 2.0.2 py36h7b6447c_0
[conda] mkl_fft 1.0.12 py36ha843d7b_0
[conda] mkl_random 1.0.2 py36hd81dba3_0
[conda] torch 1.2.0 pypi_0 pypi
[conda] torchgpipe 0.0.3 dev_0 <develop>
[conda] torchvision 0.4.0 pypi_0 pypi

cc @ezyang @ssnl @albanD @zou3519 @gqchen