Issue 1:
🐛 Bug

CTCLoss occassionally causes segfault

To Reproduce

Steps to reproduce the behavior:

# model
class StrippedResnet(nn.Module):

def __init__(self, base_model: nn.Module, n_classes: int):
super().__init__()
self.base = nn.Sequential(*list(base_model.children())[:-2])

self.last_conv = nn.Conv2d(
512,
n_classes,
(2, 3),
stride=1,
padding=(10, 1), # This has to be padded to get shape of [1,95,/23/,1]
bias=False
)

nn.init.kaiming_normal_(
self.last_conv.weight,
mode='fan_out',
nonlinearity='relu'
)

def forward(self, x):
x = self.base(x)
x = self.last_conv(x)
return x

# script.py
batch_size = 1

loader = get_loader()
model = resnet18(pretrained=True)
model = StrippedResnet(model, num_classes)

criterion = nn.CTCLoss()
model = model.train()

for data in loader:

features = data['image'].type('torch.DoubleTensor')
model = model.double()
labels = bat_to_tensor(data['text'])

logits = model.forward(features)
probs = nn.functional.log_softmax(logits, 2)
_, preds = torch.max(logits, 2)
pred_lens = Tensor([preds.size(0)] * batch_size).cpu()
label_lens = Tensor(batch_size).cpu()

probs = torch.squeeze(probs, 3).view(23, batch_size, num_classes).cpu()

# line that usually causes the segfault
loss = criterion(probs, labels, pred_lens, label_lens)

Bottom of the stack trace after running the script in gdb:

gdb --args python script.py

...
Program received signal SIGSEGV, Segmentation fault.
0x00007fffbca762cc in std::tuple<at::Tensor, at::Tensor> at::native::(anonymous namespace)::ctc_loss_cpu_template<double, (c10::ScalarType)3
>(at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, long) ()
from /data2/asher_scratch/miniconda3/envs/torch/lib/python3.7/site-packages/torch/lib/libcaffe2.so

Valgrind finds multiple instances of this error in the program:

valgrind --tool=memcheck \
--suppressions=valgrind-python.supp \
--error-limit=no \
python script.py

...
==14765== Conditional jump or move depends on uninitialised value(s)
==14765== at 0x5A8B53D: __ieee754_exp_avx (e_exp.c:67)
==14765== by 0x5A51F62: exp (w_exp.c:26)
==14765== by 0x17DF6309: std::tuple<at::Tensor, at::Tensor> at::native::(anonymous namespace)::ctc_loss_cpu_template<double, (c10::ScalarType)3>(at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, long) (in /data2/asher_scratch/miniconda3/envs/torch/lib/python3.7/site-packages/torch/lib/libcaffe2.so)
==14765== by 0x17DF9BFF: at::native::ctc_loss_cpu(at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, long)::{lambda()#1}::operator()() const (in /data2/asher_scratch/miniconda3/envs/torch/lib/python3.7/site-packages/torch/lib/libcaffe2.so)
==14765== by 0x17DFC018: at::native::ctc_loss_cpu(at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, long) (in /data2/asher_scratch/miniconda3/envs/torch/lib/python3.7/site-packages/torch/lib/libcaffe2.so)
==14765== by 0x17F30CE1: at::CPUDoubleType::_ctc_loss(at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, long) const (in /data2/asher_scratch/miniconda3/envs/torch/lib/python3.7/site-packages/torch/lib/libcaffe2.so)
==14765== by 0x1C531226: torch::autograd::VariableType::_ctc_loss(at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, long) const (in /data2/asher_scratch/miniconda3/envs/torch/lib/python3.7/site-packages/torch/lib/libtorch.so.1)
==14765== by 0x17DF3E88: at::native::ctc_loss(at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, long, long) (in /data2/asher_scratch/miniconda3/envs/torch/lib/python3.7/site-packages/torch/lib/libcaffe2.so)
==14765== by 0x17DF45AA: at::native::ctc_loss(at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, long, long) (in /data2/asher_scratch/miniconda3/envs/torch/lib/python3.7/site-packages/torch/lib/libcaffe2.so)
==14765== by 0x1802D723: at::TypeDefault::ctc_loss(at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, long, long) const (in /data2/asher_scratch/miniconda3/envs/torch/lib/python3.7/site-packages/torch/lib/libcaffe2.so)
==14765== by 0x1C542301: torch::autograd::VariableType::ctc_loss(at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, long, long) const (in /data2/asher_scratch/miniconda3/envs/torch/lib/python3.7/site-packages/torch/lib/libtorch.so.1)
==14765== by 0x16F37E5A: ??? (in /data2/asher_scratch/miniconda3/envs/torch/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
==14765==
...
==29498== Use of uninitialised value of size 8
==29498== at 0x17DF62D2: std::tuple<at::Tensor, at::Tensor> at::native::(anonymous namespace)::ctc_loss_cpu_template<double, (c10::ScalarType)3>(at::Tensor const&, at::Tensorconst&, c10::ArrayRef<long>, c10::ArrayRef<long>, long) (in /data2/asher_scratch/miniconda3/envs/torch/lib/python3.7/site-packages/torch/lib/libcaffe2.so)
==29498== by 0x17DF9BFF: at::native::ctc_loss_cpu(at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, long)::{lambda()#1}::operator()() const (in /data2/asher_scratch/miniconda3/envs/torch/lib/python3.7/site-packages/torch/lib/libcaffe2.so)
==29498== by 0x17DFC018: at::native::ctc_loss_cpu(at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, long) (in /data2/asher_scratch/miniconda3/envs/torch/lib/python3.7/site-packages/torch/lib/libcaffe2.so)
==29498== by 0x17F30CE1: at::CPUDoubleType::_ctc_loss(at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, long) const (in /data2/asher_scratch/miniconda3/envs/torch/lib/python3.7/site-packages/torch/lib/libcaffe2.so)
==29498== by 0x1C531226: torch::autograd::VariableType::_ctc_loss(at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, long) const (in /data2/asher_scratch/miniconda3/envs/torch/lib/python3.7/site-packages/torch/lib/libtorch.so.1)
==29498== by 0x17DF3E88: at::native::ctc_loss(at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, long, long) (in /data2/asher_scratch/miniconda3/envs/torch/lib/python3.7/site-packages/torch/lib/libcaffe2.so)
==29498== by 0x17DF45AA: at::native::ctc_loss(at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, long, long) (in /data2/asher_scratch/miniconda3/envs/torch/lib/python3.7/site-packages/torch/lib/libcaffe2.so)
==29498== by 0x1802D723: at::TypeDefault::ctc_loss(at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, long, long) const (in /data2/asher_scratch/miniconda3/envs/torch/lib/python3.7/site-packages/torch/lib/libcaffe2.so)
==29498== by 0x1C542301: torch::autograd::VariableType::ctc_loss(at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, long, long) const (in /data2/asher_scratch/miniconda3/envs/torch/lib/python3.7/site-packages/torch/lib/libtorch.so.1)
==29498== by 0x16F37E5A: ??? (in /data2/asher_scratch/miniconda3/envs/torch/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
==29498== by 0x281003: ??? (in /data2/asher_scratch/miniconda3/envs/torch/bin/python3.7)
==29498== by 0x2A629047: ???
...
==29498== HEAP SUMMARY:
==29498== in use at exit: 1,644,916,800 bytes in 1,389,021 blocks
==29498== total heap usage: 3,098,608 allocs, 1,709,587 frees, 2,819,434,323 bytes allocated
==29498==
==29498== LEAK SUMMARY:
==29498== definitely lost: 3,791 bytes in 34 blocks
==29498== indirectly lost: 3,024 bytes in 35 blocks
==29498== possibly lost: 629,908,519 bytes in 173,375 blocks
==29498== still reachable: 1,015,001,466 bytes in 1,215,577 blocks
==29498== suppressed: 0 bytes in 0 blocks

This finds errors starting in ctc_loss_cpu and ctc_loss_cpu_template.

Expected behavior

Normal loss calculation

Environment

PyTorch version: 1.0.0

Is debug build: No

CUDA used to build PyTorch: 8.0.61

OS: Ubuntu 14.04.5 LTS

GCC version: (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4

CMake version: version 3.2.2

Python version: 3.7

Is CUDA available: Yes

CUDA runtime version: Could not collect

GPU models and configuration:

GPU 0: TITAN X (Pascal)

GPU 1: TITAN X (Pascal)

GPU 2: TITAN X (Pascal)

GPU 3: TITAN X (Pascal)

Nvidia driver version: 375.39

cuDNN version: Probably one of the following:

/usr/local/cuda-7.5_cudnn-4/lib64/libcudnn.so.4.0.7

/usr/local/cuda-7.5_cudnn-4/lib64/libcudnn_static.a

/usr/local/cuda-8.0/lib64/libcudnn.so.6.0.21

/usr/local/cuda-8.0/lib64/libcudnn_static.a

/usr/local/cuda-8.0_cudnn-4/lib64/libcudnn.so.4.0.7

/usr/local/cuda-8.0_cudnn-4/lib64/libcudnn_static.a

/usr/local/cuda-8.0_cudnn-5/lib64/libcudnn.so.5.1.5

/usr/local/cuda-8.0_cudnn-5/lib64/libcudnn_static.a

/usr/local/cuda-9.2/lib64/libcudnn.so.7.1.4

/usr/local/cuda-9.2/lib64/libcudnn_static.a

Additional context

Does not fault every run

Issue 2:
🐛 Bug

torch.seed() fails with Overflow when unpacking long after a tensor is copied to cuda

To Reproduce

to reproduce:

# test.py
import torch
print(f"Torch version: {torch.__version__}")

x = torch.tensor(data=[[1,2],[3,4]], dtype=torch.long, device=None)
x = x.to('cuda:0')
seed = torch.seed()

$ python tests/test.py
Torch version: 1.5.1

Traceback (most recent call last):
File "tests/test.py", line 10, in <module>
seed = torch.seed()
File "/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/torch/random.py", line 45, in seed
torch.cuda.manual_seed_all(seed)
File "/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/torch/cuda/random.py", line 111, in manual_seed_all
_lazy_call(cb)
File "/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/torch/cuda/__init__.py", line 99, in _lazy_call
callable()
File "/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/torch/cuda/random.py", line 109, in cb
default_generator.manual_seed(seed)
RuntimeError: Overflow when unpacking long

It fails about 75% of time.

Expected behavior

It shouldn't fail.

Additional info

It seems to be related to: #33546

While CI passes, on my machinehuggingface/transformers tests fail with this error.

Environment

PyTorch version: 1.5.1
Is debug build: No
CUDA used to build PyTorch: 10.2

OS: Ubuntu 18.04.4 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
CMake version: version 3.10.2

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration:
GPU 0: GeForce GTX TITAN X
GPU 1: GeForce GTX TITAN X

Nvidia driver version: 440.95.01
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5

Versions of relevant libraries:
[pip3] numpy==1.13.3
[conda] blas 1.0 mkl
[conda] cudatoolkit 10.2.89 hfd86e86_1
[conda] mkl 2020.0 166
[conda] mkl-service 2.3.0 py37he904b0f_0
[conda] mkl_fft 1.0.15 py37ha843d7b_0
[conda] mkl_random 1.1.0 py37hd6b4f25_0
[conda] numpy 1.18.1 pypi_0 pypi
[conda] numpy-base 1.18.1 py37hde5b4d6_1
[conda] numpydoc 0.9.2 py_0
[conda] pytorch 1.5.1 py3.7_cuda10.2.89_cudnn7.6.5_0 pytorch
[conda] pytorch-lightning 0.8.1 pypi_0 pypi
[conda] pytorch-nlp 0.5.0 pypi_0 pypi
[conda] pytorch-pretrained-bert 0.6.2 pypi_0 pypi
[conda] pytorch-transformers 1.1.0 pypi_0 pypi
[conda] torch 1.5.0 pypi_0 pypi
[conda] torchtext 0.5.1 pypi_0 pypi
[conda] torchvision 0.6.1 py37_cu102 pytorch

cc @ezyang @gchanan @zou3519 @ngimel