Issue 1:
🐛 Bug

Recently, I plan to implement Kalman filter with pytorch. I tried to obtain the batch processing capability through torch. However, when I call "torch.cholesky (covariance)" and run it on the GPU, I randomly encounter an incomprehensible error. " CUDA error: an illegal memory access was encountered ", but it does not happen when it is processed through the loop. At the same time, what is even more surprising is that this will not have any problems on window10.

To Reproduce

If i combine the covariances like this, the error will happen randomly.

means = []
covariances = []
for row, track_idx in enumerate(track_indices):
track = tracks[track_idx]
means.append(track.mean)
covariances.append(track.covariance)
means = torch.cat(means, dim=0)
covariances = torch.cat(covariances, dim=0)
gating_distance = kf.gating_distance(means, covariances, measurements, only_position)

What kf.gating_distance does is call torch.cholesky(covariance):

def gating_distance(self, mean, covariance, measurements, only_position=False):
mean, covariance = self.project(mean, covariance)
# (n, 4, 4)
cholesky_factor = torch.cholesky(covariance)
d = - mean + measurements # (n, m, 4)
z = torch.triangular_solve(d.permute(0, 2, 1), cholesky_factor, upper=False)[0]
squared_maha = torch.sum(z ** 2, dim=1) # (n, m)
return squared_maha

def project(self, mean, covariance):
"""Project state distribution to measurement space.
std = torch.tensor([[
self._std_weight_position,
self._std_weight_position,
1e-1,
self._std_weight_position]], device=mean.device)

std = mean[:, 3:4] * std # (*, 4)
std[:, 2] = 1e-1 # (*, 4)

# (*, 4, 4)
innovation_cov = torch.diag_embed(torch.pow(std, 2))

update_mat_t = self._update_mat.t()

# (4, 8) dot (*, 8)
mean = torch.mm(mean, update_mat_t) # (*, 4)

# (*, 4, 4)
covariance = torch.matmul(torch.matmul(covariance.permute(0, 2, 1), update_mat_t).permute(0, 2, 1),
update_mat_t)
return mean, covariance + innovation_cov

The complete code is here kalman_filter.py

Expected behavior

There should be no problem。

Environment

The runtime is from docker image: pytorch/pytorch:1.5-cuda10.1-cudnn7-runtime

Collecting environment information...

PyTorch version: 1.5.0

Is debug build: No

CUDA used to build PyTorch: 10.1

OS: Ubuntu 18.04.4 LTS

GCC version: Could not collect

CMake version: Could not collect

Python version: 3.7

Is CUDA available: Yes

CUDA runtime version: Could not collect

GPU models and configuration: GPU 0: TITAN Xp

Nvidia driver version: 418.56

cuDNN version: Could not collect

Versions of relevant libraries:

[pip] numpy==1.18.1

[pip] torch==1.5.0

[pip] torchvision==0.6.0a0+82fd1c8

[conda] blas 1.0 mkl

[conda] cudatoolkit 10.1.243 h6bb024c_0

[conda] mkl 2020.0 166

[conda] mkl-service 2.3.0 py37he904b0f_0

[conda] mkl_fft 1.0.15 py37ha843d7b_0

[conda] mkl_random 1.1.0 py37hd6b4f25_0

[conda] numpy 1.18.1 py37h4f9e942_0

[conda] numpy-base 1.18.1 py37hde5b4d6_1

[conda] pytorch 1.5.0 py3.7_cuda10.1.243_cudnn7.6.3_0 pytorch

[conda] torchvision 0.6.0 py37_cu101 pytorch

cc @ezyang @gchanan @zou3519 @ngimel @vincentqb @vishwakftw @ssnl @jianyuh

Issue 2:
🐛 Bug

>>> import torch
>>> t = torch.Tensor([[0.5]])
>>> model = torch.nn.Linear(1, 1).cuda()
>>> model(t)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/private/home/tbirch/.conda/envs/torch160/lib/python3.7/site-packages/torch/tensor.py", line 153, in __repr__
return torch._tensor_str._str(self)
File "/private/home/tbirch/.conda/envs/torch160/lib/python3.7/site-packages/torch/_tensor_str.py", line 371, in _str
return _str_intern(self)
File "/private/home/tbirch/.conda/envs/torch160/lib/python3.7/site-packages/torch/_tensor_str.py", line 351, in _str_intern
tensor_str = _tensor_str(self, indent)
File "/private/home/tbirch/.conda/envs/torch160/lib/python3.7/site-packages/torch/_tensor_str.py", line 241, in _tensor_str
formatter = _Formatter(get_summarized_data(self) if summarize else self)
File "/private/home/tbirch/.conda/envs/torch160/lib/python3.7/site-packages/torch/_tensor_str.py", line 89, in __init__
nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) & tensor_view.ne(0))
RuntimeError: copy_if failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered

Expected behavior

Print out the value of the tensor and not crash

Environment

Collecting environment information...

PyTorch version: 1.6.0

Is debug build: No

CUDA used to build PyTorch: 10.1

OS: Ubuntu 18.04.3 LTS

GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0

CMake version: version 3.10.2

Python version: 3.7

Is CUDA available: Yes

CUDA runtime version: 10.1.105

GPU models and configuration:

GPU 0: Quadro GP100

GPU 1: Quadro GP100

Nvidia driver version: 418.116.00

cuDNN version: Could not collect

Versions of relevant libraries:

[pip3] numpy==1.18.5

[pip3] torch==1.6.0

[pip3] torchvision==0.7.0

[conda] blas 1.0 mkl

[conda] cudatoolkit 10.1.243 h6bb024c_0

[conda] mkl 2020.1 217

[conda] mkl-service 2.3.0 py37he904b0f_0

[conda] mkl_fft 1.1.0 py37h23d657b_0

[conda] mkl_random 1.1.1 py37h0da4684_0 conda-forge

[conda] numpy 1.18.5 py37ha1c710e_0

[conda] numpy-base 1.18.5 py37hde5b4d6_0

[conda] pytorch 1.6.0 py3.7_cuda10.1.243_cudnn7.6.3_0 pytorch

[conda] torchvision 0.7.0 py37_cu101 pytorch

cc @ezyang @gchanan @zou3519 @ngimel