Wrong result while running torch.cat on GPU

Jiang-Stan: <p>While trying to inference by a pretrained pytorch model, I found a strange phenomena that torch.cat may fail to get right answer, as shown in figure.<br>
when abc is a tensor in GPU, torch.cat([abc,abc],axis=1) may running in a weird way.<br>
but when abc is set to CPU, the answer would be correct.<br>
and when I try abc.cpu().cuda(), the answer would be correct again.<br>
I can’t understand why it happend.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/4/0459dc56593154747c8d11907d15454d97e6935e.png" data-download-href="https://discuss.pytorch.org/uploads/default/0459dc56593154747c8d11907d15454d97e6935e" title="2022-03-07 14-40-07 的屏幕截图"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/0459dc56593154747c8d11907d15454d97e6935e_2_690x415.png" alt="2022-03-07 14-40-07 的屏幕截图" data-base62-sha1="Cur0arPpoG2PLUJveUdgrzR9Ya" width="690" height="415" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/0459dc56593154747c8d11907d15454d97e6935e_2_690x415.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/0459dc56593154747c8d11907d15454d97e6935e_2_1035x622.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/4/0459dc56593154747c8d11907d15454d97e6935e.png 2x" data-dominant-color="2E1824"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2022-03-07 14-40-07 的屏幕截图</span><span class="informations">1269×764 243 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/6/f61a607aef679707cda74cb83025a0df64c68810.png" data-download-href="https://discuss.pytorch.org/uploads/default/f61a607aef679707cda74cb83025a0df64c68810" title="2022-03-07 14-48-35 的屏幕截图"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/6/f61a607aef679707cda74cb83025a0df64c68810_2_690x172.png" alt="2022-03-07 14-48-35 的屏幕截图" data-base62-sha1="z77Y8oBiC4S8DS2S5nKco1EwmhW" width="690" height="172" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/6/f61a607aef679707cda74cb83025a0df64c68810_2_690x172.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/6/f61a607aef679707cda74cb83025a0df64c68810_2_1035x258.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/6/f61a607aef679707cda74cb83025a0df64c68810.png 2x" data-dominant-color="2F1925"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2022-03-07 14-48-35 的屏幕截图</span><span class="informations">1224×306 99.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Could you post a minimal, executable code snippet as well as the output of <code>python -c torch.utils.collect_env</code>, please?</p>

Jiang-Stan: <p>Sorry I can’t reproduce this error by a minimal, executable code, the error comes from a big project. But I suspect that the error comes from uncontiguous memory, since there’s some <em>tensor.permute</em> in the code.<br>
I further tested <em>tensor.contiguous()</em>, and found that this could solve it. as shown in figure.<br>
env informations are as follows.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/a/ba396d7cf803f71158cb2c2c8618b17520ec9062.png" data-download-href="https://discuss.pytorch.org/uploads/default/ba396d7cf803f71158cb2c2c8618b17520ec9062" title="2022-03-07 15-40-10 的屏幕截图"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/ba396d7cf803f71158cb2c2c8618b17520ec9062_2_690x346.png" alt="2022-03-07 15-40-10 的屏幕截图" data-base62-sha1="qzpQyMrhPV0IY8qYtaYzrFIpu3o" width="690" height="346" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/ba396d7cf803f71158cb2c2c8618b17520ec9062_2_690x346.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/ba396d7cf803f71158cb2c2c8618b17520ec9062_2_1035x519.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/a/ba396d7cf803f71158cb2c2c8618b17520ec9062.png 2x" data-dominant-color="2F1824"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2022-03-07 15-40-10 的屏幕截图</span><span class="informations">1217×612 195 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>PyTorch version: 1.8.1+cu102<br>
Is debug build: False<br>
CUDA used to build PyTorch: 10.2<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Ubuntu 18.04.3 LTS (x86_64)<br>
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0<br>
Clang version: Could not collect<br>
CMake version: version 3.10.2</p>
<p>Python version: 3.6 (64-bit runtime)<br>
Is CUDA available: True<br>
CUDA runtime version: Could not collect<br>
GPU models and configuration: GPU 0: GeForce RTX 2080 Ti<br>
Nvidia driver version: 460.32.03<br>
cuDNN version: Could not collect<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.18.1<br>
[pip3] pytorchcv==0.0.66<br>
[pip3] torch==1.8.1<br>
[pip3] torchfile==0.1.0<br>
[pip3] torchnet==0.0.4<br>
[pip3] torchvision==0.9.1<br>
[conda] numpy                     1.18.1                   pypi_0    pypi<br>
[conda] pytorchcv                 0.0.66                   pypi_0    pypi<br>
[conda] torch                     1.8.1                    pypi_0    pypi<br>
[conda] torchfile                 0.1.0                    pypi_0    pypi<br>
[conda] torchnet                  0.0.4                    pypi_0    pypi<br>
[conda] torchvision               0.9.1                    pypi_0    pypi</p>

ptrblck: <p>Could you update PyTorch to the latest stable or nightly release and check if you are still hitting the issue?</p>

Jiang-Stan: <p>I tested with the same config on pytorch 1.10.2 and it works well. The bug disappeared.<br>
It seems that this issue has been solved in some version? Could you please help me check in which version of pytorch this bug is fixed? Thanks a lot!</p>