Wrong results on CUDA

HfCloud: <p>Hi, I compiled pytorch1.13.1, built with CUDA (CUDA11.6)，When I ran this code on the machine where pytorch was compiled, it worked properly, but when I put the compiled pytorch on another machine, (it was confirmed that torch.caua.is_available () is True), and I found wrong results on CUDA, as the following figure:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/f/3f45651ef783756c2fa6b20c9eab46a97d5753f9.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/3f45651ef783756c2fa6b20c9eab46a97d5753f9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3f45651ef783756c2fa6b20c9eab46a97d5753f9_2_690x239.jpeg" alt="image" data-base62-sha1="91IL2CVOtZ6QIBJb3cvJCI3s4gV" width="690" height="239" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3f45651ef783756c2fa6b20c9eab46a97d5753f9_2_690x239.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3f45651ef783756c2fa6b20c9eab46a97d5753f9_2_1035x358.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3f45651ef783756c2fa6b20c9eab46a97d5753f9_2_1380x478.jpeg 2x" data-dominant-color="3B355C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1608×558 139 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I used ldd and patchelf to make sure all the dependencies were bundled and migrated to the new machine. Is there anything that could be causing this problem, I have some knowledge of the PyTorch source code and need some ideas to figure this out.</p>

ptrblck: <p>What’s the difference between the two machines? Are they using the same GPUs or from the same GPU family?</p>

HfCloud: <p>Thanks for your reply and reminder. The pytorch was compiled in a docker container which was installed ubuntu 20.04 and cuda toolkit 11.6, and it worked properly (results are corret) on ArchLinux-latest(the docker container also runs on it) with 2 RTX 8000 GPUs, Nvidia driver version is 535.113.01.<br>
The another machine which got wrong results is installed ubuntu 22.04, with an A100 80G GPU, Nvidia driver version is 535.154.05.<br>
That’s a weird question.</p>
<p>I’m also trying to switch to the v1.13.1 binary library I downloaded from the official website(<a href="http://pytorch.org" rel="noopener nofollow ugc">pytorch.org</a>), because I notice that on the “another machine” official pytorch1.13.1 (installed by conda) can give correct results. but both pre-cxx11-abi version and cxx11-abi will cause tons of linking error(both g++ and clang++).  (While windows version works fine with msvc toolchain)</p>

HfCloud: <p>All operations will result in 0, not only ones, add.</p>

ptrblck: <p>I don’t know if you are building PyTorch from source, but it seems installing the binaries works:</p>
<aside class="quote no-group" data-username="HfCloud" data-post="3" data-topic="199097">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hfcloud/48/59526_2.png" class="avatar"> HfCloud:</div>
<blockquote>
<p>I’m also trying to switch to the v1.13.1 binary library I downloaded from the official website(<a href="http://pytorch.org">pytorch.org</a>), because I notice that on the “another machine” official pytorch1.13.1 (installed by conda) can give correct results.</p>
</blockquote>
</aside>
<p>so I would assume your build creates binaries which are somehow broken? Did you check the expected compute capabilities are supported?</p>

HfCloud: <p>Well, I embeded python interpreter into my application, and maked use of both libTorch and PyTorch. Official version 1.13.1 cxx11-abi libtorch dynamic libraries are not compatible with pytorch1.13.1’s _C.cpython-310-x86_64-linux-gnu.so. (Because official pytorch usees pre-cxx11 abi), so I have to compile pytorch myself.</p>
<p>Yes, I also think some binaries are somehow broken, but there are so many libraries, I can’t figure out which are broken… I remember that pytorch1.13.1 was compiled in lower compute capability than 8.0 (A100GPU support 8.0).</p>
<p>This problem has made me crazy</p>

HfCloud: <p>Compiling PyTorch by oneself is truly a terrible experience. Initially, I attempted to compile PyTorch directly on my Linux system, which led to a multitude of compilation errors. Consequently, I had no choice but to use the pytorch/pytorch Docker image for compilation. I thought this would resolve the issues, but later discovered that the computation results were incorrect when transferred to another machine. On Windows, I directly used the official PyTorch library released by the developers, which worked very well. However, on Linux, I was forced to compile PyTorch myself because the official PyTorch release’s _C.cpython-310-x86_64-linux-gnu.so is not compatible with the cxx11-abi. The cxx11abi version of libtorch provided on the PyTorch official website cannot be used by _C.cpython-310-x86_64-linux-gnu.so. My program requires the c++11 abi. I don’t understand why, in the year 2024, the official PyTorch release does not use the c++11 abi (torch._C._GLIBCXX_USE_CXX11_ABI shows False).</p>
<p>Now I’ trying to use docker to package my software with the environment of the machine which gives correct results. Hope to solve this problem <img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=12" title=":rofl:" class="emoji" alt=":rofl:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>Yes, indeed source builds might not be trivial in custom environments, but will also directly work in e.g. standard Ubuntu installations. Good luck with the docker approach!</p>

HfCloud: <p>Hi, I tried docker, the issue still occured.  I noticed that my compiled pytorch ran</p>
<pre data-code-wrap="python"><code class="lang-python">torch.cuda.get_arch_list()
</code></pre>
<p>it gave only a [‘sm_75’]<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/a/fa418c72d49398f283d80537ce1ae41dc1b3c6da.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/fa418c72d49398f283d80537ce1ae41dc1b3c6da" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa418c72d49398f283d80537ce1ae41dc1b3c6da_2_690x219.jpeg" alt="image" data-base62-sha1="zHRNU1NAl12t98rCHWWsMv2p1ma" width="690" height="219" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa418c72d49398f283d80537ce1ae41dc1b3c6da_2_690x219.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa418c72d49398f283d80537ce1ae41dc1b3c6da_2_1035x328.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/a/fa418c72d49398f283d80537ce1ae41dc1b3c6da.jpeg 2x" data-dominant-color="403A60"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1246×396 91.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>That ‘another machine’ has an A100 80G GPU (compute capability 8.0).</p>
<p>And I install pytorch by conda on this machine, it works properly:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9cb5d82ff39f4d3e80b0db3a4accfc4471c8ed97.png" data-download-href="https://discuss.pytorch.org/uploads/default/9cb5d82ff39f4d3e80b0db3a4accfc4471c8ed97" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9cb5d82ff39f4d3e80b0db3a4accfc4471c8ed97_2_690x280.png" alt="image" data-base62-sha1="mmk5ig4PbmdARtRiTQR1qNuXeTB" width="690" height="280" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9cb5d82ff39f4d3e80b0db3a4accfc4471c8ed97_2_690x280.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9cb5d82ff39f4d3e80b0db3a4accfc4471c8ed97_2_1035x420.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/c/9cb5d82ff39f4d3e80b0db3a4accfc4471c8ed97.png 2x" data-dominant-color="EAECEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1250×508 46.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>They are on the same machine. So maybe it is bacause my compiled pytorch was not compiled for sm_80?</p>
<p>But I remember, 8.0 devices should be able to run 7.5 programs, why? <img src="https://discuss.pytorch.org/images/emoji/apple/pleading_face.png?v=12" title=":pleading_face:" class="emoji" alt=":pleading_face:" loading="lazy" width="20" height="20"></p>

HfCloud: <p>Forgot to click “Reply”, sorry to reply again <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <aside class="quote no-group" data-username="HfCloud" data-post="9" data-topic="199097">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hfcloud/48/59526_2.png" class="avatar"> HfCloud:</div>
<blockquote>
<p>8.0 devices should be able to run 7.5 programs</p>
</blockquote>
</aside>
<p>That’s not the case and you will need to compile PyTorch for <code>sm_80</code> to be able to run on your Ampere GPU. Minor versions can be compatible, e.g. <code>sm_89</code> is compatible to <code>sm_86</code> and <code>sm_80</code>.</p>

HfCloud: <p>Yes，thanks. I’m sure it is the cause now. Our software contains some other CUDA kernel functions, which was compiled with sm_80, I just tested them and they worked fine on the A100 GPU machine.</p>