Pytorch 1.11.0a0 with CUDA compile failure

username: fulltopic
cooked: <p>Build Pytorch from source.</p>
<pre data-code-wrap="bash"><code class="lang-bash"> export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
USE_LMDB=ON python setup.py install
</code></pre>
<p>The build failed at very beginning (about 3/5xxx) when building protobuf related.<br>
The whole shell exited quietly without visible output.</p>
<p><strong>The output of collect_env.py:</strong><br>
PyTorch version: N/A<br>
Is debug build: N/A<br>
CUDA used to build PyTorch: N/A<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Fedora Linux 35 (Workstation Edition) (x86_64)<br>
GCC version: (GCC) 11.2.1 20211203 (Red Hat 11.2.1-7)<br>
Clang version: Could not collect<br>
CMake version: version 3.19.6<br>
Libc version: glibc-2.34</p>
<p>Python version: 3.9.7 (default, Sep 16 2021, 13:09:58)  [GCC 7.5.0] (64-bit runtime)<br>
Python platform: Linux-5.15.10-200.fc35.x86_64-x86_64-with-glibc2.34<br>
Is CUDA available: N/A<br>
CUDA runtime version: 11.5.119<br>
GPU models and configuration: GPU 0: Quadro RTX 4000<br>
Nvidia driver version: 495.29.05<br>
cuDNN version: Could not collect<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.21.2<br>
[conda] blas                      1.0                         mkl<br>
[conda] magma-cuda110             2.5.2                         1    pytorch<br>
[conda] mkl                       2021.4.0           h06a4308_640<br>
[conda] mkl-include               2021.4.0           h06a4308_640<br>
[conda] mkl-service               2.4.0            py39h7f8727e_0<br>
[conda] mkl_fft                   1.3.1            py39hd3c417c_0<br>
[conda] mkl_random                1.2.2            py39h51133e4_0<br>
[conda] numpy                     1.21.2           py39h20f2e39_0<br>
[conda] numpy-base                1.21.2           py39h79a1101_0<br>
(pytorch) [zf@192 pytorch_1_11]$</p>
<p><strong>Manually complete:</strong><br>
PyTorch version: 1.11.0a0<br>
Is debug build: Y<br>
CUDA used to build PyTorch: 11.5<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Fedora Linux 35 (Workstation Edition) (x86_64)<br>
GCC version: (GCC) 11.2.1 20211203 (Red Hat 11.2.1-7)<br>
Clang version: Could not collect<br>
CMake version: version 3.19.6<br>
Libc version: glibc-2.34</p>
<p>Python version: 3.9.7 (default, Sep 16 2021, 13:09:58)  [GCC 7.5.0] (64-bit runtime)<br>
Python platform: Linux-5.15.10-200.fc35.x86_64-x86_64-with-glibc2.34<br>
Is CUDA available: Y<br>
CUDA runtime version: 11.5.119<br>
GPU models and configuration: GPU 0: Quadro RTX 4000<br>
Nvidia driver version: 495.29.05<br>
cuDNN version: Could not collect<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.21.2<br>
[conda] blas                      1.0                         mkl<br>
[conda] magma-cuda110             2.5.2                         1    pytorch<br>
[conda] mkl                       2021.4.0           h06a4308_640<br>
[conda] mkl-include               2021.4.0           h06a4308_640<br>
[conda] mkl-service               2.4.0            py39h7f8727e_0<br>
[conda] mkl_fft                   1.3.1            py39hd3c417c_0<br>
[conda] mkl_random                1.2.2            py39h51133e4_0<br>
[conda] numpy                     1.21.2           py39h20f2e39_0<br>
[conda] numpy-base                1.21.2           py39h79a1101_0<br>
(pytorch) [zf@192 pytorch_1_11]$</p>
<p>I’ve had nvcc installed but don’t know if it worked:<br>
$ nvcc --version<br>
nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2021 NVIDIA Corporation<br>
Built on Thu_Nov_18_09:45:30_PST_2021<br>
Cuda compilation tools, release 11.5, V11.5.119<br>
Build cuda_11.5.r11.5/compiler.30672275_0</p>
<p>It may be caused by memory/cpu shortage:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/4/0483b8040caa60a9cec9c0dc18d96bfa68733417.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/0483b8040caa60a9cec9c0dc18d96bfa68733417" title="top"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/0483b8040caa60a9cec9c0dc18d96bfa68733417_2_548x500.jpeg" alt="top" data-base62-sha1="DW7cq6UCf42aDALRQZUJuLr9T9" width="548" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/0483b8040caa60a9cec9c0dc18d96bfa68733417_2_548x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/0483b8040caa60a9cec9c0dc18d96bfa68733417_2_822x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/4/0483b8040caa60a9cec9c0dc18d96bfa68733417.jpeg 2x" data-dominant-color="323232"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">top</span><span class="informations">979×893 156 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Would you please help? Thank you.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="fulltopic" data-post="1" data-topic="140100">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fulltopic/48/17884_2.png" class="avatar"> fulltopic:</div>
<blockquote>
<p>The build failed at very beginning (about 3/5xxx) when building protobuf related.<br>
The whole shell exited quietly without visible output.</p>
</blockquote>
</aside>
<p>This might point to an out of memory issue, but I would expect to see a “Killed” message in your terminal.<br>
Based on your screenshot also <code>cudafe++</code> and <code>cicc</code> processes are used, so <code>nvcc</code> is compiling something, which wouldn’t fit into the crash at the protobuf stage.</p>
<p>In any case, try to rebuild it via <code>MAX_JOBS=1 python setup.py install</code> and see if this would work.</p>

username: fulltopic
cooked: <p>I’ve tried several times. Once the build terminated when building nccl related and the screenshot was printed at that time. It’s also at the very beginning. I continued the building without clean, then it failed at protobuf compile.</p>
<p>Anyway, it’s more like a memory shortage issue as the process kswapd0 is the top1 consumer of <em>top</em> command output.</p>

username: fulltopic
cooked: <p>MAX_JOBS works. Thank you.</p>