Setting GLIBCXX_USE_CXX11_ABI to 0

Eli_Draizen: <p>Hello,</p>
<p>I am trying to run a 3D Unet from MinkowskiEngine using DistributedDataParallel, running from a singularity container with pytorch 1.6 installed via conda. The cluster I am using has P100, V100, and K80 GPUs.</p>
<ol>
<li>1 GPU, no parallelization: Model works on all GPU types</li>
<li>DataParallel with 1 node, 4 gpu: Model works on all GPU types</li>
<li>DistributedDataParallel with 1 node, 4 GPU: Model only works on P100 and V100, but fails on K80 with the following error:</li>
</ol>
<pre><code class="lang-auto">terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: the launch timed out and was terminated
Exception raised from create_event_internal at /opt/conda/conda-bld/pytorch_1595629427478/work/c10/cuda/CUDACachingAllocator.cpp:687 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x4d (0x7f7df16d977d in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0xb5d (0x7f7df1929d9d in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7f7df16c5b1d in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #3: std::vector&lt;at::Tensor, std::allocator&lt;at::Tensor&gt; &gt;::~vector() + 0x5c (0x7f7dfc36669c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
...
</code></pre>
<p>I tried again using horovod (with nccl, infiniband libraries, etc inside the singularity container):</p>
<ol>
<li>DistributedDataParallel with 1 node, 4 GPU: Model works on all GPU types</li>
<li>DistributedDataParallel with 2 nodes, 4 K80 GPUs each (cluster only supports multiple nodes with K80s): Fails with the same error as above.</li>
</ol>
<p>According to <a href="https://github.com/pytorch/pytorch/issues/13541" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/13541</a>, it has to do with GLIBCXX_USE_CXX11_ABI set to 1. How can I change it to 0?</p>
<p>I am now building pytorch from source, but can’t seem to change it 0. Here is what I have tried:</p>
<pre><code class="lang-auto"> git clone https://github.com/pytorch/pytorch.git
 cd pytorch
 git checkout tags/v1.6.0
 git submodule update --init --recursive
</code></pre>
<ol>
<li>Set TORCH_CXX_FLAGS as environmental var:</li>
</ol>
<pre><code class="lang-auto">TORCH_CUDA_ARCH_LIST="3.5 5.2 6.0 6.1" TORCH_NVCC_FLAGS="-Xfatbin -compress-all" \
    CMAKE_PREFIX_PATH="$(dirname $(which conda))/../" TORCH_CXX_FLAGS="-D_GLIBCXX_USE_CXX11_ABI=0" \
    MAX_JOBS=16 python setup.py install
</code></pre>
<ol start="2">
<li>Modify TorchConfig.cmake.in, comment out ABI line (Issue13541 <span class="hashtag">#issuecomment-512756771</span>)</li>
</ol>
<pre><code class="lang-auto">sed -i 's/set(TORCH_CXX_FLAGS/#set(TORCH_CXX_FLAGS/g' cmake/TorchConfig.cmake.in
TORCH_CUDA_ARCH_LIST="3.5 5.2 6.0 6.1" TORCH_NVCC_FLAGS="-Xfatbin -compress-all" \
    CMAKE_PREFIX_PATH="$(dirname $(which conda))/../" \
    MAX_JOBS=16 python setup.py install
</code></pre>
<ol start="3">
<li>Modify TorchConfig.cmake.in, set to 0:</li>
</ol>
<pre><code class="lang-auto">sed -i 's/@GLIBCXX_USE_CXX11_ABI@/0/g' cmake/TorchConfig.cmake.in
TORCH_CUDA_ARCH_LIST="3.5 5.2 6.0 6.1" TORCH_NVCC_FLAGS="-Xfatbin -compress-all" \
    CMAKE_PREFIX_PATH="$(dirname $(which conda))/../" \
    MAX_JOBS=16 python setup.py install
</code></pre>
<ol start="4">
<li>Add line to CMakeLists.txt</li>
</ol>
<pre><code class="lang-auto">echo 'set(TORCH_CXX_FLAGS "-D_GLIBCXX_USE_CXX11_ABI=0")' &gt; no_abi.txt
cat no_abi.txt torch/CMakeLists.txt &gt; torch/CMakeLists.txt.1
rm torch/CMakeLists.txt
mv torch/CMakeLists.txt.1 torch/CMakeLists.txt
rm no_abi.txt
TORCH_CUDA_ARCH_LIST="3.5 5.2 6.0 6.1" TORCH_NVCC_FLAGS="-Xfatbin -compress-all" \
    CMAKE_PREFIX_PATH="$(dirname $(which conda))/../" \
    MAX_JOBS=16 python setup.py install
</code></pre>
<ol start="5">
<li>Combinations of 1, 3, and 4.</li>
</ol>
<p>Finally, I check if it is correct by running<br>
<code>python -c "import torch; print('PYTORCH USING CXX11_ABI =', torch._C._GLIBCXX_USE_CXX11_ABI )"</code>,<br>
but always outputs:<br>
<code>PYTORCH USING CXX11_ABI = True </code></p>
<p>I am also now getting an error when importing MinkowskiEngine:</p>
<pre><code class="lang-auto">ImportError: /opt/conda/lib/python3.7/site-packages/MinkowskiEngine-0.4.3-py3.7-linux-x86_64.egg/MinkowskiEngineBackend.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN3c105ErrorC1ENS_14SourceLocationENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
</code></pre>
<p>which also has to do with GLIBCXX_USE_CXX11_ABI (<a href="https://discuss.pytorch.org/t/cannot-build-pybind11-libtorch-code-with-cmake/94377" class="inline-onebox">Cannot build pybind11/libtorch code with cmake</a>)</p>
<p>If you have any suggestion about rebuilding pytorch with -D_GLIBCXX_USE_CXX11_ABI=0, I would really appreciate it.</p>
<p>Here is my Singulairty definition file in case it is helpful: <code>github.com/edraizen/SingularityTorch/blob/rivana_pytorch/Singularity</code></p>
<p>Thanks for your help!</p>

yselivonchyk: <p>I am not sure what, but some of this worked:</p>
<pre><code class="lang-auto">export GLIBCXX_USE_CXX11_ABI=0
export CXXFLAGS="-D_GLIBCXX_USE_CXX11_ABI=0"
export TORCH_CXX_FLAGS="-D_GLIBCXX_USE_CXX11_ABI=0"
#export DEBUG=1
export USE_GLOO=0
export REL_WITH_DEB_INFO=1
export USE_DISTRIBUTED=0
</code></pre>
<p>resulting in python -c “import torch; print(‘PYTORCH USING CXX11_ABI =’, torch._C._GLIBCXX_USE_CXX11_ABI )” → false</p>

jie.hang: <p>@ <a href="https://discuss.pytorch.org/u/yselivonchyk">yselivonchyk</a><a href="https://discuss.pytorch.org/u/yselivonchyk">Eugene</a><br>
I try to compile pytorch1.8.1 from source as same as you, but occurs undefined reference error:</p>
<pre><code class="lang-auto">/home/jhang/bi/hj/pytorch_ori_v1.8/build/lib/libtorch_cuda.so: undefined reference to `gloo::getCudaPCIBusID(int)'
/home/jhang/bi/hj/pytorch_ori_v1.8/build/lib/libtorch_cuda.so: undefined reference to `gloo::EnforceNotMet::EnforceNotMet(char const*, int, char const*, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;)'
collect2: error: ld returned 1 exit status
</code></pre>

jie.hang: <p>@ <a href="https://discuss.pytorch.org/u/Eli_Draizen">Eli_Draizen</a><br>
I try to add flag in CMakeLists.txt, add_definitions(-D_GLIBCXX_USE_CXX11_ABI=0), and compile, torch._C._GLIBCXX_USE_CXX11_ABI=false.</p>