Conflict between libtorch and libboost, libtorch and Apache arrow

Yu_Chen: <p>Hi,</p>
<p>I’m trying to build a training program with libtorch and libboost. But it seems libtorch has conflicts with libboost. The following example has segmentation fault at libtorch’s both CPU version and CUDA version.  Here are <a href="https://gist.github.com/cycyyy/86b224ab70bd032cd70bcf7b8dab1c04" rel="noopener nofollow ugc">files</a>. Did I make any mistake in CMakelists?</p>
<p>Apache arrow also has similar issues. It will have compile errors when I import “torch/torch.h” and arrow related header files together.</p>

glaringlee: <p><a class="mention" href="/u/yu_chen">@Yu_Chen</a><br>
Will your program success if you remove all the libtorch related stuff from CMakeLists.txt?<br>
Which version of libtorch? Can you post your error message (stack)?</p>

Yu_Chen: <p><a class="mention" href="/u/glaringlee">@glaringlee</a><br>
Yes. The program will execute when I removed libtorch from the CMakelists.<br>
I tested on libtorch version on cu102 1.8.0.dev20201207 + boost version 1.67 and on cpu 1.8.0.dev20201221 + boost version 1.72.</p>
<p>The segmentation fault happens on <a href="https://gist.github.com/cycyyy/86b224ab70bd032cd70bcf7b8dab1c04#file-main2-cpp-L12" rel="noopener nofollow ugc">here</a> and the error message is<br>
“Program received signal SIGSEGV, Segmentation fault.<br>
0x00007ffff79b6b10 in boost::filesystem::path::m_append_separator_if_needed() () from /usr/lib/libboost_filesystem.so.1.72.0”</p>
<p>I can’t post Arrow errors here because there are too many (500+). But when I removed the “torch/torch.h”, it can compile successfully.</p>
<p>Similar issues also happen on RocksDB. See comments in this <a href="https://gist.github.com/cycyyy/86b224ab70bd032cd70bcf7b8dab1c04" rel="noopener nofollow ugc">file</a>. Did I make any mistake?<br>
Or this issue may be caused by “GLIBCXX_USE_CXX11_ABI” flag.</p>

Yu_Chen: <p>The issue is because of -D_GLIBCXX_USE_CXX11_ABI=0 flag. Here is the related <a href="https://discuss.pytorch.org/t/issues-linking-with-libtorch-c-11-abi/29510">disscusion</a>. Everything is fine when I download the [libtorch-cxx11-abi-shared-without-deps-latest.zip] version.</p>
<p>But the conflict between Arrow and libtorch is still there.</p>

glaringlee: <p><a class="mention" href="/u/yu_chen">@Yu_Chen</a><br>
We have 2 versions of libtorch available on <a href="http://pytorch.org" rel="noopener nofollow ugc">pytorch.org</a>. one is ABI version, we also has a pre-abi version which is compiled with -D_GLIBCXX_USE_CXX11_ABI=0.<br>
You can download the pre-abi version and compile everything else with -D_GLIBCXX_USE_CXX11_ABI=0. Does that works for you?</p>

Yu_Chen: <p>Yes. This solves the liboost conflict. But I still can’t use Apache arrow with libtorch.</p>

glaringlee: <p>I never use arrow with libtorch. If your compile error is 500+ lines, can you put it into <a href="http://gist.github.com" rel="noopener nofollow ugc">gist.github.com</a> or similar? I can take a look at, but can not guarantee I have the solution though <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Yu_Chen: <p>I think it’s because libtorch and Arrow have some same micros.<br>
Currently, I bypass the issue by separating the project into two parts.<br>
<a href="https://gist.github.com/cycyyy/b08a1bb0a0ea54361dd6ed6f1dfb46e7" rel="noopener nofollow ugc">Here</a> is the error after I add the “<span class="hashtag">#include</span> &lt;parquet/file_reader.h&gt;” in my file.</p>

glaringlee: <p>The duplicated file is c10/util/variant.h vs /usr/include/arrow/vendored/variant.hpp</p>
<p>I think they both twisted from boost lib.<br>
MPARK_LIB_HPP is defined in c10/util/variant.h first, so the code block for MPARK_LIB_HPP won’t be included from /usr/include/arrow/vendored/variant.hpp. That’s why you have so many <code>lib is not declared</code>.<br>
I think probably separate into 2 parts is the correct way given this situation.</p>