Segmentation Fault while importing torchvision 0.24.0

jdumke: <p>Hello all,</p>
<p>I’m experimenting with the Vulkan backend, hence my GPU is currently not supported by ROCm. I built a custom Version of torch 2.9.0-rc2 and vision 0.24.0 with following build configuration:</p>
<pre><code class="lang-auto">#!/usr/bin/env bash
export CMAKE_BUILD_TYPE=DEBUG
export BUILD_LIBTORCH_CPU_WITH_DEBUG=1
export BUILD_FUNCTORCH=1
export USE_VULKAN=1
export USE_VULKAN_FP16_INFERENCE=1
export USE_VULKAN_RELAXED_PRECISION=1
export USE_VULKAN_SHADERC_RUNTIME=1
export USE_OPENMP=1
export USE_CUDA=0
export USE_ROCM=0
export USE_SYSTEM_CPUINFO=1
export USE_SYSTEM_FP16=1
export USE_SYSTEM_PYBIND11=1
export USE_SYSTEM_XNNPACK=1
export USE_SYSTEM_FXDIV=1
export USE_SYSTEM_SLEEF=1
export USE_SYSTEM_PTHREADPOOL=1
export MAX_JOBS=8
export CMAKE_GENERATOR="Ninja"
# for vision
export WITH_CUDA=0
export WITH_MPS=0
export WITH_PNG=1
export WITH_JPEG=1
export WITH_WEBP=1
export WITH_AVIF=1
</code></pre>
<p>The plain torch module is working fine, unlike the syntax inconsistency when using the vulkan backend and lacking methods (like device_count()). But when I’m simply importing torchvision a segmentation fault occurs:</p>
<pre><code class="lang-auto">nv) jds@padua:~/test/vtt$ python minimal_bug.py 
[padua:248026:0:248026] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
==== backtrace (tid: 248026) ====
 0  /lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2bc) [0x7f50eba3064c]
 1  /lib/x86_64-linux-gnu/libucs.so.0(+0x3182f) [0x7f50eba3082f]
 2  /lib/x86_64-linux-gnu/libucs.so.0(+0x319fa) [0x7f50eba309fa]
 3  /lib/x86_64-linux-gnu/libc.so.6(+0x3fdf0) [0x7f50ec677df0]
 4  /lib/x86_64-linux-gnu/libc.so.6(+0x1643c9) [0x7f50ec79c3c9]
 5  /home/jds/test/vtt/env/lib/python3.13/site-packages/torch/lib/libtorch_python.so(_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE12_M_constructIPcEEvT_S7_St20forward_iterator_tag+0xca) [0x7f50e27b0e5e]
 6  /home/jds/test/vtt/env/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so(_ZN3c104impl13OperatorEntry14registerKernelERKNS_10DispatcherESt8optionalINS_11DispatchKeyEENS_14KernelFunctionES5_INS0_12CppSignatureEESt10unique_ptrINS_14FunctionSchemaESt14default_deleteISC_EENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x378) [0x7f50ca3997a2]
 7  /home/jds/test/vtt/env/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so(_ZN3c1010Dispatcher12registerImplENS_12OperatorNameESt8optionalINS_11DispatchKeyEENS_14KernelFunctionES2_INS_4impl12CppSignatureEESt10unique_ptrINS_14FunctionSchemaESt14default_deleteISA_EENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x141) [0x7f50ca385203]
 8  /home/jds/test/vtt/env/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so(_ZNR5torch7Library5_implEPKcONS_11CppFunctionENS_17_RegisterOrVerifyE+0x413) [0x7f50ca429c7b]
 9  /home/jds/test/vtt/env/lib/python3.13/site-packages/torchvision/_C.so(+0x248de) [0x7f503e6288de]
10  /lib64/ld-linux-x86-64.so.2(+0x4fae) [0x7f50ec9acfae]
11  /lib64/ld-linux-x86-64.so.2(+0x507c) [0x7f50ec9ad07c]
12  /lib64/ld-linux-x86-64.so.2(_dl_catch_exception+0x106) [0x7f50ec9aa426]
13  /lib64/ld-linux-x86-64.so.2(+0xba28) [0x7f50ec9b3a28]
14  /lib64/ld-linux-x86-64.so.2(_dl_catch_exception+0x79) [0x7f50ec9aa399]
15  /lib64/ld-linux-x86-64.so.2(+0xbda8) [0x7f50ec9b3da8]
16  /lib/x86_64-linux-gnu/libc.so.6(+0x8f258) [0x7f50ec6c7258]
17  /lib64/ld-linux-x86-64.so.2(_dl_catch_exception+0x79) [0x7f50ec9aa399]
18  /lib64/ld-linux-x86-64.so.2(+0x24bf) [0x7f50ec9aa4bf]
19  /lib/x86_64-linux-gnu/libc.so.6(+0x8ed67) [0x7f50ec6c6d67]
20  /lib/x86_64-linux-gnu/libc.so.6(dlopen+0x69) [0x7f50ec6c7309]
21  /usr/lib/python3.13/lib-dynload/_ctypes.cpython-313-x86_64-linux-gnu.so(+0x17dfd) [0x7f50ec58ddfd]
22  python() [0x588a20]
23  python(_PyObject_MakeTpCall+0x27b) [0x54726b]
24  python(_PyEval_EvalFrameDefault+0xfce) [0x56270e]
25  python() [0x5a090c]
26  python(_PyObject_MakeTpCall+0x1d5) [0x5471c5]
27  python(_PyEval_EvalFrameDefault+0x2661) [0x563da1]
28  python(PyEval_EvalCode+0xcc) [0x55d48c]
29  python() [0x5da802]
30  python() [0x57e1cd]
31  python(_PyEval_EvalFrameDefault+0x4008) [0x565748]
32  python() [0x587b36]
33  python(PyObject_CallMethodObjArgs+0xe2) [0x5cc632]
34  python(PyImport_ImportModuleLevelObject+0x23b) [0x5cb42b]
35  python(_PyEval_EvalFrameDefault+0x577e) [0x566ebe]
36  python(PyEval_EvalCode+0xcc) [0x55d48c]
37  python() [0x5da802]
38  python() [0x57e1cd]
39  python(_PyEval_EvalFrameDefault+0x4008) [0x565748]
40  python() [0x587b36]
41  python(PyObject_CallMethodObjArgs+0xe2) [0x5cc632]
42  python(PyImport_ImportModuleLevelObject+0x23b) [0x5cb42b]
43  python(_PyEval_EvalFrameDefault+0x577e) [0x566ebe]
44  python(PyEval_EvalCode+0xcc) [0x55d48c]
45  python() [0x6ab8d1]
46  python() [0x6a899c]
47  python() [0x6b9943]
48  python() [0x6b93e3]
49  python() [0x6b921e]
50  python(Py_RunMain+0x3c1) [0x6b86f1]
51  python(Py_BytesMain+0x2b) [0x6838eb]
52  /lib/x86_64-linux-gnu/libc.so.6(+0x29ca8) [0x7f50ec661ca8]
53  /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7f50ec661d65]
54  python(_start+0x21) [0x682c81]
=================================
Segmentation fault
</code></pre>
<p>Zhe minimal script is:</p>
<pre><code class="lang-auto">#!/usr/bin/env python
import torch
import torchvision
</code></pre>
<p>Two things make wonder:</p>
<ol>
<li>Why torchvision depends on a  video-broadcasting-library (libucs)?</li>
<li>What causes the trouble?</li>
</ol>
<p>My system in bullets:</p>
<ul>
<li>Debian 13 (Trixie)</li>
<li>GCC  14.2.0</li>
<li>Python 3.13</li>
<li>libc 2.41</li>
<li>libstdc++ 12.4.5</li>
</ul>
<p>The post is quite long, if further information is needed I’ll provide it later.</p>
<p>Any suggestions, what went wrong?</p>
<p>Greets,</p>
<p>Jonathan</p>

ptrblck: <ol>
<li>It doesn’t and I guess you are loading UCS from UCX installed on your system. UCS has a fault handler and is showing up in the stacktrace as it did not remove the fault handling from the call trace. You can ignore it here since it’s not causing the segfault.</li>
<li>Unclear, but based on the calls I would start by checking if your custom builds are using the same CXX ABI first.</li>
</ol>

jdumke: <p>Thanks for your reply.</p>
<p>At a first glance the ABI versions seeming to fit, but on a closer look …</p>
<p>Here is what is nm says:</p>
<p>for _C.so:</p>
<pre><code class="lang-auto">                 U _ZTVN10__cxxabiv117__class_type_infoE@CXXABI_1.3
                 U _ZTVN10__cxxabiv119__pointer_type_infoE@CXXABI_1.3
                 U _ZTVN10__cxxabiv120__function_type_infoE@CXXABI_1.3
                 U _ZTVN10__cxxabiv120__si_class_type_infoE@CXXABI_1.3
</code></pre>
<p>for libtorch_cpu.so:</p>
<pre><code class="lang-auto">                 U __cxa_allocate_exception@CXXABI_1.3
                 U __cxa_bad_cast@CXXABI_1.3
                 U __cxa_bad_typeid@CXXABI_1.3
                 U __cxa_begin_catch@CXXABI_1.3
                 U __cxa_call_terminate@CXXABI_1.3.15
                 U __cxa_demangle@CXXABI_1.3
                 U __cxa_end_catch@CXXABI_1.3
                 U __cxa_free_exception@CXXABI_1.3
                 U __cxa_guard_abort@CXXABI_1.3
                 U __cxa_guard_acquire@CXXABI_1.3
                 U __cxa_guard_release@CXXABI_1.3
                 U __cxa_init_primary_exception@CXXABI_1.3.11
                 w __cxa_pure_virtual@CXXABI_1.3
                 U __cxa_rethrow@CXXABI_1.3
                 U __cxa_thread_atexit@CXXABI_1.3.7
                 U __cxa_throw_bad_array_new_length@CXXABI_1.3.8
                 U __cxa_throw@CXXABI_1.3
                 U __dynamic_cast@CXXABI_1.3
                 U __gxx_personality_v0@CXXABI_1.3
                 U _ZdaPvm@CXXABI_1.3.9
                 U _ZdlPvm@CXXABI_1.3.9
                 U _ZdlPvmSt11align_val_t@CXXABI_1.3.11
                 U _ZNSt15__exception_ptr13exception_ptr10_M_releaseEv@CXXABI_1.3.13
                 U _ZNSt15__exception_ptr13exception_ptr9_M_addrefEv@CXXABI_1.3.13
                 U _ZNSt15__exception_ptr13exception_ptrC1EPv@CXXABI_1.3.11
                 U _ZnwmSt11align_val_t@CXXABI_1.3.11
                 U _ZSt11_Hash_bytesPKvmm@CXXABI_1.3.5
                 U _ZSt17current_exceptionv@CXXABI_1.3.3
                 U _ZSt17rethrow_exceptionNSt15__exception_ptr13exception_ptrE@CXXABI_1.3.3
                 U _ZTIb@CXXABI_1.3
                 U _ZTIc@CXXABI_1.3
                 U _ZTId@CXXABI_1.3
                 U _ZTIf@CXXABI_1.3
                 U _ZTIh@CXXABI_1.3
                 U _ZTIi@CXXABI_1.3
                 U _ZTIj@CXXABI_1.3
                 U _ZTIl@CXXABI_1.3
                 U _ZTIm@CXXABI_1.3
                 U _ZTIN10__cxxabiv115__forced_unwindE@CXXABI_1.3.2
                 U _ZTIPKc@CXXABI_1.3
                 U _ZTIv@CXXABI_1.3
                 U _ZTVN10__cxxabiv116__enum_type_infoE@CXXABI_1.3
                 U _ZTVN10__cxxabiv117__class_type_infoE@CXXABI_1.3
                 U _ZTVN10__cxxabiv119__pointer_type_infoE@CXXABI_1.3
                 U _ZTVN10__cxxabiv120__function_type_infoE@CXXABI_1.3
                 U _ZTVN10__cxxabiv120__si_class_type_infoE@CXXABI_1.3
                 U _ZTVN10__cxxabiv121__vmi_class_type_infoE@CXXABI_1.3
</code></pre>
<p>for libtorch_python.so</p>
<pre><code class="lang-auto">                 U __cxa_allocate_exception@CXXABI_1.3
                 U __cxa_bad_typeid@CXXABI_1.3
                 U __cxa_begin_catch@CXXABI_1.3
                 U __cxa_demangle@CXXABI_1.3
                 U __cxa_end_catch@CXXABI_1.3
                 U __cxa_free_exception@CXXABI_1.3
                 U __cxa_guard_abort@CXXABI_1.3
                 U __cxa_guard_acquire@CXXABI_1.3
                 U __cxa_guard_release@CXXABI_1.3
                 U __cxa_init_primary_exception@CXXABI_1.3.11
                 w __cxa_pure_virtual@CXXABI_1.3
                 U __cxa_rethrow@CXXABI_1.3
                 U __cxa_thread_atexit@CXXABI_1.3.7
                 U __cxa_throw_bad_array_new_length@CXXABI_1.3.8
                 U __cxa_throw@CXXABI_1.3
                 U __dynamic_cast@CXXABI_1.3
                 U __gxx_personality_v0@CXXABI_1.3
                 U _ZdaPvm@CXXABI_1.3.9
                 U _ZdlPvm@CXXABI_1.3.9
                 U _ZdlPvmSt11align_val_t@CXXABI_1.3.11
                 U _ZNSt15__exception_ptr13exception_ptr10_M_releaseEv@CXXABI_1.3.13
                 U _ZNSt15__exception_ptr13exception_ptr9_M_addrefEv@CXXABI_1.3.13
                 U _ZNSt15__exception_ptr13exception_ptrC1EPv@CXXABI_1.3.11
                 U _ZnwmSt11align_val_t@CXXABI_1.3.11
                 U _ZSt11_Hash_bytesPKvmm@CXXABI_1.3.5
                 U _ZSt17current_exceptionv@CXXABI_1.3.3
                 U _ZSt17rethrow_exceptionNSt15__exception_ptr13exception_ptrE@CXXABI_1.3.3
                 U _ZTIa@CXXABI_1.3
                 U _ZTIb@CXXABI_1.3
                 U _ZTId@CXXABI_1.3
                 U _ZTIDn@CXXABI_1.3.5
                 U _ZTIf@CXXABI_1.3
                 U _ZTIh@CXXABI_1.3
                 U _ZTIi@CXXABI_1.3
                 U _ZTIj@CXXABI_1.3
                 U _ZTIl@CXXABI_1.3
                 U _ZTIm@CXXABI_1.3
                 U _ZTIN10__cxxabiv115__forced_unwindE@CXXABI_1.3.2
                 U _ZTIPKc@CXXABI_1.3
                 U _ZTIPKv@CXXABI_1.3
                 U _ZTIs@CXXABI_1.3
                 U _ZTISt16nested_exception@CXXABI_1.3.5
                 U _ZTIv@CXXABI_1.3
                 U _ZTVN10__cxxabiv116__enum_type_infoE@CXXABI_1.3
                 U _ZTVN10__cxxabiv117__array_type_infoE@CXXABI_1.3
                 U _ZTVN10__cxxabiv117__class_type_infoE@CXXABI_1.3
                 U _ZTVN10__cxxabiv119__pointer_type_infoE@CXXABI_1.3
                 U _ZTVN10__cxxabiv120__function_type_infoE@CXXABI_1.3
                 U _ZTVN10__cxxabiv120__si_class_type_infoE@CXXABI_1.3
                 U _ZTVN10__cxxabiv121__vmi_class_type_infoE@CXXABI_1.3
</code></pre>

jdumke: <p>Okay I found the problem.  The Building environment for torchvision 0.24.0 pulls torch 2.8.0 instead of locally built 2.9.0 while importing both modules the vtables don’t fit.</p>
<p>How can I convince pip to use my local version of torch and not a downloaded one?</p>

ptrblck: <p>Try to use <code>–no-build-isolation</code>.</p>

jdumke: <p>My actual pip call looks like:</p>
<pre><code class="lang-auto">pip wheel -w ~/test/vtt/dist/ --no-build-isolation --extra-index-url file:///home/jds/test/vtt/dist/simple/ ./vision/
</code></pre>
<p>I found the trick, due to the wheel-name:</p>
<pre data-code-wrap="bash"><code class="lang-bash">torch-2.9.0a0+gitd7a703e-cp313-cp313-linux_x86_64.whl
</code></pre>
<p>I’ve to set following env-variable</p>
<pre data-code-wrap="bash"><code class="lang-bash">export PYTORCH_VERSION=2.9.0a0
</code></pre>

jdumke: <p>Okay, now I’ve got a new problem: the built and installed torchvision can’t be loaded. It brakes down with an std:bad_alloc and I can’t figure out where it comes from. This covers all versions of vision I built so far (0.23 – 0.25).</p>