Pytorch training crash with coredump

Smith_David: <h2><a name="pystack-core-native-all-rootcodeemotion_lip_gencore-1" class="anchor" href="#pystack-core-native-all-rootcodeemotion_lip_gencore-1"></a>pystack core --native-all /root/code/emotion_lip_gen/core</h2>
<p>Using executable found in the core file: /usr/bin/python</p>
<p>Core file information:<br>
state: R zombie: True niceness: 0<br>
pid: 5277 ppid: 5276 sid: 31<br>
uid: 0 gid: 0 pgrp: 5276<br>
executable: python arguments: python train_syncnet.py --save_dir all_hubert_15 --dataset syncnet --project sy</p>
<p>The process died due receiving signal SIGABRT<br>
Traceback for thread 5277 [Has the GIL] (most recent call last):<br>
(C) File “???”, line 0, in _start (python)<br>
(C) File “…/csu/libc-start.c”, line 392, in __libc_start_main@<span class="mention">@GLIBC_2.34</span> (libc.so.6)<br>
(C) File “…/sysdeps/nptl/libc_start_call_main.h”, line 58, in __libc_start_call_main (libc.so.6)<br>
(C) File “???”, line 0, in Py_BytesMain (python)<br>
(C) File “???”, line 0, in Py_RunMain (python)<br>
(C) File “???”, line 0, in Py_FinalizeEx (python)<br>
(C) File “???”, line 0, in pybind11_object_dealloc (libtorch_python.so)<br>
(C) File “???”, line 0, in pybind11::detail::clear_instance(<em>object*) (libtorch_python.so)<br>
(C) File “???”, line 0, in pybind11::class</em>&lt;c10d::Reducer, std::shared_ptr<a>c10d::Reducer</a> &gt;::dealloc(pybind11::detail::value_and_holder&amp;) (libtorch_python.so)<br>
(C) File “???”, line 0, in std::_Sp_counted_base&lt;(__gnu_cxx::_Lock_policy)2&gt;::_M_release() (libtorch_python.so)<br>
(C) File “???”, line 0, in std::_Sp_counted_ptr&lt;c10d::Reducer*, (__gnu_cxx::_Lock_policy)2&gt;::<em>M_dispose() (libtorch_python.so)<br>
(C) File “???”, line 0, in c10d::Reducer::~Reducer() (libtorch_cpu.so)<br>
(C) File “???”, line 0, in c10::TensorImpl::~TensorImpl() (libc10.so)<br>
(C) File “???”, line 0, in c10::TensorImpl::~TensorImpl() (libc10.so)<br>
(C) File “???”, line 0, in c10::intrusive_ptr&lt;c10::StorageImpl, c10::detail::intrusive_target_default_null_type<a>c10::StorageImpl</a> &gt;::reset</em>() (libc10.so)<br>
(C) File “???”, line 0, in c10::StorageImpl::~StorageImpl() (libtorch_python.so)<br>
(C) File “???”, line 0, in _Unwind_Resume (libgcc_s.so.1)<br>
(C) File “???”, line 0, in __gxx_personality_v0 (libstdc++.so.6)<br>
(C) File “./stdlib/abort.c”, line 79, in abort (libc.so.6)<br>
(C) File “…/sysdeps/posix/raise.c”, line 26, in raise (libc.so.6)<br>
(C) File “./nptl/pthread_kill.c”, line 89, in pthread_kill@<span class="mention">@GLIBC_2.34</span> (libc.so.6)<br>
(C) File “./nptl/pthread_kill.c”, line 78, in __pthread_kill_internal (inlined) (libc.so.6)<br>
(C) File “./nptl/pthread_kill.c”, line 44, in __pthread_kill_implementation (inlined) (libc.so.6)</p>
<p>Traceback for thread 5408 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5404 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5405 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5406 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5411 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5410 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5409 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5423 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5414 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5412 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5415 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5413 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5424 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5418 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5429 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5420 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5417 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5421 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5422 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5425 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5427 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5437 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5432 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5434 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5436 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5435 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5441 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5439 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5445 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5442 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5440 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5428 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5444 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5447 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5450 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5449 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5448 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5451 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5454 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5456 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5446 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5455 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5452 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5416 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5459 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5457 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5460 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5461 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5463 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5465 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5675 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)<br>
(C) File “???”, line 0, in PyThread_acquire_lock_timed (python)<br>
(C) File “./nptl/sem_waitcommon.c”, line 183, in __new_sem_wait_slow64 (libc.so.6)<br>
(C) File “./nptl/futex-internal.c”, line 139, in __GI___futex_abstimed_wait_cancelable64 (libc.so.6)<br>
(C) File “./nptl/futex-internal.c”, line 87, in __futex_abstimed_wait_common (inlined) (libc.so.6)<br>
(C) File “./nptl/futex-internal.c”, line 57, in __futex_abstimed_wait_common64 (inlined) (libc.so.6)</p>
<p>Traceback for thread 5419 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5462 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5426 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5433 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5430 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5661 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)<br>
(C) File “…/sysdeps/unix/sysv/linux/poll.c”, line 29, in __poll (libc.so.6)</p>
<p>Traceback for thread 6334 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)<br>
(C) File “???”, line 0, in torch::autograd::python::PythonEngine::thread_init(int, std::shared_ptr<a>torch::autograd::ReadyQueue</a> const&amp;, bool) (libtorch_python.so)<br>
(C) File “???”, line 0, in torch::autograd::Engine::thread_init(int, std::shared_ptr<a>torch::autograd::ReadyQueue</a> const&amp;, bool) (libtorch_cpu.so)<br>
(C) File “???”, line 0, in torch::autograd::Engine::thread_main(std::shared_ptr<a>torch::autograd::GraphTask</a> const&amp;) (libtorch_cpu.so)<br>
(C) File “???”, line 0, in torch::autograd::ReadyQueue::pop() (libtorch_cpu.so)<br>
(C) File “???”, line 0, in std::condition_variable::wait(std::unique_lock<a>std::mutex</a>&amp;) (libstdc++.so.6)<br>
(C) File “./nptl/pthread_cond_wait.c”, line 627, in pthread_cond_wait@<span class="mention">@GLIBC_2.3.2</span> (libc.so.6)<br>
(C) File “./nptl/pthread_cond_wait.c”, line 503, in __pthread_cond_wait_common (inlined) (libc.so.6)<br>
(C) File “./nptl/futex-internal.c”, line 139, in __GI___futex_abstimed_wait_cancelable64 (libc.so.6)<br>
(C) File “./nptl/futex-internal.c”, line 87, in __futex_abstimed_wait_common (inlined) (libc.so.6)<br>
(C) File “./nptl/futex-internal.c”, line 57, in __futex_abstimed_wait_common64 (inlined) (libc.so.6)</p>
<p>Traceback for thread 5468 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)<br>
(C) File “???”, line 0, in c10d::detail::(anonymous namespace)::TCPStoreMasterDaemon::run() (libtorch_cpu.so)<br>
(C) File “…/sysdeps/unix/sysv/linux/poll.c”, line 29, in __poll (libc.so.6)</p>
<p>Traceback for thread 5453 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5673 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)<br>
(C) File “/pytorch/third_party/nccl/nccl/src/proxy.cc”, line 692, in ncclProxyProgress(void*) (libtorch_cuda.so)<br>
(C) File “/pytorch/third_party/nccl/nccl/src/proxy.cc”, line 552, in ncclProxyGetPostedOps (inlined) (libtorch_cuda.so)<br>
(C) File “./nptl/pthread_cond_wait.c”, line 627, in pthread_cond_wait@<span class="mention">@GLIBC_2.3.2</span> (libc.so.6)<br>
(C) File “./nptl/pthread_cond_wait.c”, line 503, in __pthread_cond_wait_common (inlined) (libc.so.6)<br>
(C) File “./nptl/futex-internal.c”, line 139, in __GI___futex_abstimed_wait_cancelable64 (libc.so.6)<br>
(C) File “./nptl/futex-internal.c”, line 87, in __futex_abstimed_wait_common (inlined) (libc.so.6)<br>
(C) File “./nptl/futex-internal.c”, line 57, in __futex_abstimed_wait_common64 (inlined) (libc.so.6)</p>
<p>Traceback for thread 5458 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 6335 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)<br>
(C) File “???”, line 0, in torch::autograd::python::PythonEngine::thread_init(int, std::shared_ptr<a>torch::autograd::ReadyQueue</a> const&amp;, bool) (libtorch_python.so)<br>
(C) File “???”, line 0, in torch::autograd::Engine::thread_init(int, std::shared_ptr<a>torch::autograd::ReadyQueue</a> const&amp;, bool) (libtorch_cpu.so)<br>
(C) File “???”, line 0, in torch::autograd::Engine::thread_main(std::shared_ptr<a>torch::autograd::GraphTask</a> const&amp;) (libtorch_cpu.so)<br>
(C) File “???”, line 0, in torch::autograd::ReadyQueue::pop() (libtorch_cpu.so)<br>
(C) File “???”, line 0, in std::condition_variable::wait(std::unique_lock<a>std::mutex</a>&amp;) (libstdc++.so.6)<br>
(C) File “./nptl/pthread_cond_wait.c”, line 627, in pthread_cond_wait@<span class="mention">@GLIBC_2.3.2</span> (libc.so.6)<br>
(C) File “./nptl/pthread_cond_wait.c”, line 503, in __pthread_cond_wait_common (inlined) (libc.so.6)<br>
(C) File “./nptl/futex-internal.c”, line 139, in __GI___futex_abstimed_wait_cancelable64 (libc.so.6)<br>
(C) File “./nptl/futex-internal.c”, line 87, in __futex_abstimed_wait_common (inlined) (libc.so.6)<br>
(C) File “./nptl/futex-internal.c”, line 57, in __futex_abstimed_wait_common64 (inlined) (libc.so.6)</p>
<p>Traceback for thread 5464 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5431 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5672 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)<br>
(C) File “/pytorch/third_party/nccl/nccl/src/proxy.cc”, line 1058, in ncclProxyService(void*) (libtorch_cuda.so)<br>
(C) File “…/sysdeps/unix/sysv/linux/poll.c”, line 29, in __poll (libc.so.6)</p>
<p>Traceback for thread 5438 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 6336 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)<br>
(C) File “???”, line 0, in PyThread_acquire_lock_timed (python)<br>
(C) File “./nptl/sem_waitcommon.c”, line 183, in __new_sem_wait_slow64 (libc.so.6)<br>
(C) File “./nptl/futex-internal.c”, line 139, in __GI___futex_abstimed_wait_cancelable64 (libc.so.6)<br>
(C) File “./nptl/futex-internal.c”, line 87, in __futex_abstimed_wait_common (inlined) (libc.so.6)<br>
(C) File “./nptl/futex-internal.c”, line 57, in __futex_abstimed_wait_common64 (inlined) (libc.so.6)</p>
<p>Traceback for thread 5443 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5466 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5666 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)<br>
(C) File “./nptl/pthread_cond_wait.c”, line 652, in pthread_cond_timedwait@<span class="mention">@GLIBC_2.3.2</span> (libc.so.6)<br>
(C) File “./nptl/pthread_cond_wait.c”, line 503, in __pthread_cond_wait_common (inlined) (libc.so.6)<br>
(C) File “./nptl/futex-internal.c”, line 139, in __GI___futex_abstimed_wait_cancelable64 (libc.so.6)<br>
(C) File “./nptl/futex-internal.c”, line 87, in __futex_abstimed_wait_common (inlined) (libc.so.6)<br>
(C) File “./nptl/futex-internal.c”, line 57, in __futex_abstimed_wait_common64 (inlined) (libc.so.6)</p>
<p>Traceback for thread 6338 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)<br>
(C) File “???”, line 0, in PyThread_acquire_lock_timed (python)<br>
(C) File “./nptl/sem_waitcommon.c”, line 183, in __new_sem_wait_slow64 (libc.so.6)<br>
(C) File “./nptl/futex-internal.c”, line 139, in __GI___futex_abstimed_wait_cancelable64 (libc.so.6)<br>
(C) File “./nptl/futex-internal.c”, line 87, in __futex_abstimed_wait_common (inlined) (libc.so.6)<br>
(C) File “./nptl/futex-internal.c”, line 57, in __futex_abstimed_wait_common64 (inlined) (libc.so.6)</p>
<p>Traceback for thread 5407 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)</p>
<p>Traceback for thread 5663 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)<br>
(C) File “???”, line 0, in c10d::ProcessGroupNCCL::ncclCommWatchdog() (libtorch_cuda.so)<br>
(C) File “???”, line 0, in c10d::ProcessGroupNCCL::ncclCommWatchdogInternal() (libtorch_cuda.so)<br>
(C) File “./nptl/pthread_cond_wait.c”, line 652, in pthread_cond_timedwait@<span class="mention">@GLIBC_2.3.2</span> (libc.so.6)<br>
(C) File “./nptl/pthread_cond_wait.c”, line 503, in __pthread_cond_wait_common (inlined) (libc.so.6)<br>
(C) File “./nptl/futex-internal.c”, line 139, in __GI___futex_abstimed_wait_cancelable64 (libc.so.6)<br>
(C) File “./nptl/futex-internal.c”, line 87, in __futex_abstimed_wait_common (inlined) (libc.so.6)<br>
(C) File “./nptl/futex-internal.c”, line 57, in __futex_abstimed_wait_common64 (inlined) (libc.so.6)</p>
<p>Traceback for thread 5664 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)<br>
(C) File “…/sysdeps/unix/sysv/linux/poll.c”, line 29, in __poll (libc.so.6)</p>
<p>Traceback for thread 5659 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)<br>
(C) File “???”, line 0, in c10d::detail::(anonymous namespace)::TCPStoreWorkerDaemon::run() (libtorch_cpu.so)<br>
(C) File “…/sysdeps/unix/sysv/linux/poll.c”, line 29, in __poll (libc.so.6)</p>
<p>Traceback for thread 6613 <span class="chcklst-box fa fa-square-o fa-fw"></span> (most recent call last):<br>
(C) File “…/sysdeps/unix/sysv/linux/x86_64/clone3.S”, line 81, in __clone3 (libc.so.6)<br>
(C) File “./nptl/pthread_create.c”, line 442, in start_thread (libc.so.6)<br>
(C) File “…/sysdeps/unix/sysv/linux/poll.c”, line 29, in __poll (libc.so.6)</p>