tranformers.Trainer.train() hangs with Llama3 base model
Author: yangyangyyy123
Pull request: no
Labels: name: trainer

### System Info

transformers version :  '4.39.2'




### Who can help?


we used our existing fine-tune code, which worked with llama1 and llama2 base models

```

    trainer = Trainer(
        model=model,
        tokenizer=tokenizer,
        args=training_args,
        **data_module,
        callbacks=[ManifoldTensorBoardLoggerCallback()],
    )
    trainer.train()
```
but once the trainer starts fine-tuning from a llama3-8B, it barely makes any progress ("only prints the 0% on the progress status once, and then never updates it) after 5 hours. previously with llama2-7B, it runs through 40% of our examples within 25 minutes

I see that it's the same code as recommended by wiki
https://huggingface.co/docs/transformers/training

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

use the examples from https://huggingface.co/docs/transformers/training

start fine-tune on Llama3

### Expected behavior

should see progress bar printed out, but actually it stays 0% after 5 hours

Comments:
cc @ArthurZucker @muellerzr 

The progress bar is stuck, but is the model training / gpu used / memory and compute consumption updated? 

@amyeroberts ArthurZucker no, nothing is reported. but from out of channel , on our own dashboard, I can see GPU consumption at expected values. 

also there is no periodic report of model training loss progression either

trying to get a runnable repro based on the wiki's trainer code https://studiolab.sagemaker.aws/import/github/huggingface/notebooks/blob/main/transformers_doc/en/pytorch/training.ipynb

but even the "yelp_reviews_full" datasets loading is taking too much time, is there a much smaller dataset (say 100 rows) ?
or I could manually write a fake dataset ? the "dataset" does not seem to be a dict  : 
![Uploading Screenshot 2024-04-22 at 11.25.40 AM.png…]()



saw this "heartbeat monitor failed" error


```

[3]:E0423 02:10:44.182842  8355 ProcessGroupNCCL.cpp:1329] [PG 0 (default_pg) Rank 3] Heartbeat monitor timed out! Process will be terminated after dumping debug info. workMetaList_.size()=0
[3]:I0423 02:10:44.182893  8355 ThreadStackTrace.cpp:149] Successfully installed signal handler with 39 signo.
[3]:I0423 02:10:44.182879  8355 ProcessGroupNCCL.cpp:1333] Dumping c++ stacktraces: Stack trace for 186 thread(s) [10924 pt_main_thread, 10925 pt_main_thread, 10926 pt_main_thread, 10927 pt_main_thread, 10928 pt_main_thread, 10929 pt_main_thread, 10930 pt_main_thread, 10931 pt_main_thread, 10932 pt_main_thread, 10933 pt_main_thread, 10934 pt_main_thread, 10935 pt_main_thread, 10936 pt_main_thread, 10937 pt_main_thread, 10938 pt_main_thread, 10939 pt_main_thread, 10940 pt_main_thread, 10941 pt_main_thread, 10942 pt_main_thread, 10943 pt_main_thread, 10944 pt_main_thread, 10945 pt_main_thread, 10946 pt_main_thread, 10947 pt_main_thread, 10948 pt_main_thread, 10949 pt_main_thread, 10950 pt_main_thread, 10951 pt_main_thread, 10952 pt_main_thread, 10953 pt_main_thread, 10954 pt_main_thread, 10955 pt_main_thread, 10956 pt_main_thread, 10957 pt_main_thread, 10958 pt_main_thread, 10959 pt_main_thread, 10960 pt_main_thread, 10961 pt_main_thread, 10962 pt_main_thread, 10963 pt_main_thread, 10964 pt_main_thread, 10965 pt_main_thread, 10966 pt_main_thread, 10967 pt_main_thread, 10968 pt_main_thread, 10969 pt_main_thread, 10970 pt_main_thread, 10971 pt_main_thread, 10972 pt_main_thread, 10973 pt_main_thread, 10974 pt_main_thread, 10975 pt_main_thread, 10976 pt_main_thread, 10977 pt_main_thread, 10978 pt_main_thread, 10979 pt_main_thread, 10980 pt_main_thread, 10981 pt_main_thread, 10982 pt_main_thread, 10983 pt_main_thread, 10984 pt_main_thread, 10985 pt_main_thread, 10986 pt_main_thread, 10987 pt_main_thread, 10988 pt_main_thread, 10989 pt_main_thread, 10990 pt_main_thread, 10991 pt_main_thread, 10992 pt_main_thread, 10993 pt_main_thread, 10994 pt_main_thread, 10995 pt_main_thread, 10996 pt_main_thread, 10997 pt_main_thread, 10998 pt_main_thread, 10999 pt_main_thread, 11000 pt_main_thread, 11001 pt_main_thread, 11002 pt_main_thread, 11003 pt_main_thread, 11004 pt_main_thread, 11005 pt_main_thread, 11006 pt_main_thread, 11007 pt_main_thread, 11008 pt_main_thread, 11009 pt_main_thread, 11010 pt_main_thread, 11011 pt_main_thread, 11012 pt_main_thread, 11013 pt_main_thread, 11014 pt_main_thread, 11015 pt_main_thread, 11016 pt_main_thread, 11017 pt_main_thread, 11018 pt_main_thread, 11019 pt_main_thread, 11020 pt_main_thread, 11021 pt_main_thread, 11022 pt_main_thread, 11023 pt_main_thread, 11024 pt_main_thread, 11025 pt_main_thread, 11026 pt_main_thread, 11027 pt_main_thread, 11028 pt_main_thread, 11029 pt_main_thread, 11030 pt_main_thread, 11031 pt_main_thread, 11032 pt_main_thread, 11033 pt_main_thread, 11034 pt_main_thread, 11035 pt_main_thread, 11036 pt_main_thread, 11037 pt_main_thread, 11038 pt_main_thread, 11039 pt_main_thread, 11040 pt_main_thread, 11041 pt_main_thread, 11042 pt_main_thread, 11043 pt_main_thread, 11044 pt_main_thread, 11045 pt_main_thread, 11046 pt_main_thread, 11047 pt_main_thread, 11048 pt_main_thread, 11049 pt_main_thread, 11050 pt_main_thread, 11051 pt_main_thread, 11052 pt_main_thread, 11053 pt_main_thread, 11054 pt_main_thread, 11055 pt_main_thread, 11056 pt_main_thread, 11057 pt_main_thread, 11058 pt_main_thread, 11059 pt_main_thread, 11060 pt_main_thread, 11061 pt_main_thread, 11062 pt_main_thread, 11063 pt_main_thread, 11064 pt_main_thread, 11065 pt_main_thread, 11066 pt_main_thread, 11067 pt_main_thread, 11068 pt_main_thread, 11069 pt_main_thread, 11070 pt_main_thread, 11071 pt_main_thread, 11072 pt_main_thread, 11073 pt_main_thread, 11074 pt_main_thread, 11075 pt_main_thread, 11076 pt_main_thread, 11077 pt_main_thread, 11078 pt_main_thread, 11079 pt_main_thread, 11080 pt_main_thread, 11081 pt_main_thread, 11082 pt_main_thread, 11083 pt_main_thread, 11084 pt_main_thread, 11085 pt_main_thread, 11086 pt_main_thread, 11087 pt_main_thread, 11088 pt_main_thread, 11089 pt_main_thread, 11090 pt_main_thread, 11091 pt_main_thread, 11092 pt_main_thread, 11093 pt_main_thread, 11094 pt_main_thread, 11095 pt_main_thread, 11096 pt_main_thread, 11097 pt_main_thread, 11098 pt_main_thread, 11099 pt_main_thread, 11100 pt_main_thread, 11101 pt_main_thread, 11102 pt_main_thread, 11103 pt_main_thread, 11104 pt_main_thread, 11105 pt_main_thread, 11106 pt_main_thread, 11107 pt_main_thread, 11108 pt_main_thread, 11109 pt_main_thread]:
[3]:    @ 00000000001257b9 syscall
[3]:                       /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/misc/../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
[3]:    @ 0000000000824ed1 <std::sys::pal::unix::locks::futex_condvar::Condvar>::wait
[3]:    @ 00000000007e1302 <rayon_core::sleep::Sleep>::sleep::<<rayon_core::registry::WorkerThread>::wait_until_cold::{closure#0}>
[3]:    @ 00000000007dd142 <rayon_core::registry::WorkerThread>::wait_until_cold
[3]:    @ 00000000007db101 <rayon_core::registry::ThreadBuilder>::run
[3]:    @ 00000000007e40f9 std::sys_common::backtrace::__rust_begin_short_backtrace::<<rayon_core::registry::DefaultSpawn as rayon_core::registry::ThreadSpawn>::spawn::{closure#0}, ()>
[3]:    @ 00000000007def5f <<std::thread::Builder>::spawn_unchecked_<<rayon_core::registry::DefaultSpawn as rayon_core::registry::ThreadSpawn>::spawn::{closure#0}, ()>::{closure#1} as core::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
[3]:    @ 00000000007f38d8 <std::sys::pal::unix::thread::Thread>::new::thread_start
[3]:    @ 000000000009ac28 start_thread
[3]:                       /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/nptl/pthread_create.c:434
[3]:    @ 000000000012cfbb __clone3
[3]:                       /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
[3]:
[3]:Stack trace for 75 thread(s) [6805 IOThreadPool0, 6876 CfgrIO0, 6903 FalconClientThr, 6941 ScribeTC0, 8136 MfClientThreadP, 8137 MfClientThreadP, 8138 MfClientThreadP, 8139 MfClientThreadP, 8140 MfClientThreadP, 8141 MfClientThreadP, 8143 SR_TLSConnectio, 8144 SR_TLSConnectio, 8145 SR_TLSConnectio, 8146 SR_TLSConnectio, 8147 SR_TLSConnectio, 8148 SR_TLSConnectio, 8149 SR_TLSConnectio, 8150 SR_TLSConnectio, 8151 SR_TLSConnectio, 8152 SR_TLSConnectio, 8153 SR_TLSConnectio, 8154 SR_TLSConnectio, 8155 SR_TLSConnectio, 8156 SR_TLSConnectio, 8157 SR_TLSConnectio, 8158 SR_TLSConnectio, 8159 SR_TLSConnectio, 8160 SR_TLSConnectio, 8161 SR_TLSConnectio, 8162 SR_TLSConnectio, 8163 SR_TLSConnectio, 8164 SR_TLSConnectio, 8165 SR_TLSConnectio, 8166 SR_TLSConnectio, 8167 SR_TLSConnectio, 8168 SR_TLSConnectio, 8169 SR_TLSConnectio, 8170 SR_TLSConnectio, 8171 SR_TLSConnectio, 8172 SR_TLSConnectio, 8173 SR_TLSConnectio, 8174 SR_TLSConnectio, 8175 MfClientThreadP, 8176 MfClientThreadP, 8177 MfClientThreadP, 8178 MfClientThreadP, 8179 MfClientThreadP, 8180 MfClientThreadP, 8181 MfClientThreadP, 8182 MfClientThreadP, 8183 MfClientThreadP, 8184 MfClientThreadP, 8185 MfClientThreadP, 8186 MfClientThreadP, 8189 IOThreadPool0, 8192 ScribeTC0, 8203 MfClientThreadP, 8213 MfClientThreadP, 8214 MfClientThreadP, 8215 MfClientThreadP, 8216 MfClientThreadP, 8217 MfClientThreadP, 8224 MfClientThreadP, 8225 MfClientThreadP, 8226 MfClientThreadP, 8227 MfClientThreadP, 8228 MfClientThreadP, 8229 MfClientThreadP, 8230 MfClientThreadP, 8231 MfClientThreadP, 10261 IOThreadPool0, 10332 IOThreadPool0, 11857 IOThreadPool0, 11860 ScribeTC0, 11869 IOThreadPool0]:
[3]:    @ 000000000012c5f2 epoll_wait
[3]:                       /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/misc/../sysdeps/unix/sysv/linux/epoll_wait.c:30
[3]:    @ 000000001ac77c72 (unknown)
[3]:    @ 000000001ac75dff event_base_loop
[3]:    @ 00000000120cfac9 folly::EventBase::loopMain(int, folly::EventBase::LoopOptions)
[3]:    @ 00000000120cf5b9 folly::EventBase::loopBody(int, folly::EventBase::LoopOptions)
[3]:    @ 0000000013c59dc5 folly::EventBase::loopForever()
[3]:    @ 0000000013c59b18 folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)
[3]:    @ 000000001349f84c std::__invoke_result<void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&>::type std::__invoke<void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&>(void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&)
[3]:    @ 00000000000df4e4 execute_native_thread_routine
[3]:                       /home/engshare/third-party2/libgcc/11.x/src/gcc-11.x/x86_64-facebook-linux/libstdc++-v3/src/c++11/../../../.././libstdc++-v3/src/c++11/thread.cc:82
[3]:    @ 000000000009ac28 start_thread
[3]:                       /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/nptl/pthread_create.c:434
[3]:    @ 000000000012cfbb __clone3
[3]:                       /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
[3]:
[3]:Stack trace for 41 thread(s) [6796 FuncSched, 6865 OBC-fls, 6937 stat_manager-ag, 6993 FuncSched, 6997 FuncSched, 7000 FuncSched, 7704 servicedata-pub, 7862 CryptoPubManage, 8135 SRPoolCleanup, 8187 FuncSched, 8188 FuncSched, 8190 stat_manager-ag, 8194 FuncSched, 8197 FuncSched, 8198 FuncSched, 8200 ratelim::update, 8201 ratelim::bumpDe, 8202 ratelim::update, 8211 carbon-global-s, 8237 fci-throttle-ch, 8242 fci-usage-flush, 8243 fci-throttle-ch, 8244 fci-usage-flush, 8356 FuncSched, 8357 FuncSched, 8359 stat_manager-ag, 8360 stat_manager-ag, 8378 FuncSched, 8379 FuncSched, 8381 FuncSched, 8399 FuncSched, 8401 stat_manager-ag, 8409 FuncSched, 8410 FuncSched, 8418 FuncSched, 10334 FuncSched, 11856 FuncSched, 11858 stat_manager-ag, 11862 FuncSched, 11865 FuncSched, 11866 FuncSched]:
[3]:    @ 0000000000097321 __GI___futex_abstimed_wait_cancelable64
[3]:                       /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/nptl/futex-internal.c:57
[3]:    @ 000000000009a4c6 ___pthread_cond_clockwait
[3]:                       /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/nptl/pthread_cond_wait.c:504
[3]:    @ 000000001255fbc9 folly::FunctionScheduler::run()
[3]:    @ 00000000000df4e4 execute_native_thread_routine
[3]:                       /home/engshare/third-party2/libgcc/11.x/src/gcc-11.x/x86_64-facebook-linux/libstdc++-v3/src/c++11/../../../.././libstdc++-v3/src/c++11/thread.cc:82
[3]:    @ 000000000009ac28 start_thread
[3]:                       /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/nptl/pthread_create.c:434
[3]:    @ 000000000012cfbb __clone3
[3]:                       /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
[3]:
[3]:Stack trace for 26 thread(s) [6945 ThreadedCtrl, 8193 ThreadedCtrl, 8362 ThreadedCtrl, 8363 ThreadedCtrl, 8364 ThreadedCtrl, 8365 ThreadedCtrl, 8366 ThreadedCtrl, 8367 ThreadedCtrl, 8368 ThreadedCtrl, 8369 ThreadedCtrl, 8370 ThreadedCtrl, 8371 ThreadedCtrl, 8372 ThreadedCtrl, 8373 ThreadedCtrl, 8374 ThreadedCtrl, 8375 ThreadedCtrl, 8376 ThreadedCtrl, 8377 ThreadedCtrl, 8380 ThreadedCtrl, 8403 ThreadedCtrl, 8404 ThreadedCtrl, 8405 ThreadedCtrl, 8406 ThreadedCtrl, 8407 ThreadedCtrl, 8408 ThreadedCtrl, 11861 ThreadedCtrl]:
[3]:    @ 00000000001257b9 syscall
[3]:                       /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/misc/../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
[3]:    @ 0000000011c7f0af folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int)
[3]:    @ 0000000011c7ef94 bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&)
[3]:    @ 0000000013a981aa folly::ThreadedExecutor::control()
[3]:    @ 00000000000df4e4 execute_native_thread_routine
[3]:                       /home/engshare/third-party2/libgcc/11.x/src/gcc-11.x/x86_64-facebook-linux/libstdc++-v3/src/c++11/../../../.././libstdc++-v3/src/c++11/thread.cc:82
[3]:    @ 000000000009ac28 start_thread
[3]:                       /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/nptl/pthread_create.c:434
[3]:    @ 000000000012cfbb __clone3
[3]:                       /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
[3]:
[3]:Stack trace for 15 thread(s) [6996 ScribeQS-pr-scu, 8195 ScribeQS-pr-log, 8382 ScribeQS-pr-log, 8384 ScribeQS-pr-log, 8385 ScribeQS-pr-log, 8387 ScribeQS-pr-log, 8388 ScribeQS-pr-log, 8389 ScribeQS-pr-log, 8411 ScribeQS-pr-wor, 8412 ScribeQS-pr-wor, 8413 ScribeQS-pr-wor, 8414 ScribeQS-pr-wor, 8416 ScribeQS-pr-wor, 8417 ScribeQS-pr-wor, 11864 ScribeQS-pr-def]:
[3]:    @ 00000000001257b9 syscall
[3]:                       /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/misc/../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
[3]:    @ 0000000011c7f0af folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int)
[3]:    @ 0000000011c7ee0b bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&)
[3]:    @ 00000000127071df scribe::common::sink::detail::MemoryBatchQueue::tryDequeueUntil(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)
[3]:    @ 00000000129a309a scribe::common::sink::QueueSink::sendLoop(scribe::common::sink::detail::BatchQueue*, int, scribe::common::sink::QueueSink::QueueType)
[3]:    @ 000000001426d598 folly::ThreadedExecutor::work(folly::Function<void ()>&)
[3]:    @ 00000000000df4e4 execute_native_thread_routine
[3]:                       /home/engshare/third-party2/libgcc/11.x/src/gcc-11.x/x86_64-facebook-linux/libstdc++-v3/src/c++11/../../../.././libstdc++-v3/src/c++11/thread.cc:82
[3]:    @ 000000000009ac28 start_thread
[3]:                       /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/nptl/pthread_create.c:434
[3]:    @ 000000000012cfbb __clone3
[3]:                       /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
[3]:
[3]:Stack trace for 14 thread(s) [6855 FBAgentConn, 6911 CobRunnerWorker, 8120 pt_main_thread, 8128 SREventBase1, 8129 SREventBase2, 8130 SREventBase3, 8131 SREventBase4, 8132 SRSelection5, 8133 SRSmcEventBase6, 8134 SRWarmup, 8210 mcrcfgdump, 8222 TierUpdEvb, 8223 CanaryTierEvb, 8670 pt_main_thread]:
[3]:    @ 000000000012c5f2 epoll_wait
[3]:                       /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/misc/../sysdeps/unix/sysv/linux/epoll_wait.c:30
[3]:    @ 000000001ac77c72 (unknown)
[3]:    @ 000000001ac75dff event_base_loop
[3]:    @ 00000000120cfac9 folly::EventBase::loopMain(int, folly::EventBase::LoopOptions)
[3]:    @ 00000000120cf5b9 folly::EventBase::loopBody(int, folly::EventBase::LoopOptions)
[3]:    @ 0000000013c59dc5 folly::EventBase::loopForever()
[3]:    @ 000000001ae6f13b (unknown)
[3]:    @ 00000000000df4e4 execute_native_thread_routine
[3]:                       /home/engshare/third-party2/libgcc/11.x/src/gcc-11.x/x86_64-facebook-linux/libstdc++-v3/src/c++11/../../../.././libstdc++-v3/src/c++11/thread.cc:82
[3]:    @ 000000000009ac28 start_thread
[3]:                       /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/nptl/pthread_create.c:434
[3]:    @ 000000000012cfbb __clone3
[3]:                       /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
[3]:
[3]:Stack trace for 12 thread(s) [7809 clf-direct-chan, 8121 hazptr-tpe-0, 8383 CSIndexFetcher0, 8390 CSIndexFetcher0, 8391 CSIndexFetcher0, 8392 CSIndexFetcher0, 8393 CSIndexFetcher0, 8394 CSIndexFetcher0, 8397 CSIndexFetcher0, 8420 ScribeSR0, 8421 ScribeSR0, 11877 Auxiliary0]:
[3]:    @ 00000000001257b9 syscall
[3]:                       /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/misc/../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
[3]:    @ 0000000011c7f0af folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int)
[3]:    @ 0000000011c7ee0b bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, folly::WaitOptions const&)
[3]:    @ 0000000012aab1b6 folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)
[3]:    @ 000000001349f84c std::__invoke_result<void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&>::type std::__invoke<void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&>(void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&)
[3]:    @ 00000000000df4e4 execute_native_thread_routine
[3]:                       /home/engshare/third-party2/libgcc/11.x/src/gcc-11.x/x86_64-facebook-linux/libstdc++-v3/src/c++11/../../../.././libstdc++-v3/src/c++11/thread.cc:82
[3]:    @ 000000000009ac28 start_thread
[3]:                       /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/nptl/pthread_create.c:434
[3]:    @ 000000000012cfbb __clone3
[3]:                       /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
[3]:
[3]:Stack trace for 8 thread(s) [9393, 9401, 9409, 9417, 9425, 9433, 9440, 9447]:
[3]:    @ 000000000011b93a __libc_read
[3]:                       /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/io/../sysdeps/unix/sysv/linux/read.c:26
[3]:    @ 0000000000012963 ibv_get_async_event
[3]:    @ 0000000038b99423 wrap_ibv_get_async_event(ibv_context*, ibv_async_event*)
[3]:    @ 0000000038b93d5c (unknown)
[3]:    @ 000000000009ac28 start_thread
[3]:                       /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/nptl/pthread_create.c:434
[3]:    @ 000000000012cfbb __clone3
[3]:                       /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
[3]:
[3]:Stack trace for 6 thread(s) [6938 ScribePR0, 6994 server0, 8361 ScribePR0, 8398 server0, 8402 ScribePR0, 8415 server0]:
[3]:    @ 00000000001257b9 syscall
[3]:                       /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/misc/../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
[3]:    @ 0000000011c7f0af folly::detail::futexWaitImpl(std::atomic<unsigned int> const*, unsigned int, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, unsigned int)
[3]:    @ 0000000011c7ee0b bool folly::SaturatingSemaphore<true, std::atomic>::tryWaitSlow<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, 
```

This might be NCCL driver issues, lots of similar issues like #26724

@ArthurZucker  thanks. I tried the 3 techniques in https://github.com/huggingface/transformers/issues/26724 , (CUDA_VISIBLE_DEVICES=0 and NCCL_P2P_DISABLE=1  ddp_backend=gloo) , same behavior. 

also I don't think it's an NCCL issue, because everything works fine when I use a llama2 model as base 

But the modeling code is the same, the only difference is maybe the size (bigger LM head) 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.