进程结束前出现了如下错误：
/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/diffusers/configuration_utils.py:134: FutureWarning: Accessing config attribute `num_train_timesteps` directly via 'DDPMScheduler' object attribute is deprecated. Please access 'num_train_timesteps' over 'DDPMScheduler's config object instead, e.g. 'scheduler.config.num_train_timesteps'.
  deprecate("direct config name access", "1.0.0", deprecation_message, standard_warn=False)
/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/diffusers/configuration_utils.py:134: FutureWarning: Accessing config attribute `num_train_timesteps` directly via 'DDPMScheduler' object attribute is deprecated. Please access 'num_train_timesteps' over 'DDPMScheduler's config object instead, e.g. 'scheduler.config.num_train_timesteps'.
  deprecate("direct config name access", "1.0.0", deprecation_message, standard_warn=False)
/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/torch/utils/checkpoint.py:1399: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead.
  with device_autocast_ctx, torch.cpu.amp.autocast(**cpu_autocast_kwargs), recompute_context:  # type: ignore[attr-defined]
/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/torch/utils/checkpoint.py:1399: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead.
  with device_autocast_ctx, torch.cpu.amp.autocast(**cpu_autocast_kwargs), recompute_context:  # type: ignore[attr-defined]
Steps:   0%|                                     | 0/30000 [00:16<?, ?it/s, lr=0, step_loss=0.0128][rank0]:[W919 19:13:45.327293009 Module.cpp:175] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1...

/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/transformers/deepspeed.py:24: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
  warnings.warn(
/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/torch/optim/adamw.py:159: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at ../torch/csrc/profiler/unwind/unwind.cpp:209.)
  state["exp_avg_sq"] = torch.zeros_like(
[rank0]: Traceback (most recent call last):
[rank0]:   File "/data/fl/Wonder3D-sketch/train_mvdiffusion_image_edit_test_only.py", line 863, in <module>
[rank0]:     main(cfg)
[rank0]:   File "/data/fl/Wonder3D-sketch/train_mvdiffusion_image_edit_test_only.py", line 779, in main
[rank0]:     optimizer.step()
[rank0]:   File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/accelerate/optimizer.py", line 159, in step
[rank0]:     self.scaler.step(self.optimizer, closure)
[rank0]:   File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/torch/amp/grad_scaler.py", line 454, in step
[rank0]:     retval = self._maybe_opt_step(optimizer, optimizer_state, *args, **kwargs)
[rank0]:   File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/torch/amp/grad_scaler.py", line 352, in _maybe_opt_step
[rank0]:     retval = optimizer.step(*args, **kwargs)
[rank0]:   File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/accelerate/optimizer.py", line 214, in patched_step
[rank0]:     return method(*args, **kwargs)
[rank0]:   File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/torch/optim/lr_scheduler.py", line 130, in wrapper
[rank0]:     return func.__get__(opt, opt.__class__)(*args, **kwargs)
[rank0]:   File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/torch/optim/optimizer.py", line 484, in wrapper
[rank0]:     out = func(*args, **kwargs)
[rank0]:   File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/torch/optim/optimizer.py", line 89, in _use_grad
[rank0]:     ret = func(self, *args, **kwargs)
[rank0]:   File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/torch/optim/adamw.py", line 216, in step
[rank0]:     has_complex = self._init_group(
[rank0]:   File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/torch/optim/adamw.py", line 159, in _init_group
[rank0]:     state["exp_avg_sq"] = torch.zeros_like(
[rank0]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 26.00 MiB. GPU 0 has a total capacity of 31.73 GiB of which 18.38 MiB is free. Process 1140475 has 1.28 GiB memory in use. Process 893160 has 10.20 GiB memory in use. Including non-PyTorch memory, this process has 20.23 GiB memory in use. Of the allocated memory 19.62 GiB is allocated by PyTorch, and 130.02 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[rank0]: Exception raised from malloc at ../c10/cuda/CUDACachingAllocator.cpp:1167 (most recent call first):
[rank0]: C++ CapturedTraceback:
[rank0]: #4 std::_Function_handler<std::shared_ptr<c10::LazyValue<std::string> const> (), c10::SetStackTraceFetcher(std::function<std::string ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) from Logging.cpp:0
[rank0]: #5 c10::Error::Error(c10::SourceLocation, std::string) from ??:0
[rank0]: #6 c10::cuda::CUDACachingAllocator::Native::DeviceCachingAllocator::malloc(signed char, unsigned long, CUstream_st*) from CUDACachingAllocator.cpp:0
[rank0]: #7 c10::cuda::CUDACachingAllocator::Native::NativeCachingAllocator::malloc(void**, signed char, unsigned long, CUstream_st*) from :0
[rank0]: #8 c10::cuda::CUDACachingAllocator::Native::NativeCachingAllocator::allocate(unsigned long) from :0
[rank0]: #9 at::TensorBase at::detail::_empty_strided_generic<c10::ArrayRef<long> >(c10::ArrayRef<long>, c10::ArrayRef<long>, c10::Allocator*, c10::DispatchKeySet, c10::ScalarType) from :0
[rank0]: #10 at::detail::empty_strided_generic(c10::ArrayRef<long>, c10::ArrayRef<long>, c10::Allocator*, c10::DispatchKeySet, c10::ScalarType) from ??:0
[rank0]: #11 at::detail::empty_strided_cuda(c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ScalarType, std::optional<c10::Device>) from ??:0
[rank0]: #12 at::detail::empty_strided_cuda(c10::ArrayRef<long>, c10::ArrayRef<long>, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>) from ??:0
[rank0]: #13 at::native::empty_strided_cuda(c10::ArrayRef<long>, c10::ArrayRef<long>, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>) from ??:0
[rank0]: #14 at::(anonymous namespace)::(anonymous namespace)::wrapper_CUDA__empty_strided(c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>) from RegisterCUDA.cpp:0
[rank0]: #15 c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>), &at::(anonymous namespace)::(anonymous namespace)::wrapper_CUDA__empty_strided>, at::Tensor, c10::guts::typelist::typelist<c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool> > >, at::Tensor (c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>)>::call(c10::OperatorKernel*, c10::DispatchKeySet, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>) from RegisterCUDA.cpp:0
[rank0]: #16 at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>) from ??:0
[rank0]: #17 c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>), &at::(anonymous namespace)::empty_strided>, at::Tensor, c10::guts::typelist::typelist<c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool> > >, at::Tensor (c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>)>::call(c10::OperatorKernel*, c10::DispatchKeySet, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>) from RegisterBackendSelect.cpp:0
[rank0]: #18 at::_ops::empty_strided::call(c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>) from ??:0
[rank0]: #19 at::native::empty_like(at::Tensor const&, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>, std::optional<c10::MemoryFormat>) from ??:0
[rank0]: #20 c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (at::Tensor const&, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>, std::optional<c10::MemoryFormat>), &at::(anonymous namespace)::(anonymous namespace)::wrapper_CompositeExplicitAutograd__empty_like>, at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>, std::optional<c10::MemoryFormat> > >, at::Tensor (at::Tensor const&, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>, std::optional<c10::MemoryFormat>)>::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>, std::optional<c10::MemoryFormat>) from RegisterCompositeExplicitAutograd.cpp:0
[rank0]: #21 at::_ops::empty_like::call(at::Tensor const&, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>, std::optional<c10::MemoryFormat>) from ??:0
[rank0]: #22 at::empty_like(at::Tensor const&, c10::TensorOptions, std::optional<c10::MemoryFormat>) from :0
[rank0]: #23 at::native::zeros_like(at::Tensor const&, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>, std::optional<c10::MemoryFormat>) from ??:0
[rank0]: #24 c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (at::Tensor const&, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>, std::optional<c10::MemoryFormat>), &at::(anonymous namespace)::(anonymous namespace)::wrapper_CompositeExplicitAutograd__zeros_like>, at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>, std::optional<c10::MemoryFormat> > >, at::Tensor (at::Tensor const&, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>, std::optional<c10::MemoryFormat>)>::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>, std::optional<c10::MemoryFormat>) from RegisterCompositeExplicitAutograd.cpp:0
[rank0]: #25 at::_ops::zeros_like::call(at::Tensor const&, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>, std::optional<c10::MemoryFormat>) from ??:0
[rank0]: #26 torch::autograd::THPVariable_zeros_like(_object*, _object*, _object*) from python_torch_functions_0.cpp:0
[rank0]: #27 cfunction_call from /tmp/build/80754af9/python-split_1605449976777/work/Objects/methodobject.c:539
[rank0]: #28 _PyObject_MakeTpCall from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:191
[rank0]: #29 _PyObject_VectorcallTstate from /tmp/build/80754af9/python-split_1605449976777/work/Include/cpython/abstract.h:116
[rank0]: #30 _PyEval_EvalFrame from /tmp/build/80754af9/python-split_1605449976777/work/Include/internal/pycore_ceval.h:40
[rank0]: #31 _PyObject_VectorcallTstate from /tmp/build/80754af9/python-split_1605449976777/work/Include/cpython/abstract.h:118
[rank0]: #32 _PyEval_EvalFrame from /tmp/build/80754af9/python-split_1605449976777/work/Include/internal/pycore_ceval.h:40
[rank0]: #33 _PyFunction_Vectorcall from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:395
[rank0]: #34 PyVectorcall_Call from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:230
[rank0]: #35 do_call_core from /tmp/build/80754af9/python-split_1605449976777/work/Python/ceval.c:5092
[rank0]: #36 _PyEval_EvalFrame from /tmp/build/80754af9/python-split_1605449976777/work/Include/internal/pycore_ceval.h:40
[rank0]: #37 _PyFunction_Vectorcall from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:395
[rank0]: #38 PyVectorcall_Call from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:230
[rank0]: #39 do_call_core from /tmp/build/80754af9/python-split_1605449976777/work/Python/ceval.c:5092
[rank0]: #40 _PyEval_EvalFrame from /tmp/build/80754af9/python-split_1605449976777/work/Include/internal/pycore_ceval.h:40
[rank0]: #41 _PyFunction_Vectorcall from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:395
[rank0]: #42 _PyObject_VectorcallTstate from /tmp/build/80754af9/python-split_1605449976777/work/Include/cpython/abstract.h:118
[rank0]: #43 PyVectorcall_Call from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:230
[rank0]: #44 do_call_core from /tmp/build/80754af9/python-split_1605449976777/work/Python/ceval.c:5092
[rank0]: #45 _PyEval_EvalFrame from /tmp/build/80754af9/python-split_1605449976777/work/Include/internal/pycore_ceval.h:40
[rank0]: #46 _PyFunction_Vectorcall from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:395
[rank0]: #47 PyVectorcall_Call from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:230
[rank0]: #48 do_call_core from /tmp/build/80754af9/python-split_1605449976777/work/Python/ceval.c:5092
[rank0]: #49 _PyEval_EvalFrame from /tmp/build/80754af9/python-split_1605449976777/work/Include/internal/pycore_ceval.h:40
[rank0]: #50 _PyFunction_Vectorcall from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:395
[rank0]: #51 PyVectorcall_Call from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:230
[rank0]: #52 do_call_core from /tmp/build/80754af9/python-split_1605449976777/work/Python/ceval.c:5092
[rank0]: #53 _PyEval_EvalFrame from /tmp/build/80754af9/python-split_1605449976777/work/Include/internal/pycore_ceval.h:40
[rank0]: #54 _PyFunction_Vectorcall from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:395
[rank0]: #55 _PyObject_VectorcallTstate from /tmp/build/80754af9/python-split_1605449976777/work/Include/cpython/abstract.h:118
[rank0]: #56 PyVectorcall_Call from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:230
[rank0]: #57 do_call_core from /tmp/build/80754af9/python-split_1605449976777/work/Python/ceval.c:5092
[rank0]: #58 _PyEval_EvalFrame from /tmp/build/80754af9/python-split_1605449976777/work/Include/internal/pycore_ceval.h:40
[rank0]: #59 _PyFunction_Vectorcall from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:395
[rank0]: #60 _PyObject_VectorcallTstate from /tmp/build/80754af9/python-split_1605449976777/work/Include/cpython/abstract.h:118
[rank0]: #61 _PyEval_EvalFrame from /tmp/build/80754af9/python-split_1605449976777/work/Include/internal/pycore_ceval.h:40
[rank0]: #62 _PyFunction_Vectorcall from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:395
[rank0]: #63 _PyObject_VectorcallTstate from /tmp/build/80754af9/python-split_1605449976777/work/Include/cpython/abstract.h:118
[rank0]: #64 _PyEval_EvalFrame from /tmp/build/80754af9/python-split_1605449976777/work/Include/internal/pycore_ceval.h:40
[rank0]: #65 _PyFunction_Vectorcall from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:395
[rank0]: #66 _PyObject_VectorcallTstate from /tmp/build/80754af9/python-split_1605449976777/work/Include/cpython/abstract.h:118
[rank0]: #67 _PyEval_EvalFrame from /tmp/build/80754af9/python-split_1605449976777/work/Include/internal/pycore_ceval.h:40
[rank0]: #68 _PyEval_EvalCodeWithName from /tmp/build/80754af9/python-split_1605449976777/work/Python/ceval.c:4331
[rank0]: #69 PyEval_EvalCodeEx from /tmp/build/80754af9/python-split_1605449976777/work/Python/ceval.c:4347
[rank0]: #70 PyEval_EvalCode from /tmp/build/80754af9/python-split_1605449976777/work/Python/ceval.c:809
[rank0]: #71 run_eval_code_obj from /tmp/build/80754af9/python-split_1605449976777/work/Python/pythonrun.c:1178
[rank0]: #72 run_mod from /tmp/build/80754af9/python-split_1605449976777/work/Python/pythonrun.c:1199
[rank0]: #73 PyRun_FileExFlags from /tmp/build/80754af9/python-split_1605449976777/work/Python/pythonrun.c:1116
[rank0]: #74 PyRun_SimpleFileExFlags from /tmp/build/80754af9/python-split_1605449976777/work/Python/pythonrun.c:438
[rank0]: #75 pymain_run_file from /tmp/build/80754af9/python-split_1605449976777/work/Modules/main.c:379
[rank0]: #76 Py_BytesMain from /tmp/build/80754af9/python-split_1605449976777/work/Modules/main.c:1129
[rank0]: #77 __libc_start_main from ??:0
[rank0]: #78 _start from /home/rdonnelly/mc/conda-bld/compilers_linux-64_1534865402226/work/.build/src/glibc-2.12.2/csu/../sysdeps/x86_64/elf/start.S:103
[rank0]: #79 <unwind unsupported> from ??:0

Steps:   0%|                                     | 0/30000 [00:20<?, ?it/s, lr=0, step_loss=0.0128]
[rank1]:[E919 19:13:48.759261807 ProcessGroupGloo.cpp:143] Rank 1 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
[rank1]:[W919 19:13:48.760329940 unwind.cpp:209] Warning: Unsupported unwinding pattern: Address not in range (function unwinderFor)
[rank1]:[W919 19:13:48.760365428 Module.cpp:175] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1...

W0919 19:13:48.848006 140368826976064 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 908789 closing signal SIGTERM
E0919 19:13:49.668027 140368826976064 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 1) local_rank: 0 (pid: 908788) of binary: /home/fl/.conda/envs/wonder3d-harsh/bin/python
Traceback (most recent call last):
  File "/home/fl/.conda/envs/wonder3d-harsh/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
    args.func(args)
  File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/accelerate/commands/launch.py", line 1165, in launch_command
    multi_gpu_launcher(args)
  File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/accelerate/commands/launch.py", line 799, in multi_gpu_launcher
    distrib_run.run(args)
  File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/torch/distributed/run.py", line 892, in run
    elastic_launch(
  File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 133, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
train_mvdiffusion_image_edit_test_only.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------