INFO     move text encoders to gpu ]8;id=660757;file:///kaggle/temp/kohya_ss/sd-scripts/./flux_train_network.py\flux_train_network.py]8;;\:]8;id=351556;file:///kaggle/temp/kohya_ss/sd-scripts/./flux_train_network.py#216\216]8;;\
[rank1]: Traceback (most recent call last):
[rank1]:   File "/kaggle/temp/kohya_ss/sd-scripts/./flux_train_network.py", line 519, in <module>
[rank1]:     trainer.train(args)
[rank1]:   File "/kaggle/temp/kohya_ss/sd-scripts/train_network.py", line 402, in train
[rank1]:     self.cache_text_encoder_outputs_if_needed(args, accelerator, unet, vae, text_encoders, train_dataset_group, weight_dtype)
[rank1]:   File "/kaggle/temp/kohya_ss/sd-scripts/./flux_train_network.py", line 218, in cache_text_encoder_outputs_if_needed
[rank1]:     text_encoders[1].to(accelerator.device)
[rank1]:   File "/opt/conda/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2883, in to
[rank1]:     return super().to(*args, **kwargs)
[rank1]:   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1174, in to
[rank1]:     return self._apply(convert)
[rank1]:   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 780, in _apply
[rank1]:     module._apply(fn)
[rank1]:   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 805, in _apply
[rank1]:     param_applied = fn(param)
[rank1]:   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1167, in convert
[rank1]:     raise NotImplementedError(
[rank1]: NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device.
[rank0]: Traceback (most recent call last):
[rank0]:   File "/kaggle/temp/kohya_ss/sd-scripts/./flux_train_network.py", line 519, in <module>
[rank0]:     trainer.train(args)
[rank0]:   File "/kaggle/temp/kohya_ss/sd-scripts/train_network.py", line 402, in train
[rank0]:     self.cache_text_encoder_outputs_if_needed(args, accelerator, unet, vae, text_encoders, train_dataset_group, weight_dtype)
[rank0]:   File "/kaggle/temp/kohya_ss/sd-scripts/./flux_train_network.py", line 218, in cache_text_encoder_outputs_if_needed
[rank0]:     text_encoders[1].to(accelerator.device)
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2883, in to
[rank0]:     return super().to(*args, **kwargs)
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1174, in to
[rank0]:     return self._apply(convert)
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 780, in _apply
[rank0]:     module._apply(fn)
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 805, in _apply
[rank0]:     param_applied = fn(param)
[rank0]:   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1167, in convert
[rank0]:     raise NotImplementedError(
[rank0]: NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device.
[rank0]:[W921 21:39:53.674576622 ProcessGroupNCCL.cpp:1168] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())
W0921 21:39:55.660000 135639820904256 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 193 closing signal SIGTERM
E0921 21:39:55.724000 135639820904256 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 1) local_rank: 1 (pid: 194) of binary: /opt/conda/bin/python3.10
Traceback (most recent call last):
  File "/opt/conda/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
    args.func(args)
  File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1097, in launch_command
    multi_gpu_launcher(args)
  File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 734, in multi_gpu_launcher
    distrib_run.run(args)
  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 892, in run
    elastic_launch(
  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 133, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
./flux_train_network.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-09-21_21:39:55
  host      : 458ef6321e1e
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 194)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

What's happen? think in english then respond in 中文