GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
/home/developer6/.local/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/logger_connector/logger_connector.py:75: Starting from v1.9.0, `tensorboardX` has been removed as a dependency of the `pytorch_lightning` package, due to potential conflicts with other packages in the ML ecosystem. For this reason, `logger=True` will use `CSVLogger` as the default logger, unless the `tensorboard` or `tensorboardX` packages are found. Please `pip install lightning[extra]` or one of them to enable TensorBoard support by default
You are using a CUDA device ('NVIDIA H100 80GB HBM3') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/1
----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All distributed processes registered. Starting with 1 processes
----------------------------------------------------------------------------------------------------

Using the latest cached version of the dataset since kotoba-speech/peoples_speech_long_tokenized_bpe couldn't be found on the Hugging Face Hub
Traceback (most recent call last):
  File "/home/developer6/kotoba-voice/fam/llm/train.py", line 558, in <module>
    trainer.fit(
  File "/home/developer6/.local/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 548, in fit
    call._call_and_handle_interrupt(
  File "/home/developer6/.local/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 43, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "/home/developer6/.local/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 105, in launch
    return function(*args, **kwargs)
  File "/home/developer6/.local/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 584, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/home/developer6/.local/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 953, in _run
    call._call_setup_hook(self)  # allow user to set up LightningModule in accelerator environment
  File "/home/developer6/.local/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 92, in _call_setup_hook
    _call_lightning_datamodule_hook(trainer, "setup", stage=fn)
  File "/home/developer6/.local/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 179, in _call_lightning_datamodule_hook
    return fn(*args, **kwargs)
  File "/home/developer6/kotoba-voice/fam/llm/training/datamodule.py", line 35, in setup
    self.voice_train = VoiceDataset(split="train", data_list=self.data_list,local_path=self.local_path, ratio_filter=self.ratio_filter, per_video_spk=self.per_video_spk)
  File "/home/developer6/kotoba-voice/fam/llm/training/dataset.py", line 42, in __init__
    self.data = self._load_data(split, ratio_filter)
  File "/home/developer6/kotoba-voice/fam/llm/training/dataset.py", line 50, in _load_data
    dataset_object = load_dataset(f"kotoba-speech/{dataset}_tokenized_bpe", subset, cache_dir=self.local_path, num_proc=os.cpu_count(), columns=["text_tokens", "dataset_id", "spk_embed", "audio_tokens_bpe"])
  File "/home/developer6/.local/lib/python3.10/site-packages/datasets/load.py", line 2592, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/developer6/.local/lib/python3.10/site-packages/datasets/load.py", line 2301, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
  File "/home/developer6/.local/lib/python3.10/site-packages/datasets/packaged_modules/cache/cache.py", line 140, in __init__
    config_name, version, hash = _find_hash_in_cache(
  File "/home/developer6/.local/lib/python3.10/site-packages/datasets/packaged_modules/cache/cache.py", line 65, in _find_hash_in_cache
    raise ValueError(
ValueError: Couldn't find cache for kotoba-speech/peoples_speech_long_tokenized_bpe for config 'default-67671f1ceb3e51bf'
Available configs in the cache: ['default-677cd9bae01bc50b']