RuntimeError: cuFFT error: CUFFT_INTERNAL_ERROR

stevenvana: <p>I am trying to run piper forked from github/rhasspy/piper in google compute engine vm with L4 GPU. I have cuda toolkit 12.6</p>
<p>This is my error log:</p>
<p>python3 -m piper_train --dataset-dir /home/stevenvana/piper/out-train/ --accelerator ‘gpu’ --devices 1 --batch-size 32 --validation-split 0.0 --num-test-examples 0 --max_epochs 10000 --resume_from_checkpoint /home/stevenvana/piper/out-train/epoch=2218-step=838782.ckpt?download=true --checkpoint-epochs 1 --precision 32 --quality high<br>
DEBUG:piper_train:Namespace(dataset_dir=‘/home/stevenvana/piper/out-train/’, checkpoint_epochs=1, quality=‘high’, resume_from_single_speaker_checkpoint=None, logger=True, enable_checkpointing=True, default_root_dir=None, gradient_clip_val=None, gradient_clip_algorithm=None, num_nodes=1, num_processes=None, devices=‘1’, gpus=None, auto_select_gpus=False, tpu_cores=None, ipus=None, enable_progress_bar=True, overfit_batches=0.0, track_grad_norm=-1, check_val_every_n_epoch=1, fast_dev_run=False, accumulate_grad_batches=None, max_epochs=10000, min_epochs=None, max_steps=-1, min_steps=None, max_time=None, limit_train_batches=None, limit_val_batches=None, limit_test_batches=None, limit_predict_batches=None, val_check_interval=None, log_every_n_steps=50, accelerator=‘gpu’, strategy=None, sync_batchnorm=False, precision=32, enable_model_summary=True, weights_save_path=None, num_sanity_val_steps=2, resume_from_checkpoint=‘/home/stevenvana/piper/out-train/epoch=2218-step=838782.ckpt?download=true’, profiler=None, benchmark=None, deterministic=None, reload_dataloaders_every_n_epochs=0, auto_lr_find=False, replace_sampler_ddp=True, detect_anomaly=False, auto_scale_batch_size=False, plugins=None, amp_backend=‘native’, amp_level=None, move_metrics_to_cpu=False, multiple_trainloader_mode=‘max_size_cycle’, batch_size=32, validation_split=0.0, num_test_examples=0, max_phoneme_ids=None, hidden_channels=192, inter_channels=192, filter_channels=768, n_layers=6, n_heads=2, seed=1234)<br>
/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/checkpoint_connector.py:52: LightningDeprecationWarning: Setting <code>Trainer(resume_from_checkpoint=)</code> is deprecated in v1.5 and will be removed in v1.7. Please pass <code>Trainer.fit(ckpt_path=)</code> directly instead.<br>
rank_zero_deprecation(<br>
GPU available: True (cuda), used: True<br>
TPU available: False, using: 0 TPU cores<br>
IPU available: False, using: 0 IPUs<br>
HPU available: False, using: 0 HPUs<br>
DEBUG:piper_train:Checkpoints will be saved every 1 epoch(s)<br>
DEBUG:vits.dataset:Loading dataset: /home/stevenvana/piper/out-train/dataset.jsonl<br>
/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py:731: LightningDeprecationWarning: <code>trainer.resume_from_checkpoint</code> is deprecated in v1.5 and will be removed in v2.0. Specify the fit checkpoint path with <code>trainer.fit(ckpt_path=)</code> instead.<br>
ckpt_path = ckpt_path or self.resume_from_checkpoint<br>
Restoring states from the checkpoint path at /home/stevenvana/piper/out-train/epoch=2218-step=838782.ckpt?download=true<br>
DEBUG:fsspec.local:open file: /home/stevenvana/piper/out-train/epoch=2218-step=838782.ckpt?download=true<br>
/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py:1659: UserWarning: Be aware that when using <code>ckpt_path</code>, callbacks used to create the checkpoint need to be provided during <code>Trainer</code> instantiation. Please add the following callbacks: [“ModelCheckpoint{‘monitor’: None, ‘mode’: ‘min’, ‘every_n_train_steps’: 0, ‘every_n_epochs’: 1, ‘train_time_interval’: None}”].<br>
rank_zero_warn(<br>
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]<br>
DEBUG:fsspec.local:open file: /home/stevenvana/piper/out-train/lightning_logs/version_13/hparams.yaml<br>
Restored all states from the checkpoint file at /home/stevenvana/piper/out-train/epoch=2218-step=838782.ckpt?download=true<br>
/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/utilities/data.py:153: UserWarning: Total length of <code>DataLoader</code> across ranks is zero. Please make sure this was your intention.<br>
rank_zero_warn(<br>
/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:236: PossibleUserWarning: The dataloader, train_dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the <code>num_workers</code> argument<code>(try 8 which is the number of cpus on this machine) in the</code>DataLoader` init to improve performance.<br>
rank_zero_warn(<br>
/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py:1892: PossibleUserWarning: The number of training batches (2) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.<br>
rank_zero_warn(<br>
Traceback (most recent call last):<br>
File “/usr/lib/python3.10/runpy.py”, line 196, in _run_module_as_main<br>
return _run_code(code, main_globals, None,<br>
File “/usr/lib/python3.10/runpy.py”, line 86, in _run_code<br>
exec(code, run_globals)<br>
File “/home/stevenvana/piper/src/python/piper_train/<strong>main</strong>.py”, line 147, in<br>
main()<br>
File “/home/stevenvana/piper/src/python/piper_train/<strong>main</strong>.py”, line 124, in main<br>
trainer.fit(model)<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py”, line 696, in fit<br>
self._call_and_handle_interrupt(<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py”, line 650, in _call_and_handle_interrupt<br>
return trainer_fn(*args, **kwargs)<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py”, line 735, in _fit_impl<br>
results = self._run(model, ckpt_path=self.ckpt_path)<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py”, line 1166, in _run<br>
results = self._run_stage()<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py”, line 1252, in _run_stage<br>
return self._run_train()<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py”, line 1283, in _run_train<br>
self.fit_loop.run()<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py”, line 200, in run<br>
self.advance(*args, **kwargs)<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py”, line 271, in advance<br>
self._outputs = self.epoch_loop.run(self._data_fetcher)<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py”, line 200, in run<br>
self.advance(*args, **kwargs)<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py”, line 203, in advance<br>
batch_output = self.batch_loop.run(kwargs)<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py”, line 200, in run<br>
self.advance(*args, **kwargs)<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py”, line 87, in advance<br>
outputs = self.optimizer_loop.run(optimizers, kwargs)<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py”, line 200, in run<br>
self.advance(*args, **kwargs)<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py”, line 201, in advance<br>
result = self._run_optimization(kwargs, self._optimizers[self.optim_progress.optimizer_position])<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py”, line 248, in _run_optimization<br>
self._optimizer_step(optimizer, opt_idx, kwargs.get(“batch_idx”, 0), closure)<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py”, line 358, in _optimizer_step<br>
self.trainer._call_lightning_module_hook(<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py”, line 1550, in _call_lightning_module_hook<br>
output = fn(*args, **kwargs)<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/core/module.py”, line 1705, in optimizer_step<br>
optimizer.step(closure=optimizer_closure)<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/core/optimizer.py”, line 168, in step<br>
step_output = self._strategy.optimizer_step(self._optimizer, self._optimizer_idx, closure, **kwargs)<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/strategies/strategy.py”, line 216, in optimizer_step<br>
return self.precision_plugin.optimizer_step(model, optimizer, opt_idx, closure, **kwargs)<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/plugins/precision/precision_plugin.py”, line 153, in optimizer_step<br>
return optimizer.step(closure=closure, **kwargs)<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/torch/optim/lr_scheduler.py”, line 68, in wrapper<br>
return wrapped(*args, **kwargs)<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/torch/optim/optimizer.py”, line 140, in wrapper<br>
out = func(*args, **kwargs)<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/torch/autograd/grad_mode.py”, line 27, in decorate_context<br>
return func(*args, **kwargs)<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/torch/optim/adamw.py”, line 120, in step<br>
loss = closure()<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/plugins/precision/precision_plugin.py”, line 138, in _wrap_closure<br>
closure_result = closure()<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py”, line 146, in <strong>call</strong><br>
self._result = self.closure(*args, **kwargs)<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py”, line 132, in closure<br>
step_output = self._step_fn()<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py”, line 407, in _training_step<br>
training_step_output = self.trainer._call_strategy_hook(“training_step”, *kwargs.values())<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py”, line 1704, in _call_strategy_hook<br>
output = fn(*args, **kwargs)<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/pytorch_lightning/strategies/strategy.py”, line 358, in training_step<br>
return self.model.training_step(*args, **kwargs)<br>
File “/home/stevenvana/piper/src/python/piper_train/vits/lightning.py”, line 191, in training_step<br>
return self.training_step_g(batch)<br>
File “/home/stevenvana/piper/src/python/piper_train/vits/lightning.py”, line 230, in training_step_g<br>
y_hat_mel = mel_spectrogram_torch(<br>
File “/home/stevenvana/piper/src/python/piper_train/vits/mel_processing.py”, line 120, in mel_spectrogram_torch<br>
torch.stft(<br>
File “/home/stevenvana/piper/src/python/.venv/lib/python3.10/site-packages/torch/functional.py”, line 632, in stft<br>
return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined]<br>
RuntimeError: cuFFT error: CUFFT_INTERNAL_ERROR</p>
<p>These are my installed dependencies:</p>
<p>Package Version Editable project location</p>
<hr>
<p>absl-py 2.1.0<br>
aiohappyeyeballs 2.4.0<br>
aiohttp 3.10.5<br>
aiosignal 1.3.1<br>
async-timeout 4.0.3<br>
attrs 24.2.0<br>
audioread 3.0.1<br>
build 1.2.1<br>
certifi 2024.7.4<br>
cffi 1.17.0<br>
charset-normalizer 3.3.2<br>
coloredlogs 15.0.1<br>
Cython 0.29.37<br>
decorator 5.1.1<br>
flatbuffers 24.3.25<br>
frozenlist 1.4.1<br>
fsspec 2024.6.1<br>
grpcio 1.66.0<br>
humanfriendly 10.0<br>
idna 3.8<br>
joblib 1.4.2<br>
lazy_loader 0.4<br>
librosa 0.10.2.post1<br>
lightning-utilities 0.11.6<br>
llvmlite 0.43.0<br>
Markdown 3.7<br>
MarkupSafe 2.1.5<br>
mpmath 1.3.0<br>
msgpack 1.0.8<br>
multidict 6.0.5<br>
numba 0.60.0<br>
numpy 1.26.4<br>
nvidia-cublas-cu11 11.10.3.66<br>
nvidia-cuda-nvrtc-cu11 11.7.99<br>
nvidia-cuda-runtime-cu11 11.7.99<br>
nvidia-cudnn-cu11 8.5.0.96<br>
onnxruntime 1.19.0<br>
packaging 24.1<br>
pip 24.0<br>
piper-phonemize 1.1.0<br>
piper_train 1.0.0 /home/stevenvana/piper/src/python<br>
piper-tts 1.2.0<br>
platformdirs 4.2.2<br>
pooch 1.8.2<br>
protobuf 5.27.3<br>
pycparser 2.22<br>
pyDeprecate 0.3.2<br>
pyproject_hooks 1.1.0<br>
pytorch-lightning 1.7.7<br>
PyYAML 6.0.2<br>
requests 2.32.3<br>
scikit-learn 1.5.1<br>
scipy 1.14.1<br>
setuptools 73.0.1<br>
six 1.16.0<br>
soundfile 0.12.1<br>
soxr 0.4.0<br>
sympy 1.13.2<br>
tensorboard 2.17.1<br>
tensorboard-data-server 0.7.2<br>
threadpoolctl 3.5.0<br>
tomli 2.0.1<br>
torch 1.13.1<br>
torchmetrics 0.11.4<br>
tqdm 4.66.5<br>
typing_extensions 4.12.2<br>
urllib3 2.2.2<br>
Werkzeug 3.0.4<br>
wheel 0.44.0</p>
<h3><a name="versions-1" class="anchor" href="#versions-1"></a>Versions</h3>
<p>ubuntu 22.04<br>
python 3.10 venv</p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the issue?</p>