Repository: voicepaw/so-vits-svc-fork

Issue:
How to resume training from checkpoint?
Right now if the training gets interrupted, I have to start over from scratch. Is there a way to continue training from the latest checkpoint?


Discussion:
This is not the expected behavior. Are you placing different models such as 4.0 v2? Please post the output as there is not enough information.
For example, lets say I trained this:

![01](https://user-images.githubusercontent.com/125531852/226784159-fb53f8a2-2016-40ab-90af-10e380bc1e8c.png)

Then I stopped training at G5600.pth

How do I continue training from G5600.pth? Do I need to edit the config file, or add extra arguments after `svc train`?
Can you please post the output of svc train? This is a bug.
I think there is a misunderstanding. I'm not making a bug report, I'm asking if continuing training is possible?

Here is the stack trace:

```
(venv) C:\Users\LXC PC\Desktop\sovits\venv\Scripts>svc train
  0%|                                                                                         | 0/9934 [00:00<?, ?it/s]C:\Users\LXC PC\Desktop\sovits\venv\lib\site-packages\torch\functional.py:641: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\SpectralOps.cpp:867.)
  return _VF.stft(input, n_fft, hop_length, win_length, window,  # type: ignore[attr-defined]
C:\Users\LXC PC\Desktop\sovits\venv\lib\site-packages\torch\autograd\__init__.py:200: UserWarning: Grad strides do not match bucket view strides. This may indicate grad was not created according to the gradient layout contract, or that the param's strides changed since DDP was constructed.  This is not an error, but may impair performance.
grad.sizes() = [32, 1, 4], strides() = [4, 1, 1]
bucket_view.sizes() = [32, 1, 4], strides() = [4, 4, 1] (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\distributed\c10d\reducer.cpp:337.)
  Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  1%|▌                                                                            | 69/9934 [08:20<21:52:24,  7.98s/it]
  ```
  
When training again, it overwrites the old checkpoints:
  
![02](https://user-images.githubusercontent.com/125531852/226796406-d670186d-d9cc-4b63-b588-4ea6daf46f40.png)

Is it possible to continue training from the latest checkpoint `G_5600.pth`, instead of starting over from scratch?

  

Resolving patch:
diff --git a/src/so_vits_svc_fork/utils.py b/src/so_vits_svc_fork/utils.py
--- a/src/so_vits_svc_fork/utils.py
+++ b/src/so_vits_svc_fork/utils.py
@@ -476,7 +476,9 @@ def summarize(
 
 
 def latest_checkpoint_path(dir_path: Path | str, regex: str = "G_*.pth"):
-    return list(sorted(Path(dir_path).glob(regex)))[-1]
+    dir_path = Path(dir_path)
+    name_key = lambda p: int(re.match(r"._(\d+)\.pth", p.name).group(1))
+    return list(sorted(dir_path.glob(regex), key=name_key))[-1]
 
 
 def plot_spectrogram_to_numpy(spectrogram):


Test patch: