Repository: voicepaw/so-vits-svc-fork

Issue:
Colab: Training steps / epochs not appearing on log, only D_0.pth and G_0.pth being created
**Describe the bug**
Ttarining step does not work. The log shows:
2023-04-13 22:53:18.009147: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-04-13 22:53:19.311998: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
[22:53:20] INFO     [22:53:20] NumExpr defaulting to 2 threads.     

**To Reproduce**
From step "Automatic preprocessing" onwards the log finishes at "NumExpr defaulting to 2 threads." message. 

**Additional context**
Stoped working 2 days ago. 

![Screenshot 2023-04-14 at 01 01 48](https://user-images.githubusercontent.com/116147475/231901199-4ebdcdf7-9be7-45ba-99b0-34ee1537bac5.png)



Discussion:
#317 made this bug
So how exactly do I fix it? It stopped working for me as well today.
+ It just stops running after about a minute, still nothing has been trained or done.
![image](https://user-images.githubusercontent.com/122406301/231947891-487c5472-0fcc-4860-906e-7dbb7891eb0a.png)

> #317 made this bug

How do I revert to like... a previous version or something (in colab)?
Probably it is working but just not logging, probably
That's what I begin to think to but it's still a bummer since Its useful to
know if it's progresses or not.

On Fri, Apr 14, 2023, 8:32 AM 34j ***@***.***> wrote:

> Probably it is working but simply not logging, probably
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/34j/so-vits-svc-fork/issues/321#issuecomment-1507996716>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/A3WEKEZ2WFXN233QHTG7S73XBDVODANCNFSM6AAAAAAW5WIJ64>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>

> Probably it is working but just not logging, probably

Nope. It's not working.
It's crasing after around one minute of running with a green checkmark too. No logs, no checkpoints. Nothing.
> How do I revert to like... a previous version or something (in colab)?

u can install 33180e9 branch to revert to v3.5.0
> > How do I revert to like... a previous version or something (in colab)?
> 
> u can install [33180e9](https://github.com/34j/so-vits-svc-fork/commit/33180e9670b5f5282bee6b9344244c0281d7d21c) branch to revert to v3.5.0

Okay thanks that seems to *kind of* work but it still doesn't save checkpoints when I need to. Usually stopping the cell works just fine, 3.5 gives an eror and starts over every time. The end step seems to be set to 9999. Is there a way I can set it to something like 2500 instead?
> #317 made this bug

The logic is inverted.
`callbacks=[pl.callbacks.RichProgressBar()] if is_notebook() else None,`

It uses the RichProgressBar if it **is** a notebook, not if it **isn't**
> > #317 made this bug
> 
> The logic is inverted. `callbacks=[pl.callbacks.RichProgressBar()] if is_notebook() else None,`
> 
> It uses the RichProgressBar if it **is** a notebook, not if it **isn't**

What do you mean? Can you explain me how to solve it pls?
> What do you mean? Can you explain me how to solve it pls?

Okay so, that code snippet checks if it's running in Colab. If it is, it will use the fancier progress bar.

However, this progress bar does not seem to be available inside Colab so 34j made this method to check.

The logic for it is the wrong way around though - instead of using the old / default progress bar in Colab when it detects it's running in Colab, it is using the fancy one that's not available.

The fix would be to do `if not is_notebook` instead.
___
I noticed this issue locally when the fancy progress bar wasn't available after an update.

If 34j (or someone else) won't be getting to a pull request I can do one once I'm back at my computer in around an hour

Resolving patch:
diff --git a/src/so_vits_svc_fork/train.py b/src/so_vits_svc_fork/train.py
--- a/src/so_vits_svc_fork/train.py
+++ b/src/so_vits_svc_fork/train.py
@@ -88,7 +88,7 @@ def train(
         if hparams.train.get("bf16_run", False)
         else 32,
         strategy=strategy,
-        callbacks=[pl.callbacks.RichProgressBar()] if is_notebook() else None,
+        callbacks=[pl.callbacks.RichProgressBar()] if not is_notebook() else None,
     )
     model = VitsLightning(reset_optimizer=reset_optimizer, **hparams)
     trainer.fit(model, datamodule=datamodule)


Test patch: