Repository: coqui-ai/TTS

Issue:
ParallelWaveGAN config should be adjusted
Hi,

I have tried training with the current default config and it drops the learning rate too fast and the model converges to generate noise. Because when `scheduler_after_epoch=False`, `ExponentialLR` with `gamma=0.999` will cause the learning rate to reach 0.000 in 10k steps.

Config from the original paper are:
 ```
    batch_size=8,
    stft_loss_weight=1.0,
    mse_G_loss_weight=4.0,
    steps_to_start_discriminator=100000,
    lr_gen=0.0001,
    lr_disc=0.00005,
    lr_scheduler_gen="StepLR",
    lr_scheduler_gen_params={"gamma": 0.5, "step_size": 200000, "last_epoch": -1},
    lr_scheduler_disc="StepLR",
    lr_scheduler_disc_params={"gamma": 0.5, "step_size": 200000, "last_epoch": -1},
    scheduler_after_epoch=False,
```

It is also possible to use ExponentialLR with some float rounding error:
```
    lr_scheduler_gen="ExponentialLR",  # one of the schedulers from https:#pytorch.org/docs/stable/optim.html
    lr_scheduler_gen_params={"gamma": 0.5**(1/200000), "last_epoch": -1},
    lr_scheduler_disc="ExponentialLR",  # one of the schedulers from https:#pytorch.org/docs/stable/optim.html
    lr_scheduler_disc_params={"gamma": 0.5**(1/200000), "last_epoch": -1},
```

With more GPU memory the batch_size can be increased and steps reduced.


Discussion:
Good catch. Would you send a PR including these changes? 
@erogol Unfortunately i cloned the repo in Oct and made a bunch of changes along the way... here's the new config file:
[parallel_wavegan_config.txt](https://github.com/coqui-ai/TTS/files/8015055/parallel_wavegan_config.txt)
 

Resolving patch:
diff --git a/TTS/vocoder/configs/parallel_wavegan_config.py b/TTS/vocoder/configs/parallel_wavegan_config.py
--- a/TTS/vocoder/configs/parallel_wavegan_config.py
+++ b/TTS/vocoder/configs/parallel_wavegan_config.py
@@ -70,11 +70,11 @@ class ParallelWaveganConfig(BaseGANVocoderConfig):
         lr_scheduler_gen (torch.optim.Scheduler):
             Learning rate scheduler for the generator. Defaults to `ExponentialLR`.
         lr_scheduler_gen_params (dict):
-            Parameters for the generator learning rate scheduler. Defaults to `{"gamma": 0.999, "last_epoch": -1}`.
+            Parameters for the generator learning rate scheduler. Defaults to `{"gamma": 0.5, "step_size": 200000, "last_epoch": -1}`.
         lr_scheduler_disc (torch.optim.Scheduler):
             Learning rate scheduler for the discriminator. Defaults to `ExponentialLR`.
         lr_scheduler_dict_params (dict):
-            Parameters for the discriminator learning rate scheduler. Defaults to `{"gamma": 0.999, "last_epoch": -1}`.
+            Parameters for the discriminator learning rate scheduler. Defaults to `{"gamma": 0.5, "step_size": 200000, "last_epoch": -1}`.
     """
 
     model: str = "parallel_wavegan"
@@ -124,7 +124,8 @@ class ParallelWaveganConfig(BaseGANVocoderConfig):
     lr_disc: float = 0.0002  # Initial learning rate.
     optimizer: str = "AdamW"
     optimizer_params: dict = field(default_factory=lambda: {"betas": [0.8, 0.99], "weight_decay": 0.0})
-    lr_scheduler_gen: str = "ExponentialLR"  # one of the schedulers from https:#pytorch.org/docs/stable/optim.html
-    lr_scheduler_gen_params: dict = field(default_factory=lambda: {"gamma": 0.999, "last_epoch": -1})
-    lr_scheduler_disc: str = "ExponentialLR"  # one of the schedulers from https:#pytorch.org/docs/stable/optim.html
-    lr_scheduler_disc_params: dict = field(default_factory=lambda: {"gamma": 0.999, "last_epoch": -1})
+    lr_scheduler_gen: str = "StepLR"  # one of the schedulers from https:#pytorch.org/docs/stable/optim.html
+    lr_scheduler_gen_params: dict = field(default_factory=lambda: {"gamma": 0.5, "step_size": 200000, "last_epoch": -1})
+    lr_scheduler_disc: str = "StepLR"  # one of the schedulers from https:#pytorch.org/docs/stable/optim.html
+    lr_scheduler_disc_params: dict = field(default_factory=lambda: {"gamma": 0.5, "step_size": 200000, "last_epoch": -1})
+    scheduler_after_epoch: bool = False


Test patch: