Repository: NVIDIA/NeMo

Issue:
from_pretrained() does not update configuration in exp_manager
**Describe the bug**

When fine tuning from a NeMo model (ex. stt_en_cn1024), the exp_manager's cfg is not updated properly.
I can see that in my run the model uses one config, but WandB reports another.

This issue did not occur in v1.4.0 and happened after I upgraded to v1.5.0.
Maybe it has to do with order of operations? See below.

**Steps/Code to reproduce bug**

```py
import pytorch_lightning as pl
from nemo.collections.asr.models import EncDecCTCModelBPE
from nemo.core.config import hydra_runner
from nemo.utils.exp_manager import exp_manager

@hydra_runner(config_path="conf/citrinet/", config_name="config")
def main(cfg):
    trainer = pl.Trainer(**cfg.trainer)
    log_dir = exp_manager(trainer, cfg.get("exp_manager", None))
    asr_model = EncDecCTCModelBPE.from_pretrained(model_name=cfg.init_from_pretrained_model)
    asr_model.encoder.unfreeze()
    asr_model.change_vocabulary(
      new_tokenizer_dir=cfg.model.tokenizer.dir,
      new_tokenizer_type=cfg.model.tokenizer.type
    )
    asr_model.setup_optimization(cfg.model.optim)
    asr_model.setup_training_data(cfg.model.train_ds)
    asr_model.setup_multiple_validation_data(cfg.model.validation_ds)
    asr_model.spec_augmentation = asr_model.from_config_dict(cfg.model.spec_augment)
    asr_model.set_trainer(trainer)
    trainer.fit(asr_model)
```

**Expected behavior**

WandB cfg should display the proper config ([Pastebin of the WandB config](https://pastebin.com/Bc29P9FL))

**Environment overview (please complete the following information)**

 - Environment location: Docker (nvcr.io/nvidia/pytorch:21.10-py3) on AWS EC2 using `docker run -it bash <image>`
 - Method of NeMo install: `pip install nemo_toolkit[asr]==1.5.1`

**Additional context**

GPU model: V100
Nvidia driver: 460



Discussion:
In general, the set_trainer() should be the first call after creating the model - or pass the trainer in the restore_from() or from_pretrained() method itself. 

However this should not affect exp manager - the only thing it depends on is the config passed at this line `log_dir = exp_manager(trainer, cfg.get("exp_manager", None))`. Could you print out this config (cfg.exp_manager) and check whether it matches what you expect ?
My `exp_manager` cfg:

```yaml
exp_manager:
  exp_dir: "experiments"
  name: *name
  create_tensorboard_logger: true
  create_checkpoint_callback: true
  checkpoint_callback_params:
    monitor: "val_wer"
    mode: "min"
    save_top_k: 2
    always_save_nemo: true
  create_wandb_logger: true
  wandb_logger_kwargs:
    job_type: train
    name: *name
    entity: <WANDB_ENTITY>
    project: nemo-experiments
    tags: ["CitriNet-Lg", "train"]
```
Oh I think I got the issue, the config passed to the main method is a new config, but the model is restored by the old config embedded in the Nemo file - you should be able to do model.cfg = cfg before your train step to correctly update the internal config of the model. That's a good catch, I believe only one of the notebooks shows this. Might as well add to the documentation. 

Still, trainer should be set before calling any other function, or passed into the restore_from or from_pretrained call itself 
Sounds good, I will try this out and report back if the proposed solution works.
Will modifying the model's internal `cfg` affect training at all or is that what all the `setup/change_*` methods are for? I'm assuming the latter ofc, but making sure.
The change* methods are to change the subparts of the model itself after it has been built - the data loaders, the optimizer/schedule and any extra things - but they don't update the config. The config details how the model *was* built, not how it currently *is* built. If you update the config then the model is rebuilt the next time with the updated info. 
Unfortunately, setting `model.cfg = cfg` before `trainer.fit()` did not change the model cfg in W&B :/
Colab seems to work fine, the wandb report does show different hyper parameters. Locally I could get it working as well with PTL 1.5.5 and main branch 
I will try the main branch then and report back.
Using `main` branch still does not work.

```py
asr_model = EncDecRNNTBPEModel.from_pretrained(model_name=cfg.init_from_pretrained_model)
asr_model.cfg = cfg.model
asr_model.set_trainer(trainer)
...
trainer.fit(asr_model)
```
It also doesn't change things like:
* `loss` and kwargs
* `log_prediction`
Weird. We'll take a look at it. 
Ok, it looks like the model cfg changed but not the W&B config.
You were correct, apologies for the confusion.

Taking a look as to why the W&B cfg is not changing.
So before `trainer.fit(asr_model)` is called, `wandb.config` is empty.
After training is done, `wandb.config` is populated, but with the cfg from the pretrained model.
Looks like the config is updated [here](https://github.com/PyTorchLightning/pytorch-lightning/blob/16a04b29eb811f105a11db4f7f3003489252f9f8/pytorch_lightning/trainer/trainer.py#L1164) which calls the `WandbLogger`'s [`log_hyperparams`](https://github.com/PyTorchLightning/pytorch-lightning/blob/1c28f361d47257feb0bd34cbc8d41642a1f0a2ab/pytorch_lightning/loggers/wandb.py#L358)

Edit: This seems to be a PTL issue since it uses `hparams_initial` by default to log hyperparameters ([here](https://github.com/PyTorchLightning/pytorch-lightning/blob/16a04b29eb811f105a11db4f7f3003489252f9f8/pytorch_lightning/trainer/trainer.py#L1228))

Edit: I think I will close this and open up an issue in PTL (unless there's something we can do from NeMo side @titu1994).
It appears to be that PTL assumes `hparams_initial` in a PTL module is the model's hyperparameters by default, which is not a valid assumption (as is the case here).

My current work around to getting the right config in W&B is the following:

`asr_model._hparams_initial["cfg"] = OmegaConf.to_object(cfg.model)`

as well as

`asr_model.wer.log_prediction = cfg.model.get("log_prediction", False)`

to update the `log_prediction` value. For this, I suggest introducing some config or `setup_metric()` method for the model to update the metric if needed.

Resolving patch:
diff --git a/nemo/core/classes/modelPT.py b/nemo/core/classes/modelPT.py
--- a/nemo/core/classes/modelPT.py
+++ b/nemo/core/classes/modelPT.py
@@ -1153,6 +1153,10 @@ def cfg(self, cfg):
         self._cfg = cfg
         self._set_hparams(OmegaConf.create({'cfg': self._cfg}))
 
+        # TODO: Remove in NeMo 1.7 (or when PTL fixes this on their end)
+        if hasattr(self, '_hparams_initial') and 'cfg' in self._hparams_initial:
+            self._hparams_initial['cfg'] = OmegaConf.to_object(self._cfg)
+
     @staticmethod
     def _is_model_being_restored() -> bool:
         app_state = AppState()


Test patch: