Repository: Lightning-AI/pytorch-lightning

Issue:
bug(logger): wandb fails on sweep
## 🐛 Bug

When using `wandb` sweeps for hyperparameters search, I get this error:
> wandb: ERROR Attempted to change value of key "dropout_std" from 0.030424838979365657 to 0.030424838979365654

The reason is I ran:
```python
wandb_logger.log_hyperparams(params)
```

Which I guess has some problem with floating-point numbers in high accuracy?

bug(logger): wandb fails on sweep
## 🐛 Bug

When using `wandb` sweeps for hyperparameters search, I get this error:
> wandb: ERROR Attempted to change value of key "dropout_std" from 0.030424838979365657 to 0.030424838979365654

The reason is I ran:
```python
wandb_logger.log_hyperparams(params)
```

Which I guess has some problem with floating-point numbers in high accuracy?



Discussion:
+1, I faced the same issue when using pytorch lightning with wandb sweeps. To summarize, wandb automatically logs hyperparams when we run the wandb sweep agent on a machine. Later, pytorch lightning again tries to log same hyperparams but due to precision error between lightning and wandb already logged hyperparams, wandb throws this error. Just a guess: wandb sweep agent might be using double format to generate new hyperparams and when lightning receives those args from command line, it converts them to float and tries to log it. I haven't digged in detail where these hyperparams get altered, it could be on wandb side or lightning side.

I reported this issue to wandb and got the following response: 


> It is preferred to either pass your config parameters all at once to: wandb.init(config=config_dict_that_could_have_params_set_by_sweep)  
> or:
> experiment = wandb.init()
> experiment.config.setdefaults(config_dict_that_could_have_params_set_by_sweep) 
> 
> The advantage of doing this is that it will ignore setting any key that has already been set by the sweep.
> 
> We will look into the torch lightning integration and see if we can make this safer for those using sweeps.
cc: @neggert, @Borda, @williamFalcon 
@borisdayma @calclavia pls ^^
The problem is that it tries to log this value twice and is probably called before automatically by pytorch-lightning.
The callback will automatically log every parameter which is in `pl.LightningModule.params` (where you probably already have the dropout).

See an example of using pytorch-lightning with wandb (including sweeps) here: https://github.com/borisdayma/lightning-kitti

I'll be adding it to the pytorch-lightning repo later but still need to push a PR related to the `watch` method for it to work properly.
@borisdayma is it fixed now?
+1, I faced the same issue when using pytorch lightning with wandb sweeps. To summarize, wandb automatically logs hyperparams when we run the wandb sweep agent on a machine. Later, pytorch lightning again tries to log same hyperparams but due to precision error between lightning and wandb already logged hyperparams, wandb throws this error. Just a guess: wandb sweep agent might be using double format to generate new hyperparams and when lightning receives those args from command line, it converts them to float and tries to log it. I haven't digged in detail where these hyperparams get altered, it could be on wandb side or lightning side.

I reported this issue to wandb and got the following response: 


> It is preferred to either pass your config parameters all at once to: wandb.init(config=config_dict_that_could_have_params_set_by_sweep)  
> or:
> experiment = wandb.init()
> experiment.config.setdefaults(config_dict_that_could_have_params_set_by_sweep) 
> 
> The advantage of doing this is that it will ignore setting any key that has already been set by the sweep.
> 
> We will look into the torch lightning integration and see if we can make this safer for those using sweeps.
cc: @neggert, @Borda, @williamFalcon 
@borisdayma @calclavia pls ^^
The problem is that it tries to log this value twice and is probably called before automatically by pytorch-lightning.
The callback will automatically log every parameter which is in `pl.LightningModule.params` (where you probably already have the dropout).

See an example of using pytorch-lightning with wandb (including sweeps) here: https://github.com/borisdayma/lightning-kitti

I'll be adding it to the pytorch-lightning repo later but still need to push a PR related to the `watch` method for it to work properly.
@borisdayma is it fixed now?

Resolving patch:
diff --git a/pytorch_lightning/loggers/wandb.py b/pytorch_lightning/loggers/wandb.py
--- a/pytorch_lightning/loggers/wandb.py
+++ b/pytorch_lightning/loggers/wandb.py
@@ -114,7 +114,7 @@ def watch(self, model: nn.Module, log: str = 'gradients', log_freq: int = 100):
     @rank_zero_only
     def log_hyperparams(self, params: Union[Dict[str, Any], Namespace]) -> None:
         params = self._convert_params(params)
-        self.experiment.config.update(params)
+        self.experiment.config.update(params, allow_val_change=True)
 
     @rank_zero_only
     def log_metrics(self, metrics: Dict[str, float], step: Optional[int] = None) -> None:


Test patch:
diff --git a/tests/loggers/test_wandb.py b/tests/loggers/test_wandb.py
--- a/tests/loggers/test_wandb.py
+++ b/tests/loggers/test_wandb.py
@@ -23,7 +23,7 @@ def test_wandb_logger(wandb):
     wandb.init().log.assert_called_once_with({'global_step': 3, 'acc': 1.0})
 
     logger.log_hyperparams({'test': None})
-    wandb.init().config.update.assert_called_once_with({'test': None})
+    wandb.init().config.update.assert_called_once_with({'test': None}, allow_val_change=True)
 
     logger.watch('model', 'log', 10)
     wandb.init().watch.assert_called_once_with('model', log='log', log_freq=10)