Repository: pytorch/ignite

Issue:
Possibly a bug when try to simulate lr scheduling values
The following code fails:
```python
lr_scheduler = create_lr_scheduler_with_warmup(
    StepLR(optimizer, step_size=2.4, gamma=0.97),
    warmup_start_value=0.0,
    warmup_end_value=scaled_lr,
    warmup_duration=5,
    output_simulated_values=[None] * 50
)
```
with the error
```
ignite/ignite/contrib/handlers/param_scheduler.py in _replicate_lr_scheduler(lr_scheduler, new_optimizer_param_groups)
    489         t = torch.zeros([1], requires_grad=True)
    490         dummy_optimizer = optimizer_cls([t], lr=0.1)
--> 491         dummy_optimizer.load_state_dict(lr_scheduler.optimizer.state_dict())
    492         if new_optimizer_param_groups is not None:
    493             dummy_optimizer.param_groups = new_optimizer_param_groups

/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py in load_state_dict(self, state_dict)
    113         saved_lens = (len(g['params']) for g in saved_groups)
    114         if any(p_len != s_len for p_len, s_len in zip(param_lens, saved_lens)):
--> 115             raise ValueError("loaded state dict contains a parameter group "
    116                              "that doesn't match the size of optimizer's group")
    117 

ValueError: loaded state dict contains a parameter group that doesn't match the size of optimizer's group
```

In this case optimizer has a real model inside.


Discussion:
Another observation: we need to multiply by 1.25 `warmup_end_value` in order to end the warmup with `warmup_end_value` value. This should be fixed.

@vfdev-5 could you provide some more code to reproduce this issue?
@anmolsjoshi following snippet should reproduce the error:
```python
import torch
from torch.optim.lr_scheduler import StepLR

from ignite.contrib.handlers import create_lr_scheduler_with_warmup

from torchvision.models.resnet import resnet18

model = resnet18()

optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

output_simulated_values = [None] * 50

lr_scheduler = create_lr_scheduler_with_warmup(
    StepLR(optimizer, step_size=2.4, gamma=0.97),
    warmup_start_value=0.0,
    warmup_end_value=0.01,
    warmup_duration=5,
    output_simulated_values=output_simulated_values
)
```
@vfdev-5 updating [this line of code](https://github.com/pytorch/ignite/blob/master/ignite/contrib/handlers/param_scheduler.py#L490) to the code below seems to resolve the issue and the tests are still passing.

Do you know if this correctly resolves the issue? I don't want things to silently fail. 

```python
dummy_optimizer = optimizer_cls(lr_scheduler.optimizer.param_groups[0]["params"], lr=0.1)
```

Could you clarify [this comment](https://github.com/pytorch/ignite/issues/541#issuecomment-498022062) further?
@anmolsjoshi, the only problem I feel with this fix
```python
dummy_optimizer = optimizer_cls(lr_scheduler.optimizer.param_groups[0]["params"], lr=0.1)
```
is that we pass the real model's parameters to the replicated optimizer and lr scheduler. Maybe the thing wont work if there more than 1 `param_groups` ?

I was thinking about whether the following line is really necessary : 
https://github.com/pytorch/ignite/blob/c5dda81be18cc938ba8f1e96dde4eb7b05730ca6/ignite/contrib/handlers/param_scheduler.py#L491
maybe can omit it ?


Concerning https://github.com/pytorch/ignite/issues/541#issuecomment-498022062, I think I fixed this [here](https://github.com/pytorch/ignite/pull/556)


@vfdev-5 commenting that line out caused other tests to fail. [This error](https://github.com/pytorch/pytorch/blob/master/torch/optim/lr_scheduler.py#L24-L25) is being raised. 

Could you explain [this test](https://github.com/pytorch/ignite/blob/master/tests/ignite/contrib/handlers/test_param_scheduler.py#L338-L339)?
@anmolsjoshi this should check this condition:
https://github.com/pytorch/ignite/blob/c5dda81be18cc938ba8f1e96dde4eb7b05730ca6/ignite/contrib/handlers/param_scheduler.py#L435

I agree that it would be better to write it as : 
```python
with pytest.raises(ValueError, match="Optimizer passed to lr_scheduler should have a single param group"):
        LRScheduler.simulate_values(num_events=100, lr_scheduler=lr_scheduler)
```


> commenting that line out caused other tests to fail. This error is being raised.

we can possibly set manually `initial_lr` into dummy optimizer ?
@anmolsjoshi any updates on this ?

Resolving patch:
diff --git a/ignite/contrib/handlers/param_scheduler.py b/ignite/contrib/handlers/param_scheduler.py
--- a/ignite/contrib/handlers/param_scheduler.py
+++ b/ignite/contrib/handlers/param_scheduler.py
@@ -487,10 +487,11 @@ def _replicate_lr_scheduler(lr_scheduler, new_optimizer_param_groups=None):
         lr_scheduler_cls = lr_scheduler.__class__
         optimizer_cls = lr_scheduler.optimizer.__class__
         t = torch.zeros([1], requires_grad=True)
-        dummy_optimizer = optimizer_cls([t], lr=0.1)
-        dummy_optimizer.load_state_dict(lr_scheduler.optimizer.state_dict())
+        dummy_optimizer = optimizer_cls([t], **lr_scheduler.optimizer.defaults)
         if new_optimizer_param_groups is not None:
             dummy_optimizer.param_groups = new_optimizer_param_groups
+        for group in dummy_optimizer.param_groups:
+            group.setdefault('initial_lr', group['lr'])
         kwargs = lr_scheduler.state_dict()
         for k in ['base_lrs', '_step_count']:
             del kwargs[k]


Test patch:
diff --git a/tests/ignite/contrib/handlers/test_param_scheduler.py b/tests/ignite/contrib/handlers/test_param_scheduler.py
--- a/tests/ignite/contrib/handlers/test_param_scheduler.py
+++ b/tests/ignite/contrib/handlers/test_param_scheduler.py
@@ -710,3 +710,24 @@ def save_lr(engine):
     lr_scheduler2 = LinearCyclicalScheduler(optimizer.param_groups[1], "lr",
                                             start_value=1.0, end_value=0.0, cycle_size=10)
     _test([lr_scheduler1, lr_scheduler2], optimizer)
+
+
+def test_create_lr_scheduler_with_warmup_with_real_model(dummy_model_factory):
+
+    model = dummy_model_factory(with_grads=False, with_frozen_layer=False)
+    optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
+    scaled_lr = 0.02
+
+    output_simulated_values = [None] * 50
+
+    create_lr_scheduler_with_warmup(
+        torch.optim.lr_scheduler.StepLR(optimizer, step_size=2, gamma=0.97),
+        warmup_start_value=0.0,
+        warmup_end_value=scaled_lr,
+        warmup_duration=5,
+        output_simulated_values=output_simulated_values
+    )
+
+    assert output_simulated_values[0] == [0, 0.0]
+    assert output_simulated_values[5] == [5, scaled_lr]
+    assert output_simulated_values[6] == [5 + 1, 0.01]