Repository: pytorch/ignite

Issue:
Engine restarts training from scratch unexpected
## 🐛 Bug description

Engine `_is_done` function only takes into account `max_epochs` parameter read from the state of the object. This leads to the followling unexpected behavior:

1. Instantiate a simple engine, train for 2 epochs (set max_epochs=2). Save checkpoints.
2. Train for 2 more epochs (set max_epochs=4) using last checkpoint.
3. This leads to the training restarting from scratch instead of continuing from the last checkpoint.

Why?

Engine.run functions has a `max_epochs` parameter that should have the precedence over the `max_epochs` stored in the checkpoint state in the [_is_done](https://github.com/pytorch/ignite/blob/master/ignite/engine/engine.py#L605). 
Currently, at runtime, to check if the training is done, we consider `max_epochs` from the state of the checkpoint, not the `max_epochs` parameter from the user. Therefore if the user specifies a different value (in the case he wants to train for more epochs), his input is not taken into account.

Here is a possible workaround to explain how to solve the problem:
```
    @staticmethod
    def _is_done(state, max_epochs=None):
        return state.iteration == state.epoch_length * (state.max_epochs if max_epochs is None else max_epochs)
```


## Environment

 - PyTorch Version (e.g., 1.4): 1.4.0
 - Ignite Version (e.g., 0.3.0): 0.3.0
 - OS (e.g., Linux): macOS
 - How you installed Ignite (`conda`, `pip`, source): pip
 - Python version: 3.6.10
 - Any other relevant information:




Discussion:
@H4dr1en thanks for the report and a nice catch !

I could reproduce the problem:
```python
import torch
import ignite
print(ignite.__version__)
from ignite.engine import Engine, Events
from ignite.utils import setup_logger


torch.manual_seed(12)


def train_step(trainer, batch):
    s = trainer.state
    print(
        "{}/{} : {} - {:.3f}".format(s.epoch, s.max_epochs, s.iteration, batch)
    )

trainer = Engine(train_step)
trainer.logger = setup_logger()

saved_states = []

# This emulates Checkpoint to save trainer and others
@trainer.on(Events.COMPLETED)
def save_state():
    saved_states.append(trainer.state_dict())


data = [0, 1, 2]

trainer.run(data, max_epochs=2)
print(saved_states)

# This is a workaround
# saved_states[0]['max_epochs'] = 4  
trainer.load_state_dict(saved_states[0])
trainer.run(data, max_epochs=4);
```
output
```
2020-04-28 12:58:11,001 root INFO: Engine run starting with max_epochs=2.
2020-04-28 12:58:11,003 root INFO: Epoch[1] Complete. Time taken: 00:00:00
2020-04-28 12:58:11,006 root INFO: Epoch[2] Complete. Time taken: 00:00:00
2020-04-28 12:58:11,006 root INFO: Engine run complete. Time taken 00:00:00
2020-04-28 12:58:11,008 root INFO: Engine run starting with max_epochs=4.
2020-04-28 12:58:11,011 root INFO: Epoch[1] Complete. Time taken: 00:00:00
2020-04-28 12:58:11,013 root INFO: Epoch[2] Complete. Time taken: 00:00:00
2020-04-28 12:58:11,016 root INFO: Epoch[3] Complete. Time taken: 00:00:00
2020-04-28 12:58:11,018 root INFO: Epoch[4] Complete. Time taken: 00:00:00
2020-04-28 12:58:11,018 root INFO: Engine run complete. Time taken 00:00:00
0.4.0.dev20200427
...
```
Yes, this looks like a bug. As you said, if run's input params override defined state, it should taken into account before checking `_is_done`. 

A workaround to that can be done is commented in the code
```
saved_states[0]['max_epochs'] = 4  
trainer.load_state_dict(saved_states[0])
trainer.run(data, max_epochs=4);
> 
2020-04-28 13:05:02,712 root INFO: Engine run starting with max_epochs=2.
2020-04-28 13:05:02,715 root INFO: Epoch[1] Complete. Time taken: 00:00:00
2020-04-28 13:05:02,717 root INFO: Epoch[2] Complete. Time taken: 00:00:00
2020-04-28 13:05:02,718 root INFO: Engine run complete. Time taken 00:00:00
2020-04-28 13:05:02,720 root INFO: Engine run resuming from iteration 6, epoch 2 until 4 epochs
2020-04-28 13:05:02,722 root INFO: Epoch[3] Complete. Time taken: 00:00:00
2020-04-28 13:05:02,725 root INFO: Epoch[4] Complete. Time taken: 00:00:00
2020-04-28 13:05:02,725 root INFO: Engine run complete. Time taken 00:00:00
0.4.0.dev20200427
1/2 : 1 - 0.000
1/2 : 2 - 1.000
1/2 : 3 - 2.000
2/2 : 4 - 0.000
2/2 : 5 - 1.000
2/2 : 6 - 2.000
[OrderedDict([('seed', 662124363), ('epoch_length', 3), ('max_epochs', 2), ('iteration', 6)])]
3/4 : 7 - 0.000
3/4 : 8 - 1.000
3/4 : 9 - 2.000
4/4 : 10 - 0.000
4/4 : 11 - 1.000
4/4 : 12 - 2.000

```
 



Resolving patch:
diff --git a/ignite/engine/engine.py b/ignite/engine/engine.py
--- a/ignite/engine/engine.py
+++ b/ignite/engine/engine.py
@@ -552,11 +552,12 @@ def run(
             data (Iterable): Collection of batches allowing repeated iteration (e.g., list or `DataLoader`).
             max_epochs (int, optional): Max epochs to run for (default: None).
                 If a new state should be created (first run or run again from ended engine), it's default value is 1.
-                This argument should be `None` if run is resuming from a state.
+                If run is resuming from a state, provided `max_epochs` will be taken into account and should be larger
+                than `engine.state.max_epochs`.
             epoch_length (int, optional): Number of iterations to count as one epoch. By default, it can be set as
                 `len(data)`. If `data` is an iterator and `epoch_length` is not set, then it will be automatically
                 determined as the iteration on which data iterator raises `StopIteration`.
-                This argument should be `None` if run is resuming from a state.
+                This argument should not change if run is resuming from a state.
             seed (int, optional): Deprecated argument. Please, use `torch.manual_seed` or
                 :meth:`~ignite.utils.manual_seed`.
 
@@ -582,20 +583,8 @@ def switch_batch(engine):
                 "Please, use torch.manual_seed or ignite.utils.manual_seed"
             )
 
-        if self.state is None or self._is_done(self.state):
-            # Create new state
-            if max_epochs is None:
-                max_epochs = 1
-            if epoch_length is None:
-                if hasattr(data, "__len__"):
-                    epoch_length = len(data)
-                    if epoch_length < 1:
-                        raise ValueError("Input data has zero size. Please provide non-empty data")
-
-            self.state = State(iteration=0, epoch=0, max_epochs=max_epochs, epoch_length=epoch_length)
-            self.logger.info("Engine run starting with max_epochs={}.".format(max_epochs))
-        else:
-            # Keep actual state and override it if input args provided
+        if self.state is not None:
+            # Check and apply overridden parameters
             if max_epochs is not None:
                 if max_epochs < self.state.epoch:
                     raise ValueError(
@@ -610,6 +599,20 @@ def switch_batch(engine):
                             epoch_length, self.state.epoch_length
                         )
                     )
+
+        if self.state is None or self._is_done(self.state):
+            # Create new state
+            if max_epochs is None:
+                max_epochs = 1
+            if epoch_length is None:
+                if hasattr(data, "__len__"):
+                    epoch_length = len(data)
+                    if epoch_length < 1:
+                        raise ValueError("Input data has zero size. Please provide non-empty data")
+
+            self.state = State(iteration=0, epoch=0, max_epochs=max_epochs, epoch_length=epoch_length)
+            self.logger.info("Engine run starting with max_epochs={}.".format(max_epochs))
+        else:
             self.logger.info(
                 "Engine run resuming from iteration {}, epoch {} until {} epochs".format(
                     self.state.iteration, self.state.epoch, self.state.max_epochs


Test patch:
diff --git a/tests/ignite/engine/test_engine_state_dict.py b/tests/ignite/engine/test_engine_state_dict.py
--- a/tests/ignite/engine/test_engine_state_dict.py
+++ b/tests/ignite/engine/test_engine_state_dict.py
@@ -150,6 +150,26 @@ def test_load_state_dict_with_params_overriding_integration():
         engine.run(data, epoch_length=90)
 
 
+def test_continue_training():
+    # Tests issue : https://github.com/pytorch/ignite/issues/993
+    max_epochs = 2
+    data = range(10)
+    engine = Engine(lambda e, b: 1)
+    state = engine.run(data, max_epochs=max_epochs)
+    assert state.max_epochs == max_epochs
+    assert state.iteration == len(data) * max_epochs
+    assert state.epoch == max_epochs
+
+    @engine.on(Events.STARTED)
+    def assert_continue_training():
+        assert engine.state.epoch == max_epochs
+
+    state = engine.run(data, max_epochs=max_epochs * 2)
+    assert state.max_epochs == max_epochs * 2
+    assert state.iteration == len(data) * max_epochs * 2
+    assert state.epoch == max_epochs * 2
+
+
 def test_state_dict_with_user_keys_integration(dirname):
     engine = Engine(lambda e, b: 1)
     engine.state_dict_user_keys.append("alpha")