Repository: chainer/chainer

Issue:
Adam-based training with resumed trainer changes even if randomness is removed
A trainer with Adam has been run until the end and it did snapshot at a middle checkpoint.
After that, if I run a trainer which is resumed with the snapshot, its (future) result is different from the end. Even if random seed is fixed and other explicit randomness is removed, it happens.
Furthermore, I found trainers with SGD do not cause the behavior.

Reproduction code is here.
https://gist.github.com/soskek/3df39e70b116fba848eaa402cb9bbf5a

You can see the behavior as follows:
```
# Result changes if using Adam
python train_mnist.py --adam
python train_mnist.py --adam --resume result/snapshot_iter_4 --out result2
# Not if using SGD
python train_mnist.py
python train_mnist.py --resume result/snapshot_iter_4 --out result2
```


Discussion:
I suppose the bug occurred because `t` is not treated as `Hyperparameter`, while Adam rule depends on it.

I checked with
```
    trainer.run()
    print(optimizer.t)
    print(optimizer.alpha)
    print(model.predictor.l1.W.update_rule.t)
    print(model.predictor.l1.W.update_rule.hyperparam.alpha)
```
and found `t` is not loaded as an attr of `update_rule` (of each **layer**), while `optimizer.t` is loaded.

```
% python train_mnist.py --adam
epoch       main/loss   validation/main/loss  main/accuracy  validation/main/accuracy  elapsed_time
1           2.37069                           0                                        0.00347109
2           2.25364                           0                                        0.0645422
3           2.14884                           0                                        0.123161
4           2.05857                           0                                        0.198989
5           1.97709                           1                                        0.255294
6           1.90448                           1                                        0.318864
7           1.82524                           1                                        0.374704
8           1.74365                           1                                        0.434473
8
0.001
8
0.001
```
```
% python train_mnist.py --adam --resume result/snapshot_iter_4 --out result2
epoch       main/loss   validation/main/loss  main/accuracy  validation/main/accuracy  elapsed_time
1           2.37069                           0                                        0.00337989
2           2.25364                           0                                        0.0641428
3           2.14884                           0                                        0.133283
4           2.05857                           0                                        0.199131
5           1.97709                           1                                        0.204199
6           1.84183                           1                                        0.261725
7           1.72684                           1                                        0.326133
8           1.61708                           1                                        0.387963
8
0.001
4
0.001
```
Thank you for the great debug!!! Your idea looks correct.
Could I begin to make PR to fix it? Otherwise, if you are willing to fix it, it is also welcome.
I'd discuss how to fix this. There are at least two way:
- by serializing `t` at `chainer.optimizer.UpdateRule`; or
- by storing `t` of `UpdateRule` when `t` of `Optimizer` is loaded.

(I don't really think `t` is a hyperparameter, while my previous comment might be read as suggesting that.)
The first one looks more general and safe, because it does not refer to implementation of `Optimizer`. But, I am not very familiar with implementation of serializers. Which do you prefer?
> The first one looks more general and safe, because it does not refer to implementation of Optimizer.

I agree.  I'll write a PR.

FYI:
- `Trainer.serialize` calls `self.updater.serialize`
- `StandardUpdater.serialize` calls all `optimizer.serialize`
- `Optimizer.serialize` calls existing `rule.serialize`
  - `Optimizer.serialize` also serializes `self.t`
- `UpdateRule.serialize` calls `serializer` for each `self._state[key]`
  - `UpdateRule.serialize` does not touch `self.t`


Resolving patch:
diff --git a/chainer/optimizer.py b/chainer/optimizer.py
--- a/chainer/optimizer.py
+++ b/chainer/optimizer.py
@@ -274,6 +274,7 @@ def serialize(self, serializer):
             serializer (~chainer.AbstractSerializer): Serializer object.
 
         """
+        self.t = serializer('t', self.t)
         if self.state is None:
             if isinstance(serializer, serializer_module.Deserializer):
                 # try to initialize the state to retrieve state entries


Test patch:
diff --git a/tests/chainer_tests/serializers_tests/test_npz.py b/tests/chainer_tests/serializers_tests/test_npz.py
--- a/tests/chainer_tests/serializers_tests/test_npz.py
+++ b/tests/chainer_tests/serializers_tests/test_npz.py
@@ -512,10 +512,13 @@ def _check_chain_group(self, npzfile, state, prefix=''):
         self.assertSetEqual(set(npzfile.keys()), {prefix + x for x in keys})
 
     def _check_optimizer_group(self, npzfile, state, prefix=''):
-        keys = ('child/linear/W/msg',
+        keys = ('child/linear/W/t',
+                'child/linear/W/msg',
                 'child/linear/W/msdx',
+                'child/linear/b/t',
                 'child/linear/b/msg',
                 'child/linear/b/msdx',
+                'child/Wc/t',
                 'child/Wc/msg',
                 'child/Wc/msdx') + state
         self.assertEqual(set(npzfile.keys()),
@@ -536,7 +539,7 @@ def test_save_optimizer(self):
 
         with numpy.load(self.file) as npzfile:
             self._check_optimizer_group(
-                npzfile, ('Wp/msg', 'Wp/msdx', 'epoch', 't'), 'test/')
+                npzfile, ('Wp/t', 'Wp/msg', 'Wp/msdx', 'epoch', 't'), 'test/')
 
     def test_save_chain2(self):
         self._save_npz(self.file, self.parent, self.compress)
@@ -547,7 +550,7 @@ def test_save_optimizer2(self):
         self._save_npz(self.file, self.optimizer, self.compress)
         with numpy.load(self.file) as npzfile:
             self._check_optimizer_group(
-                npzfile, ('Wp/msg', 'Wp/msdx', 'epoch', 't'))
+                npzfile, ('Wp/t', 'Wp/msg', 'Wp/msdx', 'epoch', 't'))
 
     def test_load_optimizer_with_strict(self):
         for param in self.parent.params():