Repository: espnet/espnet

Issue:
Apex raise TypeError
I met following error when running transformer multi-gpu training.

```
# Environment
- date: `Sat Aug 24 01:26:59 JST 2019`
- python version: `3.7.3 (default, Mar 27 2019, 22:11:17)  [GCC 7.3.0]`
- espnet version: `espnet 0.5.1`
- chainer version: `chainer 6.0.0`
- pytorch version: `pytorch 1.0.1.post2`
- Git hash: `53d9979f23a3b08460ada1d5d3039b88c0962374`
  - Commit date: `Sat Aug 24 01:19:50 2019 +0900`
```

```
# asr_train.py --config conf/tuning/train_pytorch_transformer.v3.yaml --preprocess-conf conf/specaug.yaml --ngpu 4 --backend pytorch --outdir exp/train_960_pytorch_train_pytorch_transformer.v3_specaug/results --tensorboard-dir tensorboard/train_960_pytorch_train_pytorch_transformer.v3_specaug --debugmode 1 --dict data/lang_char/train_960_unigram5000_units.txt --debugdir exp/train_960_pytorch_train_pytorch_transformer.v3_specaug --minibatches 0 --verbose 0 --resume --train-json dump/train_960/deltafalse/data_unigram5000.json --valid-json dump/dev/deltafalse/data_unigram5000.json
/work5/t_hayashi/work/espnet_dev/tools/venv/lib/python3.7/site-packages/torch/nn/_reduction.py:49: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
  warnings.warn(warning.format(ret))
2019-08-24 01:26:02,701 (asr_train:286) WARNING: Skip DEBUG/INFO messages
Selected optimization level O1:  Insert automatic casts around Pytorch functions and Tensor methods.

Defaults for this optimization level are:
enabled                : True
opt_level              : O1
cast_model_type        : None
patch_torch_functions  : True
keep_batchnorm_fp32    : None
master_weights         : None
loss_scale             : dynamic
Processing user overrides (additional kwargs that are not None)...
After processing overrides, optimization options are:
enabled                : True
opt_level              : O1
cast_model_type        : None
patch_torch_functions  : True
keep_batchnorm_fp32    : None
master_weights         : None
loss_scale             : dynamic
Traceback (most recent call last):
  File "/work5/t_hayashi/work/espnet_dev/egs/librispeech/asr1/../../../espnet/bin/asr_train.py", line 352, in <module>
    main(sys.argv[1:])
  File "/work5/t_hayashi/work/espnet_dev/egs/librispeech/asr1/../../../espnet/bin/asr_train.py", line 339, in main
    train(args)
  File "/work5/t_hayashi/work/espnet_dev/espnet/asr/pytorch_backend/asr.py", line 425, in train
    model, optimizer = amp.initialize(model, optimizer, opt_level=args.train_dtype)
  File "/work5/t_hayashi/work/espnet_dev/tools/venv/lib/python3.7/site-packages/apex/amp/frontend.py", line 357, in initialize
    return _initialize(models, optimizers, _amp_state.opt_properties, num_losses, cast_model_outputs)
  File "/work5/t_hayashi/work/espnet_dev/tools/venv/lib/python3.7/site-packages/apex/amp/_initialize.py", line 164, in _initialize
    raise TypeError("optimizers must be either a single optimizer or a list of optimizers.")
TypeError: optimizers must be either a single optimizer or a list of optimizers.
```


Discussion:
https://github.com/NVIDIA/apex/blob/880ab925bce9f817a93988b021e12db5f67f7787/apex/amp/_initialize.py#L155-L164
we can pass the adam inside noam to apex.
https://github.com/espnet/espnet/blob/f9e8f676be21d530b473384fb8c45afb17396be4/espnet/nets/pytorch_backend/transformer/optimizer.py#L8
> we can pass the adam inside noam to apex.

Is it OK? (I’m wondering because `initialize` returns optimizer. if we pass `optimizer.optimizer`, it seems that it becomes just an Adam, not Noam.)
https://github.com/espnet/espnet/blob/f9e8f676be21d530b473384fb8c45afb17396be4/espnet/asr/pytorch_backend/asr.py#L425
Ah maybe I understand your point.
We can re-copy returned optimizer inside the Noam instance, right?
Yes. I think the patch like this can fix the problem
```
if args.opt == "noam":
    model, optimizer.optimizer = amp.initialize(model, optimizer.optimizer, opt_level=args.train_dtype)
```
Noam is just calling its optimizer. So if calling `optimizer.step()` in trainer is valid, from noam is also valid.
https://github.com/espnet/espnet/blob/0063dc373029e94518d4e0ca58528e3fb32f4b74/espnet/nets/pytorch_backend/transformer/optimizer.py#L26
I found that we also need to change here, then it works!
https://github.com/espnet/espnet/blob/0063dc373029e94518d4e0ca58528e3fb32f4b74/espnet/asr/pytorch_backend/asr.py#L188

I changed as follows:
```python
            # NOTE: for a compatibility with noam optimizer
            opt = optimizer.optimizer if hasattr(optimizer, "optimizer") else optimizer
            with amp.scale_loss(loss, opt) as scaled_loss:
```
I will make PR.

Resolving patch:
diff --git a/espnet/asr/pytorch_backend/asr.py b/espnet/asr/pytorch_backend/asr.py
--- a/espnet/asr/pytorch_backend/asr.py
+++ b/espnet/asr/pytorch_backend/asr.py
@@ -185,7 +185,9 @@ def update_core(self):
             loss = data_parallel(self.model, x, range(self.ngpu)).mean() / self.accum_grad
         if self.use_apex:
             from apex import amp
-            with amp.scale_loss(loss, optimizer) as scaled_loss:
+            # NOTE: for a compatibility with noam optimizer
+            opt = optimizer.optimizer if hasattr(optimizer, "optimizer") else optimizer
+            with amp.scale_loss(loss, opt) as scaled_loss:
                 scaled_loss.backward()
         else:
             loss.backward()
@@ -422,7 +424,10 @@ def train(args):
             logging.error(f"You need to install apex for --train-dtype {args.train_dtype}. "
                           "See https://github.com/NVIDIA/apex#linux")
             raise e
-        model, optimizer = amp.initialize(model, optimizer, opt_level=args.train_dtype)
+        if args.opt == 'noam':
+            model, optimizer.optimizer = amp.initialize(model, optimizer.optimizer, opt_level=args.train_dtype)
+        else:
+            model, optimizer = amp.initialize(model, optimizer, opt_level=args.train_dtype)
         use_apex = True
     else:
         use_apex = False


Test patch: