'AmpOptimizerState' object has no attribute 'all_fp16_params' when Trainer amp_level
Problem
I encounter some questions when using Trainer.
Because I used precision=16 and amp_backend='apex' and amp_level='O2' in Trainer class.
But ger some error. Hope someone can help.
Traniner code
import pytorch_lightning as pl

trainer = pl.Trainer(gpus=1, 
                     precision=16, 
                     amp_backend='apex',
                     amp_level='O2', 
                     max_epochs=2, 
                     logger=logger,
                     callbacks=[checkpoint_callback])

trainer.fit(model, train_dataloader, valid_dataloader)
Error Message
/usr/local/lib/python3.6/dist-packages/pytorch_lightning/utilities/distributed.py:45: UserWarning: You have set progress_bar_refresh_rate < 20 on Google Colab. This may crash. Consider using progress_bar_refresh_rate >= 20 in Trainer.
  warnings.warn(*args, **kwargs)
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Using APEX 16bit precision.
Selected optimization level O2:  FP16 training with FP32 batchnorm and FP32 master weights.

Defaults for this optimization level are:
enabled                : True
opt_level              : O2
cast_model_type        : torch.float16
patch_torch_functions  : False
keep_batchnorm_fp32    : True
master_weights         : True
loss_scale             : dynamic
Processing user overrides (additional kwargs that are not None)...
After processing overrides, optimization options are:
enabled                : True
opt_level              : O2
cast_model_type        : torch.float16
patch_torch_functions  : False
keep_batchnorm_fp32    : True
master_weights         : True
loss_scale             : dynamic

| Name  | Type                            | Params
----------------------------------------------------------
0 | model | AlbertForSequenceClassification | 4.1 M 
/usr/local/lib/python3.6/dist-packages/datasets/arrow_dataset.py:850: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
  return torch.tensor(x, **format_kwargs)
Epoch 0: 0%
0/873 [00:00<?, ?it/s]
/usr/local/lib/python3.6/dist-packages/pytorch_lightning/utilities/distributed.py:45: UserWarning: training_step returned None if it was on purpose, ignore this warning...
  warnings.warn(*args, **kwargs)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-9-73afea14b588> in <module>()
     11 trainer.fit(sms_model, 
     12             train_sms_ds_dataloader,
---> 13             valid_sms_ds_dataloader)

10 frames
/usr/local/lib/python3.6/dist-packages/apex/amp/_process_optimizer.py in _master_params_to_model_params(self)
     15     stash = self._amp_stash
     16     if multi_tensor_applier.available:
---> 17         if len(stash.all_fp16_params) > 0:
     18             multi_tensor_applier(
     19                 stash.multi_tensor_scale,

AttributeError: 'AmpOptimizerState' object has no attribute 'all_fp16_params'

Reference link
NVIDIA/apex#341
I found the same problem in torch train loop. So, i not sure whether it is lib problem.
Hope someone can help and answer. I will thanks a lot.