Error while using Apex
Author: chenyangh
Pull request: no

Hi, I am trying to do mixed precision training, but I have countered a problem that seems to be related to the LayerNorm implementation of Apex.  I have the following error msg while running the example (same error for my other code). 

`
Traceback (most recent call last):
  File "run_lm_finetuning.py", line 648, in <module>
    main()
  File "run_lm_finetuning.py", line 529, in main
    model = BertForPreTraining.from_pretrained(args.bert_model)
  File "/home/chenyang/anaconda3/envs/pytorch10/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 506, in from_pretrained
    model = cls(config, *inputs, **kwargs)
  File "/home/chenyang/anaconda3/envs/pytorch10/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 689, in __init__
    self.bert = BertModel(config)
  File "/home/chenyang/anaconda3/envs/pytorch10/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 600, in __init__
    self.embeddings = BertEmbeddings(config)
  File "/home/chenyang/anaconda3/envs/pytorch10/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 183, in __init__
    self.LayerNorm = BertLayerNorm(config.hidden_size, eps=1e-12)
  File "/home/chenyang/anaconda3/envs/pytorch10/lib/python3.6/site-packages/apex-0.1-py3.6.egg/apex/normalization/fused_layer_norm.py", line 126, in __init__
  File "/home/chenyang/anaconda3/envs/pytorch10/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'fused_layer_norm_cuda'
`

I am wondering if it is related to the version of Apex, so may I know which Apex checkpoint you used.

Comments:
Hi @chenyangh,
You need to install `apex` with the C++ and CUDA extensions:
```bash
git clone https://github.com/NVIDIA/apex.git
cd apex
python setup.py install --cuda_ext --cpp_ext
```

@thomwolf 
Thanks!

@thomwolf After doing what you wrote, I got this error.

torch.__version__  =  1.0.1.post2
Traceback (most recent call last):
  File "setup.py", line 60, in <module>
    raise RuntimeError("--cuda_ext was requested, but nvcc was not found.  Are you sure your environment has nvcc available?  If you're installing within a container from https://hub.docker.com/r/pytorch/pytorch, only images whose names contain 'devel' will provide nvcc.")
RuntimeError: --cuda_ext was requested, but nvcc was not found.  Are you sure your environment has nvcc available?  If you're installing within a container from https://hub.docker.com/r/pytorch/pytorch, only images whose names contain 'devel' will provide nvcc.

What else should I do for this?

You should refer to apex installation instructions.
Apex has slightly changed since my comment so best is to go read NVIDIA's README and installation instructions here: https://github.com/NVIDIA/apex

@kbulutozler You can change pytorch docker image version to pytorch/pytorch:1.3-cuda10.1-cudnn7-devel