TypeError: 'LayerNorm' object does not support indexing
Author: kyle-bong
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.11.0
- Platform: Linux-4.15.0-142-generic-x86_64-with-Ubuntu-16.04-xenial
- Python version: 3.6.13
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


### Who can help

@LysandreJik


## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. I made pre-trained BERT checkpoint file using google-research official code. (https://github.com/google-research/bert)
2. To using the pre-trainded BERT ckpt file in Pytorch, I run `transformers-cli convert` which you introduced here(https://github.com/huggingface/transformers/blob/master/docs/source/converting_tensorflow_models.rst#id5) 
3. But it throw an error "AttributeError: 'BertEmbeddings' object has no attribute 'bias' . And I refer to here(https://stackoverflow.com/questions/63689270/bertembeddings-object-has-no-attribute-bias-while-converting-tf-checkpoint), I changed the variable name of ckpt file. (layer_normalization -> LayerNorm)
4. Finally, I run again `transformers-cli convert`. But It throw an error "TypeError: 'LayerNorm' object does not support indexing".




## Expected behavior

I expected this code acted well.
```
transformers-cli convert --model_type bert\
  --tf_checkpoint=./bert_model.ckpt \
  --config=./bert_config.json \
  --pytorch_dump_output=./ckpt/pytorch_model.bin
```
But it throw out an error 
```
Traceback (most recent call last):
  File "/home/user/venv3.6/bin/transformers-cli", line 8, in <module>
    sys.exit(main())
  File "/home/user/venv3.6/lib/python3.6/site-packages/transformers/commands/transformers_cli.py", line 33, in main
    service.run()
  File "/home/user/venv3.6/lib/python3.6/site-packages/transformers/commands/convert.py", line 92, in run
    convert_tf_checkpoint_to_pytorch(self._tf_checkpoint, self._config, self._pytorch_dump_output)
  File "/home/user/venv3.6/lib/python3.6/site-packages/transformers/convert_bert_original_tf_checkpoint_to_pytorch.py", line 36, in convert_tf_checkpoint_to_pytorch
    load_tf_weights_in_bert(model, config, tf_checkpoint_path)
  File "/home/user/venv3.6/lib/python3.6/site-packages/transformers/modeling_bert.py", line 119, in load_tf_weights_in_bert
    pointer = pointer[num]
TypeError: 'LayerNorm' object does not support indexing
```

Please help me.
Thank you!

Comments:
The version of `transformers` you used seems very old, could you try updating it to the latest and doing conversion again?

@qqaatw Thank you for response. I tried in transformers 4.11.0, But it threw same error.

Thanks for reporting. I have tested the official [checkpoint](https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-12_H-768_A-12.zip), which could be converted successfully. Could you provide your checkpoint file that can reproduce this error?

@qqaatw Thank you very much! I made pre-training model using Korean data. 
But I cannot upload model here. I think It is because of the big size of zip file.
How can I provide you my checkpoint file?
I can only upload bert_config, json, vocab.txt and traing data file(wiki_00.txt). 
(Because I run pre-training for just test, I'm using very small data.) 
[jsonAndVocab.zip](https://github.com/huggingface/transformers/files/7250104/jsonAndVocab.zip)
[wiki_00.txt](https://github.com/huggingface/transformers/files/7250116/wiki_00.txt)



We can't inspect variable names without a checkpoint file, maybe using Google Drive to share it?

Before doing so, could you elaborate how you changed the variable names of the ckpt file? As I find that the latest official checkpoints are not using the name `layer_normalization` anymore, they are by default using `LayerNorm`. So could you also check if your training code is as latest as that of Google?

@qqaatw 

I uploaded my checkpoint file in this link: https://drive.google.com/drive/folders/1AGQ4ApxvIe8N3_eEGu60TobNlQTJ6LlF
Additionally, I uploaded the python file  which can change `layer_nomralization` to `LayerNorm` in the checkpoint file. 
I downloaded that file, and I ran below.
```
python tensorflow_rename_variables_backup.py \
	--checkpoint_dir=./model.ckpt \
  --replace_from=layer_normalization \
  --replace_to=LayerNorm
```



And, I used latest Google code for pre-training(I changed only path of some files). That is like below.

```
python create_pretraining_data.py \
  --input_file=./sample_text.txt \
  --output_file=/tmp/tf_examples.tfrecord \
  --vocab_file=$BERT_BASE_DIR/vocab.txt \
  --do_lower_case=True \
  --max_seq_length=128 \
  --max_predictions_per_seq=20 \
  --masked_lm_prob=0.15 \
  --random_seed=12345 \
  --dupe_factor=5
```

```
python run_pretraining.py \
  --input_file=/tmp/tf_examples.tfrecord \
  --output_dir=/tmp/pretraining_output \
  --do_train=True \
  --do_eval=True \
  --bert_config_file=$BERT_BASE_DIR/bert_config.json \
  --init_checkpoint=$BERT_BASE_DIR/bert_model.ckpt \
  --train_batch_size=32 \
  --max_seq_length=128 \
  --max_predictions_per_seq=20 \
  --num_train_steps=20 \
  --num_warmup_steps=10 \
  --learning_rate=2e-5
```
these code are introduced in README.md of https://github.com/google-research/bert


Thank you very much!

Hey @kyle-bong,

Sorry for the late reply.

Could you try this [script](https://gist.github.com/qqaatw/82b47c2b3da602fa1df604167bfcb9b0) with the following command to rename variables and then try doing conversion again?

```bash
python tensorflow_rename_variables_backup.py \
--checkpoint_dir=path/to/ckpt \
--replace_from="layer_normalization(_[0-9]+)*" \
--replace_to=LayerNorm
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.