Problems with running 'run_lm_finetuning.py' with bert
Author: bigzhouj
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
My rough modification configuration (local)
    root_path = "F://IdeaProjects/transformers"
    bert_path = "F://BERT/chinese_L-12_H-768_A-12" （Downloaded from the 'https://storage.googleapis.com/bert_models/2018_11_03/chinese_L-12_H-768_A-12.zip'）
    ## Required parameters
    parser.add_argument("--train_data_file", default=os.path.join(root_path, "data/train.txt"),...)
    parser.add_argument("--output_dir", default=os.path.join(root_path, "output")...)
    ## Other parameters
    parser.add_argument("--eval_data_file", default=os.path.join(root_path, "data/dev.txt"),...)
    parser.add_argument("--model_type", default="bert"...)
    parser.add_argument("--model_name_or_path", default=bert_path, ...)
    parser.add_argument("--mlm", action='store_true', default=True,...)
    parser.add_argument("--mlm_probability", type=float, default=0.15,...)
    parser.add_argument("--config_name", default=os.path.join(bert_path, "bert_config.json"),..)
    parser.add_argument("--tokenizer_name", default=bert_path, ...)

  
- [ Is my configuration correct?] 
 I changed this line, 'model = model_class.from_pretrained(args.model_name_or_path,
from_tf=True,#bool('.ckpt' in args.model_name_or_path),' Because my filename does not have this character（'.ckpt'）.

Then running 'run_lm_finetuning.py' causes the problem in the 'modeling_bert.py' file of the 'load_tf_weights_in_bert' method：
     'BertOnlyMLMHead' object has no attribute 'bias'

  
    

Comments:
@LysandreJik do we support this one?

We do support BERT in `run_lm_finetuning`, however, we do not support loading BERT checkpoints from the original BERT implementation.

If you wish to load a checkpoint that was pre-trained/fine-tuned using the original implementation (which seems to be what you're doing here), you can first convert to our implementation using [convert_bert_original_tf_checkpoint_to_pytorch](https://github.com/huggingface/transformers/blob/master/transformers/convert_bert_original_tf_checkpoint_to_pytorch.py), it will then be usable by `run_lm_finetuning`.

If you wish to use TensorFlow with the outputted model, you can use the script [convert_pytorch_checkpoint_to_tf2](https://github.com/huggingface/transformers/blob/master/transformers/convert_pytorch_checkpoint_to_tf2.py) which will convert the pytorch model back to tensorflow 2.

@LysandreJik - not related to fine-tuning but converting one of the fine-tuned (rum_lm_finetuning.py) model to tensorflow checkpoint. 

Here is the command I used:
bin/python3.6 convert_pytorch_checkpoint_to_tf2.py --tf_dump_path="../tf_test/" --model_type="bert" --pytorch_checkpoint_path="../pytorch_model.bin" --config_file='../config.json'

However, it was throwing the below error (log and stack trace)

 Converting model type 1/1 bert
 Converting checkpoint 1/15: ../pytorch_model.bin - model_type bert

Building TensorFlow model from configuration: {
  "attention_probs_dropout_prob": 0.1,
  "finetuning_task": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "is_decoder": false,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "num_labels": 2,
  "output_attentions": true,
  "output_hidden_states": true,
  "output_past": true,
  "pruned_heads": {},
  "torchscript": false,
  "type_vocab_size": 2,
  "use_bfloat16": false,
  "vocab_size": 28996
}

Traceback (most recent call last):
  File "/home/imagen/skc/bert/transformers-2.2.1/transformers/convert_pytorch_checkpoint_to_tf2.py", line 248, in <module>
    only_convert_finetuned_models=args.only_convert_finetuned_models)
  File "/home/imagen/skc/bert/transformers-2.2.1/transformers/convert_pytorch_checkpoint_to_tf2.py", line 194, in convert_all_pt_checkpoints_to_tf
    compare_with_pt_model=compare_with_pt_model)
  File "/home/imagen/skc/bert/transformers-2.2.1/transformers/convert_pytorch_checkpoint_to_tf2.py", line 115, in convert_pt_checkpoint_to_tf
    tf_model = load_pytorch_checkpoint_in_tf2_model(tf_model, pytorch_checkpoint_path)
  File "/home/imagen/skc/environments/.virtualenvs/lstm_dev_tf2x/lib/python3.6/site-packages/transformers/modeling_tf_pytorch_utils.py", line 82, in load_pytorch_checkpoint_in_tf2_model
    return load_pytorch_weights_in_tf2_model(tf_model, pt_state_dict, tf_inputs=tf_inputs, allow_missing_keys=allow_missing_keys)
  File "/home/imagen/skc/environments/.virtualenvs/lstm_dev_tf2x/lib/python3.6/site-packages/transformers/modeling_tf_pytorch_utils.py", line 145, in load_pytorch_weights_in_tf2_model
    assert name in pt_state_dict, "{} not found in PyTorch model".format(name)
**AssertionError: cls.seq_relationship.weight not found in PyTorch model**

Can you please explain what did go wrong with the conversion? This is one of the BERT-base fine-tuned model. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.