'LayerNorm' object has no attribute 'cls'
Author: chiyuzhang94
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I try to use `load_tf_weights_in_bert` to convert my fine-tuned tf classification model in Pytorch. I original trained the model by tensorflow BERT.
I used this code:
```import torch
from transformers.modeling_bert import BertConfig, BertForPreTraining, load_tf_weights_in_bert, BertForSequenceClassification

tf_checkpoint_path="./model.ckpt-98400"
bert_config_file = "./bert_config.json"
pytorch_dump_path="pytorch_bert"

config = BertConfig.from_json_file(bert_config_file)
config.num_labels = 21

print("Building PyTorch model from configuration: {}".format(str(config)))
model = BertForSequenceClassification(config)

# Load weights from tf checkpoint
load_tf_weights_in_bert(model, config, tf_checkpoint_path)

# Save pytorch-model
print("Save PyTorch model to {}".format(pytorch_dump_path))
torch.save(model.state_dict(), pytorch_dump_path)
```
I noticed  this method to solve my problem:  https://github.com/huggingface/transformers/issues/580#issuecomment-489519231

 I add to line in `modeling_bert.py`.

```          elif l[0] == 'output_bias' or l[0] == 'beta':
                pointer = getattr(pointer, 'cls')
                pointer = getattr(pointer, 'bias')
            elif l[0] == 'output_weights':
                pointer = getattr(pointer, 'cls')
                pointer = getattr(pointer, 'weight')
```
But I still get error `AttributeError: 'LayerNorm' object has no attribute 'cls'`

```
Building PyTorch model from configuration: {
  "attention_probs_dropout_prob": 0.1,
  "directionality": "bidi",
  "finetuning_task": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "num_labels": 21,
  "output_attentions": false,
  "output_hidden_states": false,
  "pooler_fc_size": 768,
  "pooler_num_attention_heads": 12,
  "pooler_num_fc_layers": 3,
  "pooler_size_per_head": 128,
  "pooler_type": "first_token_transform",
  "pruned_heads": {},
  "torchscript": false,
  "type_vocab_size": 2,
  "use_bfloat16": false,
  "vocab_size": 119547
}

I1009 15:47:03.337315 47631408520768 modeling_bert.py:65] Converting TensorFlow checkpoint from ./model.ckpt-98400
I1009 15:47:03.344796 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/LayerNorm/beta with shape [768]
I1009 15:47:03.350771 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:03.356130 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:03.361214 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/LayerNorm/gamma with shape [768]
I1009 15:47:03.366278 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:03.371291 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:03.376359 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/position_embeddings with shape [512, 768]
I1009 15:47:03.383015 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/position_embeddings/adam_m with shape [512, 768]
I1009 15:47:03.388718 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/position_embeddings/adam_v with shape [512, 768]
I1009 15:47:03.394378 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/token_type_embeddings with shape [2, 768]
I1009 15:47:03.400012 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/token_type_embeddings/adam_m with shape [2, 768]
I1009 15:47:03.405249 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/token_type_embeddings/adam_v with shape [2, 768]
I1009 15:47:03.410350 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/word_embeddings with shape [119547, 768]
I1009 15:47:03.575059 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/word_embeddings/adam_m with shape [119547, 768]
I1009 15:47:03.743357 47631408520768 modeling_bert.py:71] Loading TF weight bert/embeddings/word_embeddings/adam_v with shape [119547, 768]
I1009 15:47:03.908991 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/beta with shape [768]
I1009 15:47:03.915453 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:03.921177 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:03.926633 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/gamma with shape [768]
I1009 15:47:03.932333 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:03.937757 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:03.942878 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/output/dense/bias with shape [768]
I1009 15:47:03.947972 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/output/dense/bias/adam_m with shape [768]
I1009 15:47:03.953052 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/output/dense/bias/adam_v with shape [768]
I1009 15:47:03.958150 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/output/dense/kernel with shape [768, 768]
I1009 15:47:03.964268 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/output/dense/kernel/adam_m with shape [768, 768]
I1009 15:47:03.970259 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/output/dense/kernel/adam_v with shape [768, 768]
I1009 15:47:03.976348 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/key/bias with shape [768]
I1009 15:47:03.981755 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/key/bias/adam_m with shape [768]
I1009 15:47:03.986970 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/key/bias/adam_v with shape [768]
I1009 15:47:03.992337 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/key/kernel with shape [768, 768]
I1009 15:47:03.998313 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/key/kernel/adam_m with shape [768, 768]
I1009 15:47:04.004308 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/key/kernel/adam_v with shape [768, 768]
I1009 15:47:04.010322 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/query/bias with shape [768]
I1009 15:47:04.015577 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/query/bias/adam_m with shape [768]
I1009 15:47:04.020884 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/query/bias/adam_v with shape [768]
I1009 15:47:04.026228 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/query/kernel with shape [768, 768]
I1009 15:47:04.032151 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/query/kernel/adam_m with shape [768, 768]
I1009 15:47:04.038157 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/query/kernel/adam_v with shape [768, 768]
I1009 15:47:04.044193 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/value/bias with shape [768]
I1009 15:47:04.049786 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/value/bias/adam_m with shape [768]
I1009 15:47:04.055586 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/value/bias/adam_v with shape [768]
I1009 15:47:04.060960 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/value/kernel with shape [768, 768]
I1009 15:47:04.067193 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/value/kernel/adam_m with shape [768, 768]
I1009 15:47:04.073462 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/attention/self/value/kernel/adam_v with shape [768, 768]
I1009 15:47:04.079773 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/intermediate/dense/bias with shape [3072]
I1009 15:47:04.084890 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/intermediate/dense/bias/adam_m with shape [3072]
I1009 15:47:04.090381 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/intermediate/dense/bias/adam_v with shape [3072]
I1009 15:47:04.096321 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/intermediate/dense/kernel with shape [768, 3072]
I1009 15:47:04.105394 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/intermediate/dense/kernel/adam_m with shape [768, 3072]
I1009 15:47:04.114429 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/intermediate/dense/kernel/adam_v with shape [768, 3072]
I1009 15:47:04.123474 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/output/LayerNorm/beta with shape [768]
I1009 15:47:04.128776 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:04.133818 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:04.138939 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/output/LayerNorm/gamma with shape [768]
I1009 15:47:04.144054 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:04.149456 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:04.154736 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/output/dense/bias with shape [768]
I1009 15:47:04.159813 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/output/dense/bias/adam_m with shape [768]
I1009 15:47:04.165272 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/output/dense/bias/adam_v with shape [768]
I1009 15:47:04.170530 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/output/dense/kernel with shape [3072, 768]
I1009 15:47:04.179566 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/output/dense/kernel/adam_m with shape [3072, 768]
I1009 15:47:04.188571 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_0/output/dense/kernel/adam_v with shape [3072, 768]
I1009 15:47:04.197654 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/beta with shape [768]
I1009 15:47:04.203165 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:04.208374 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:04.213510 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/gamma with shape [768]
I1009 15:47:04.218636 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:04.223767 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:04.229122 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/output/dense/bias with shape [768]
I1009 15:47:04.234088 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/output/dense/bias/adam_m with shape [768]
I1009 15:47:04.239236 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/output/dense/bias/adam_v with shape [768]
I1009 15:47:04.244547 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/output/dense/kernel with shape [768, 768]
I1009 15:47:04.250508 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/output/dense/kernel/adam_m with shape [768, 768]
I1009 15:47:04.256635 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/output/dense/kernel/adam_v with shape [768, 768]
I1009 15:47:04.262910 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/key/bias with shape [768]
I1009 15:47:04.268387 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/key/bias/adam_m with shape [768]
I1009 15:47:04.273391 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/key/bias/adam_v with shape [768]
I1009 15:47:04.278625 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/key/kernel with shape [768, 768]
I1009 15:47:04.284610 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/key/kernel/adam_m with shape [768, 768]
I1009 15:47:04.290718 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/key/kernel/adam_v with shape [768, 768]
I1009 15:47:04.296881 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/query/bias with shape [768]
I1009 15:47:04.302387 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/query/bias/adam_m with shape [768]
I1009 15:47:04.307479 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/query/bias/adam_v with shape [768]
I1009 15:47:04.312391 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/query/kernel with shape [768, 768]
I1009 15:47:04.318128 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/query/kernel/adam_m with shape [768, 768]
I1009 15:47:04.324467 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/query/kernel/adam_v with shape [768, 768]
I1009 15:47:04.330887 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/value/bias with shape [768]
I1009 15:47:04.336219 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/value/bias/adam_m with shape [768]
I1009 15:47:04.341366 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/value/bias/adam_v with shape [768]
I1009 15:47:04.346387 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/value/kernel with shape [768, 768]
I1009 15:47:04.352337 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/value/kernel/adam_m with shape [768, 768]
I1009 15:47:04.358580 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/attention/self/value/kernel/adam_v with shape [768, 768]
I1009 15:47:04.365348 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/intermediate/dense/bias with shape [3072]
I1009 15:47:04.371170 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/intermediate/dense/bias/adam_m with shape [3072]
I1009 15:47:04.376696 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/intermediate/dense/bias/adam_v with shape [3072]
I1009 15:47:04.381763 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/intermediate/dense/kernel with shape [768, 3072]
I1009 15:47:04.390794 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/intermediate/dense/kernel/adam_m with shape [768, 3072]
I1009 15:47:04.400085 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/intermediate/dense/kernel/adam_v with shape [768, 3072]
I1009 15:47:04.409393 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/output/LayerNorm/beta with shape [768]
I1009 15:47:04.414902 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:04.420410 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:04.425521 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/output/LayerNorm/gamma with shape [768]
I1009 15:47:04.430895 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:04.436064 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:04.441035 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/output/dense/bias with shape [768]
I1009 15:47:04.446047 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/output/dense/bias/adam_m with shape [768]
I1009 15:47:04.451446 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/output/dense/bias/adam_v with shape [768]
I1009 15:47:04.456856 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/output/dense/kernel with shape [3072, 768]
I1009 15:47:04.465820 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/output/dense/kernel/adam_m with shape [3072, 768]
I1009 15:47:04.474748 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_1/output/dense/kernel/adam_v with shape [3072, 768]
I1009 15:47:04.484099 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/beta with shape [768]
I1009 15:47:04.489739 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:04.495579 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:04.500854 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/gamma with shape [768]
I1009 15:47:04.506203 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:04.511337 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:04.516572 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/output/dense/bias with shape [768]
I1009 15:47:04.521539 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/output/dense/bias/adam_m with shape [768]
I1009 15:47:04.526908 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/output/dense/bias/adam_v with shape [768]
I1009 15:47:04.532043 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/output/dense/kernel with shape [768, 768]
I1009 15:47:04.538394 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/output/dense/kernel/adam_m with shape [768, 768]
I1009 15:47:04.544577 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/output/dense/kernel/adam_v with shape [768, 768]
I1009 15:47:04.550608 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/key/bias with shape [768]
I1009 15:47:04.555929 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/key/bias/adam_m with shape [768]
I1009 15:47:04.561278 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/key/bias/adam_v with shape [768]
I1009 15:47:04.566563 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/key/kernel with shape [768, 768]
I1009 15:47:04.573011 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/key/kernel/adam_m with shape [768, 768]
I1009 15:47:04.579270 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/key/kernel/adam_v with shape [768, 768]
I1009 15:47:04.585390 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/query/bias with shape [768]
I1009 15:47:04.590508 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/query/bias/adam_m with shape [768]
I1009 15:47:04.595518 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/query/bias/adam_v with shape [768]
I1009 15:47:04.601018 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/query/kernel with shape [768, 768]
I1009 15:47:04.607660 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/query/kernel/adam_m with shape [768, 768]
I1009 15:47:04.614161 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/query/kernel/adam_v with shape [768, 768]
I1009 15:47:04.620493 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/value/bias with shape [768]
I1009 15:47:04.625949 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/value/bias/adam_m with shape [768]
I1009 15:47:04.631571 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/value/bias/adam_v with shape [768]
I1009 15:47:04.637210 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/value/kernel with shape [768, 768]
I1009 15:47:04.643343 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/value/kernel/adam_m with shape [768, 768]
I1009 15:47:04.649336 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/attention/self/value/kernel/adam_v with shape [768, 768]
I1009 15:47:04.655313 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/intermediate/dense/bias with shape [3072]
I1009 15:47:04.660685 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/intermediate/dense/bias/adam_m with shape [3072]
I1009 15:47:04.666079 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/intermediate/dense/bias/adam_v with shape [3072]
I1009 15:47:04.671396 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/intermediate/dense/kernel with shape [768, 3072]
I1009 15:47:04.680429 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/intermediate/dense/kernel/adam_m with shape [768, 3072]
I1009 15:47:04.689695 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/intermediate/dense/kernel/adam_v with shape [768, 3072]
I1009 15:47:04.698902 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/output/LayerNorm/beta with shape [768]
I1009 15:47:04.704335 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:04.710071 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:04.715506 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/output/LayerNorm/gamma with shape [768]
I1009 15:47:04.720938 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:04.726950 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:04.732291 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/output/dense/bias with shape [768]
I1009 15:47:04.737487 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/output/dense/bias/adam_m with shape [768]
I1009 15:47:04.742859 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/output/dense/bias/adam_v with shape [768]
I1009 15:47:04.748318 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/output/dense/kernel with shape [3072, 768]
I1009 15:47:04.757482 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/output/dense/kernel/adam_m with shape [3072, 768]
I1009 15:47:04.766298 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_10/output/dense/kernel/adam_v with shape [3072, 768]
I1009 15:47:04.775268 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/beta with shape [768]
I1009 15:47:04.780487 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:04.785785 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:04.791079 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/gamma with shape [768]
I1009 15:47:04.796396 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:04.801618 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:04.806764 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/output/dense/bias with shape [768]
I1009 15:47:04.811792 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/output/dense/bias/adam_m with shape [768]
I1009 15:47:04.817137 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/output/dense/bias/adam_v with shape [768]
I1009 15:47:04.822446 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/output/dense/kernel with shape [768, 768]
I1009 15:47:04.828487 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/output/dense/kernel/adam_m with shape [768, 768]
I1009 15:47:04.834693 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/output/dense/kernel/adam_v with shape [768, 768]
I1009 15:47:04.840850 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/key/bias with shape [768]
I1009 15:47:04.846439 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/key/bias/adam_m with shape [768]
I1009 15:47:04.851520 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/key/bias/adam_v with shape [768]
I1009 15:47:04.856792 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/key/kernel with shape [768, 768]
I1009 15:47:04.862816 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/key/kernel/adam_m with shape [768, 768]
I1009 15:47:04.869044 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/key/kernel/adam_v with shape [768, 768]
I1009 15:47:04.875606 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/query/bias with shape [768]
I1009 15:47:04.880790 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/query/bias/adam_m with shape [768]
I1009 15:47:04.886349 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/query/bias/adam_v with shape [768]
I1009 15:47:04.892000 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/query/kernel with shape [768, 768]
I1009 15:47:04.898570 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/query/kernel/adam_m with shape [768, 768]
I1009 15:47:04.904909 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/query/kernel/adam_v with shape [768, 768]
I1009 15:47:04.911062 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/value/bias with shape [768]
I1009 15:47:04.916296 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/value/bias/adam_m with shape [768]
I1009 15:47:04.921895 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/value/bias/adam_v with shape [768]
I1009 15:47:04.927209 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/value/kernel with shape [768, 768]
I1009 15:47:04.933332 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/value/kernel/adam_m with shape [768, 768]
I1009 15:47:04.939111 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/attention/self/value/kernel/adam_v with shape [768, 768]
I1009 15:47:04.945370 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/intermediate/dense/bias with shape [3072]
I1009 15:47:04.950793 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/intermediate/dense/bias/adam_m with shape [3072]
I1009 15:47:04.955806 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/intermediate/dense/bias/adam_v with shape [3072]
I1009 15:47:04.961072 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/intermediate/dense/kernel with shape [768, 3072]
I1009 15:47:04.970463 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/intermediate/dense/kernel/adam_m with shape [768, 3072]
I1009 15:47:04.979833 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/intermediate/dense/kernel/adam_v with shape [768, 3072]
I1009 15:47:04.989530 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/output/LayerNorm/beta with shape [768]
I1009 15:47:04.995094 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:05.000446 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:05.005793 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/output/LayerNorm/gamma with shape [768]
I1009 15:47:05.011140 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:05.016781 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:05.022187 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/output/dense/bias with shape [768]
I1009 15:47:05.027360 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/output/dense/bias/adam_m with shape [768]
I1009 15:47:05.032415 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/output/dense/bias/adam_v with shape [768]
I1009 15:47:05.037586 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/output/dense/kernel with shape [3072, 768]
I1009 15:47:05.046856 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/output/dense/kernel/adam_m with shape [3072, 768]
I1009 15:47:05.055976 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_11/output/dense/kernel/adam_v with shape [3072, 768]
I1009 15:47:05.065228 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/beta with shape [768]
I1009 15:47:05.070782 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:05.075919 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:05.081003 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/gamma with shape [768]
I1009 15:47:05.086196 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:05.091385 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:05.096379 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/output/dense/bias with shape [768]
I1009 15:47:05.101588 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/output/dense/bias/adam_m with shape [768]
I1009 15:47:05.106899 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/output/dense/bias/adam_v with shape [768]
I1009 15:47:05.112151 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/output/dense/kernel with shape [768, 768]
I1009 15:47:05.117895 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/output/dense/kernel/adam_m with shape [768, 768]
I1009 15:47:05.123686 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/output/dense/kernel/adam_v with shape [768, 768]
I1009 15:47:05.129775 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/key/bias with shape [768]
I1009 15:47:05.135063 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/key/bias/adam_m with shape [768]
I1009 15:47:05.140272 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/key/bias/adam_v with shape [768]
I1009 15:47:05.145637 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/key/kernel with shape [768, 768]
I1009 15:47:05.151918 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/key/kernel/adam_m with shape [768, 768]
I1009 15:47:05.158202 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/key/kernel/adam_v with shape [768, 768]
I1009 15:47:05.164404 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/query/bias with shape [768]
I1009 15:47:05.170278 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/query/bias/adam_m with shape [768]
I1009 15:47:05.175527 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/query/bias/adam_v with shape [768]
I1009 15:47:05.180814 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/query/kernel with shape [768, 768]
I1009 15:47:05.187093 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/query/kernel/adam_m with shape [768, 768]
I1009 15:47:05.193276 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/query/kernel/adam_v with shape [768, 768]
I1009 15:47:05.199446 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/value/bias with shape [768]
I1009 15:47:05.204873 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/value/bias/adam_m with shape [768]
I1009 15:47:05.210621 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/value/bias/adam_v with shape [768]
I1009 15:47:05.215833 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/value/kernel with shape [768, 768]
I1009 15:47:05.221964 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/value/kernel/adam_m with shape [768, 768]
I1009 15:47:05.228211 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/attention/self/value/kernel/adam_v with shape [768, 768]
I1009 15:47:05.234716 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/intermediate/dense/bias with shape [3072]
I1009 15:47:05.240120 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/intermediate/dense/bias/adam_m with shape [3072]
I1009 15:47:05.245725 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/intermediate/dense/bias/adam_v with shape [3072]
I1009 15:47:05.250889 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/intermediate/dense/kernel with shape [768, 3072]
I1009 15:47:05.260031 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/intermediate/dense/kernel/adam_m with shape [768, 3072]
I1009 15:47:05.269109 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/intermediate/dense/kernel/adam_v with shape [768, 3072]
I1009 15:47:05.278731 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/output/LayerNorm/beta with shape [768]
I1009 15:47:05.283982 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:05.289493 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:05.294659 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/output/LayerNorm/gamma with shape [768]
I1009 15:47:05.299949 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:05.305051 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:05.310529 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/output/dense/bias with shape [768]
I1009 15:47:05.315993 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/output/dense/bias/adam_m with shape [768]
I1009 15:47:05.321487 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/output/dense/bias/adam_v with shape [768]
I1009 15:47:05.326727 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/output/dense/kernel with shape [3072, 768]
I1009 15:47:05.335873 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/output/dense/kernel/adam_m with shape [3072, 768]
I1009 15:47:05.345036 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_2/output/dense/kernel/adam_v with shape [3072, 768]
I1009 15:47:05.354362 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/beta with shape [768]
I1009 15:47:05.359932 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:05.365148 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:05.370391 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/gamma with shape [768]
I1009 15:47:05.375550 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:05.380681 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:05.385793 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/output/dense/bias with shape [768]
I1009 15:47:05.390934 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/output/dense/bias/adam_m with shape [768]
I1009 15:47:05.396157 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/output/dense/bias/adam_v with shape [768]
I1009 15:47:05.401309 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/output/dense/kernel with shape [768, 768]
I1009 15:47:05.407191 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/output/dense/kernel/adam_m with shape [768, 768]
I1009 15:47:05.413287 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/output/dense/kernel/adam_v with shape [768, 768]
I1009 15:47:05.419596 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/key/bias with shape [768]
I1009 15:47:05.424950 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/key/bias/adam_m with shape [768]
I1009 15:47:05.430454 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/key/bias/adam_v with shape [768]
I1009 15:47:05.435939 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/key/kernel with shape [768, 768]
I1009 15:47:05.441898 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/key/kernel/adam_m with shape [768, 768]
I1009 15:47:05.448148 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/key/kernel/adam_v with shape [768, 768]
I1009 15:47:05.454164 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/query/bias with shape [768]
I1009 15:47:05.459583 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/query/bias/adam_m with shape [768]
I1009 15:47:05.465055 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/query/bias/adam_v with shape [768]
I1009 15:47:05.470114 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/query/kernel with shape [768, 768]
I1009 15:47:05.476166 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/query/kernel/adam_m with shape [768, 768]
I1009 15:47:05.482553 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/query/kernel/adam_v with shape [768, 768]
I1009 15:47:05.489023 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/value/bias with shape [768]
I1009 15:47:05.494502 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/value/bias/adam_m with shape [768]
I1009 15:47:05.500063 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/value/bias/adam_v with shape [768]
I1009 15:47:05.505194 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/value/kernel with shape [768, 768]
I1009 15:47:05.511651 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/value/kernel/adam_m with shape [768, 768]
I1009 15:47:05.517767 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/attention/self/value/kernel/adam_v with shape [768, 768]
I1009 15:47:05.524090 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/intermediate/dense/bias with shape [3072]
I1009 15:47:05.529507 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/intermediate/dense/bias/adam_m with shape [3072]
I1009 15:47:05.534897 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/intermediate/dense/bias/adam_v with shape [3072]
I1009 15:47:05.540130 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/intermediate/dense/kernel with shape [768, 3072]
I1009 15:47:05.549204 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/intermediate/dense/kernel/adam_m with shape [768, 3072]
I1009 15:47:05.558589 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/intermediate/dense/kernel/adam_v with shape [768, 3072]
I1009 15:47:05.568243 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/output/LayerNorm/beta with shape [768]
I1009 15:47:05.573777 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:05.578975 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:05.584223 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/output/LayerNorm/gamma with shape [768]
I1009 15:47:05.589388 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:05.594596 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:05.600200 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/output/dense/bias with shape [768]
I1009 15:47:05.605475 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/output/dense/bias/adam_m with shape [768]
I1009 15:47:05.610627 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/output/dense/bias/adam_v with shape [768]
I1009 15:47:05.616336 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/output/dense/kernel with shape [3072, 768]
I1009 15:47:05.625247 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/output/dense/kernel/adam_m with shape [3072, 768]
I1009 15:47:05.634833 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_3/output/dense/kernel/adam_v with shape [3072, 768]
I1009 15:47:05.644434 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/beta with shape [768]
I1009 15:47:05.649833 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:05.655650 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:05.660920 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/gamma with shape [768]
I1009 15:47:05.666323 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:05.671495 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:05.676601 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/output/dense/bias with shape [768]
I1009 15:47:05.681863 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/output/dense/bias/adam_m with shape [768]
I1009 15:47:05.686859 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/output/dense/bias/adam_v with shape [768]
I1009 15:47:05.692172 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/output/dense/kernel with shape [768, 768]
I1009 15:47:05.698084 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/output/dense/kernel/adam_m with shape [768, 768]
I1009 15:47:05.703886 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/output/dense/kernel/adam_v with shape [768, 768]
I1009 15:47:05.709926 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/key/bias with shape [768]
I1009 15:47:05.715487 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/key/bias/adam_m with shape [768]
I1009 15:47:05.720532 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/key/bias/adam_v with shape [768]
I1009 15:47:05.725626 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/key/kernel with shape [768, 768]
I1009 15:47:05.731551 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/key/kernel/adam_m with shape [768, 768]
I1009 15:47:05.737704 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/key/kernel/adam_v with shape [768, 768]
I1009 15:47:05.743711 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/query/bias with shape [768]
I1009 15:47:05.748904 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/query/bias/adam_m with shape [768]
I1009 15:47:05.754198 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/query/bias/adam_v with shape [768]
I1009 15:47:05.759485 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/query/kernel with shape [768, 768]
I1009 15:47:05.765789 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/query/kernel/adam_m with shape [768, 768]
I1009 15:47:05.772267 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/query/kernel/adam_v with shape [768, 768]
I1009 15:47:05.778261 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/value/bias with shape [768]
I1009 15:47:05.783635 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/value/bias/adam_m with shape [768]
I1009 15:47:05.788740 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/value/bias/adam_v with shape [768]
I1009 15:47:05.794108 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/value/kernel with shape [768, 768]
I1009 15:47:05.800311 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/value/kernel/adam_m with shape [768, 768]
I1009 15:47:05.806456 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/attention/self/value/kernel/adam_v with shape [768, 768]
I1009 15:47:05.812658 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/intermediate/dense/bias with shape [3072]
I1009 15:47:05.818162 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/intermediate/dense/bias/adam_m with shape [3072]
I1009 15:47:05.823700 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/intermediate/dense/bias/adam_v with shape [3072]
I1009 15:47:05.828876 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/intermediate/dense/kernel with shape [768, 3072]
I1009 15:47:05.837978 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/intermediate/dense/kernel/adam_m with shape [768, 3072]
I1009 15:47:05.847428 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/intermediate/dense/kernel/adam_v with shape [768, 3072]
I1009 15:47:05.856863 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/output/LayerNorm/beta with shape [768]
I1009 15:47:05.862351 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:05.867667 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:05.873092 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/output/LayerNorm/gamma with shape [768]
I1009 15:47:05.878434 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:05.883496 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:05.888758 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/output/dense/bias with shape [768]
I1009 15:47:05.894265 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/output/dense/bias/adam_m with shape [768]
I1009 15:47:05.899498 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/output/dense/bias/adam_v with shape [768]
I1009 15:47:05.904521 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/output/dense/kernel with shape [3072, 768]
I1009 15:47:05.913688 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/output/dense/kernel/adam_m with shape [3072, 768]
I1009 15:47:05.922709 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_4/output/dense/kernel/adam_v with shape [3072, 768]
I1009 15:47:05.931876 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/beta with shape [768]
I1009 15:47:05.937582 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:05.942691 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:05.947911 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/gamma with shape [768]
I1009 15:47:05.953302 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:05.958757 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:05.963871 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/output/dense/bias with shape [768]
I1009 15:47:05.969243 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/output/dense/bias/adam_m with shape [768]
I1009 15:47:05.974361 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/output/dense/bias/adam_v with shape [768]
I1009 15:47:05.979577 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/output/dense/kernel with shape [768, 768]
I1009 15:47:05.985836 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/output/dense/kernel/adam_m with shape [768, 768]
I1009 15:47:05.991879 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/output/dense/kernel/adam_v with shape [768, 768]
I1009 15:47:05.998053 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/key/bias with shape [768]
I1009 15:47:06.003427 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/key/bias/adam_m with shape [768]
I1009 15:47:06.008820 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/key/bias/adam_v with shape [768]
I1009 15:47:06.014085 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/key/kernel with shape [768, 768]
I1009 15:47:06.020233 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/key/kernel/adam_m with shape [768, 768]
I1009 15:47:06.026285 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/key/kernel/adam_v with shape [768, 768]
I1009 15:47:06.032388 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/query/bias with shape [768]
I1009 15:47:06.038056 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/query/bias/adam_m with shape [768]
I1009 15:47:06.043298 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/query/bias/adam_v with shape [768]
I1009 15:47:06.048563 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/query/kernel with shape [768, 768]
I1009 15:47:06.054869 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/query/kernel/adam_m with shape [768, 768]
I1009 15:47:06.061104 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/query/kernel/adam_v with shape [768, 768]
I1009 15:47:06.067418 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/value/bias with shape [768]
I1009 15:47:06.072948 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/value/bias/adam_m with shape [768]
I1009 15:47:06.078219 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/value/bias/adam_v with shape [768]
I1009 15:47:06.083603 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/value/kernel with shape [768, 768]
I1009 15:47:06.090325 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/value/kernel/adam_m with shape [768, 768]
I1009 15:47:06.096712 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/attention/self/value/kernel/adam_v with shape [768, 768]
I1009 15:47:06.102761 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/intermediate/dense/bias with shape [3072]
I1009 15:47:06.108000 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/intermediate/dense/bias/adam_m with shape [3072]
I1009 15:47:06.113360 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/intermediate/dense/bias/adam_v with shape [3072]
I1009 15:47:06.118700 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/intermediate/dense/kernel with shape [768, 3072]
I1009 15:47:06.128049 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/intermediate/dense/kernel/adam_m with shape [768, 3072]
I1009 15:47:06.137369 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/intermediate/dense/kernel/adam_v with shape [768, 3072]
I1009 15:47:06.146910 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/output/LayerNorm/beta with shape [768]
I1009 15:47:06.152923 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:06.158300 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:06.163956 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/output/LayerNorm/gamma with shape [768]
I1009 15:47:06.169229 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:06.174710 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:06.179936 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/output/dense/bias with shape [768]
I1009 15:47:06.185377 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/output/dense/bias/adam_m with shape [768]
I1009 15:47:06.190515 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/output/dense/bias/adam_v with shape [768]
I1009 15:47:06.196292 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/output/dense/kernel with shape [3072, 768]
I1009 15:47:06.205745 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/output/dense/kernel/adam_m with shape [3072, 768]
I1009 15:47:06.215335 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_5/output/dense/kernel/adam_v with shape [3072, 768]
I1009 15:47:06.224854 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/beta with shape [768]
I1009 15:47:06.230671 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:06.235839 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:06.241382 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/gamma with shape [768]
I1009 15:47:06.246639 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:06.251890 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:06.257052 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/output/dense/bias with shape [768]
I1009 15:47:06.262314 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/output/dense/bias/adam_m with shape [768]
I1009 15:47:06.267627 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/output/dense/bias/adam_v with shape [768]
I1009 15:47:06.272980 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/output/dense/kernel with shape [768, 768]
I1009 15:47:06.279123 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/output/dense/kernel/adam_m with shape [768, 768]
I1009 15:47:06.285258 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/output/dense/kernel/adam_v with shape [768, 768]
I1009 15:47:06.291564 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/key/bias with shape [768]
I1009 15:47:06.296821 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/key/bias/adam_m with shape [768]
I1009 15:47:06.302075 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/key/bias/adam_v with shape [768]
I1009 15:47:06.307455 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/key/kernel with shape [768, 768]
I1009 15:47:06.313520 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/key/kernel/adam_m with shape [768, 768]
I1009 15:47:06.319566 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/key/kernel/adam_v with shape [768, 768]
I1009 15:47:06.325647 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/query/bias with shape [768]
I1009 15:47:06.331008 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/query/bias/adam_m with shape [768]
I1009 15:47:06.336525 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/query/bias/adam_v with shape [768]
I1009 15:47:06.342104 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/query/kernel with shape [768, 768]
I1009 15:47:06.348503 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/query/kernel/adam_m with shape [768, 768]
I1009 15:47:06.354544 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/query/kernel/adam_v with shape [768, 768]
I1009 15:47:06.361086 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/value/bias with shape [768]
I1009 15:47:06.366425 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/value/bias/adam_m with shape [768]
I1009 15:47:06.371979 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/value/bias/adam_v with shape [768]
I1009 15:47:06.377128 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/value/kernel with shape [768, 768]
I1009 15:47:06.383063 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/value/kernel/adam_m with shape [768, 768]
I1009 15:47:06.389229 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/attention/self/value/kernel/adam_v with shape [768, 768]
I1009 15:47:06.395555 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/intermediate/dense/bias with shape [3072]
I1009 15:47:06.401404 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/intermediate/dense/bias/adam_m with shape [3072]
I1009 15:47:06.406746 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/intermediate/dense/bias/adam_v with shape [3072]
I1009 15:47:06.411887 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/intermediate/dense/kernel with shape [768, 3072]
I1009 15:47:06.420855 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/intermediate/dense/kernel/adam_m with shape [768, 3072]
I1009 15:47:06.430287 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/intermediate/dense/kernel/adam_v with shape [768, 3072]
I1009 15:47:06.439881 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/output/LayerNorm/beta with shape [768]
I1009 15:47:06.445644 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:06.450891 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:06.456028 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/output/LayerNorm/gamma with shape [768]
I1009 15:47:06.461563 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:06.467115 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:06.472524 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/output/dense/bias with shape [768]
I1009 15:47:06.478100 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/output/dense/bias/adam_m with shape [768]
I1009 15:47:06.483274 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/output/dense/bias/adam_v with shape [768]
I1009 15:47:06.488587 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/output/dense/kernel with shape [3072, 768]
I1009 15:47:06.497782 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/output/dense/kernel/adam_m with shape [3072, 768]
I1009 15:47:06.506931 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_6/output/dense/kernel/adam_v with shape [3072, 768]
I1009 15:47:06.516527 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/beta with shape [768]
I1009 15:47:06.522197 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:06.527507 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:06.532787 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/gamma with shape [768]
I1009 15:47:06.538027 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:06.543310 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:06.548686 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/output/dense/bias with shape [768]
I1009 15:47:06.553850 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/output/dense/bias/adam_m with shape [768]
I1009 15:47:06.559104 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/output/dense/bias/adam_v with shape [768]
I1009 15:47:06.564384 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/output/dense/kernel with shape [768, 768]
I1009 15:47:06.570770 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/output/dense/kernel/adam_m with shape [768, 768]
I1009 15:47:06.576959 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/output/dense/kernel/adam_v with shape [768, 768]
I1009 15:47:06.583284 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/key/bias with shape [768]
I1009 15:47:06.588658 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/key/bias/adam_m with shape [768]
I1009 15:47:06.593865 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/key/bias/adam_v with shape [768]
I1009 15:47:06.598938 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/key/kernel with shape [768, 768]
I1009 15:47:06.604933 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/key/kernel/adam_m with shape [768, 768]
I1009 15:47:06.611174 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/key/kernel/adam_v with shape [768, 768]
I1009 15:47:06.617308 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/query/bias with shape [768]
I1009 15:47:06.622585 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/query/bias/adam_m with shape [768]
I1009 15:47:06.627687 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/query/bias/adam_v with shape [768]
I1009 15:47:06.633019 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/query/kernel with shape [768, 768]
I1009 15:47:06.639252 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/query/kernel/adam_m with shape [768, 768]
I1009 15:47:06.645524 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/query/kernel/adam_v with shape [768, 768]
I1009 15:47:06.651858 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/value/bias with shape [768]
I1009 15:47:06.657348 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/value/bias/adam_m with shape [768]
I1009 15:47:06.662594 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/value/bias/adam_v with shape [768]
I1009 15:47:06.667656 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/value/kernel with shape [768, 768]
I1009 15:47:06.673909 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/value/kernel/adam_m with shape [768, 768]
I1009 15:47:06.680046 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/attention/self/value/kernel/adam_v with shape [768, 768]
I1009 15:47:06.686088 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/intermediate/dense/bias with shape [3072]
I1009 15:47:06.691427 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/intermediate/dense/bias/adam_m with shape [3072]
I1009 15:47:06.696527 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/intermediate/dense/bias/adam_v with shape [3072]
I1009 15:47:06.701802 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/intermediate/dense/kernel with shape [768, 3072]
I1009 15:47:06.710916 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/intermediate/dense/kernel/adam_m with shape [768, 3072]
I1009 15:47:06.719931 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/intermediate/dense/kernel/adam_v with shape [768, 3072]
I1009 15:47:06.729498 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/output/LayerNorm/beta with shape [768]
I1009 15:47:06.734817 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:06.739971 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:06.745015 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/output/LayerNorm/gamma with shape [768]
I1009 15:47:06.750005 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:06.755349 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:06.760453 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/output/dense/bias with shape [768]
I1009 15:47:06.765552 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/output/dense/bias/adam_m with shape [768]
I1009 15:47:06.770809 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/output/dense/bias/adam_v with shape [768]
I1009 15:47:06.775943 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/output/dense/kernel with shape [3072, 768]
I1009 15:47:06.785551 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/output/dense/kernel/adam_m with shape [3072, 768]
I1009 15:47:06.795017 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_7/output/dense/kernel/adam_v with shape [3072, 768]
I1009 15:47:06.804197 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/beta with shape [768]
I1009 15:47:06.809925 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:06.815610 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:06.820969 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/gamma with shape [768]
I1009 15:47:06.825976 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:06.831141 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:06.836268 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/output/dense/bias with shape [768]
I1009 15:47:06.841736 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/output/dense/bias/adam_m with shape [768]
I1009 15:47:06.847084 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/output/dense/bias/adam_v with shape [768]
I1009 15:47:06.852262 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/output/dense/kernel with shape [768, 768]
I1009 15:47:06.858251 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/output/dense/kernel/adam_m with shape [768, 768]
I1009 15:47:06.864098 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/output/dense/kernel/adam_v with shape [768, 768]
I1009 15:47:06.870030 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/key/bias with shape [768]
I1009 15:47:06.875577 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/key/bias/adam_m with shape [768]
I1009 15:47:06.881125 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/key/bias/adam_v with shape [768]
I1009 15:47:06.886339 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/key/kernel with shape [768, 768]
I1009 15:47:06.892580 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/key/kernel/adam_m with shape [768, 768]
I1009 15:47:06.898903 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/key/kernel/adam_v with shape [768, 768]
I1009 15:47:06.905303 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/query/bias with shape [768]
I1009 15:47:06.910978 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/query/bias/adam_m with shape [768]
I1009 15:47:06.916463 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/query/bias/adam_v with shape [768]
I1009 15:47:06.921678 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/query/kernel with shape [768, 768]
I1009 15:47:06.927710 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/query/kernel/adam_m with shape [768, 768]
I1009 15:47:06.933935 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/query/kernel/adam_v with shape [768, 768]
I1009 15:47:06.940702 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/value/bias with shape [768]
I1009 15:47:06.946018 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/value/bias/adam_m with shape [768]
I1009 15:47:06.951382 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/value/bias/adam_v with shape [768]
I1009 15:47:06.957077 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/value/kernel with shape [768, 768]
I1009 15:47:06.963493 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/value/kernel/adam_m with shape [768, 768]
I1009 15:47:06.969939 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/attention/self/value/kernel/adam_v with shape [768, 768]
I1009 15:47:06.976311 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/intermediate/dense/bias with shape [3072]
I1009 15:47:06.982092 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/intermediate/dense/bias/adam_m with shape [3072]
I1009 15:47:06.987428 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/intermediate/dense/bias/adam_v with shape [3072]
I1009 15:47:06.992400 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/intermediate/dense/kernel with shape [768, 3072]
I1009 15:47:07.001886 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/intermediate/dense/kernel/adam_m with shape [768, 3072]
I1009 15:47:07.011502 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/intermediate/dense/kernel/adam_v with shape [768, 3072]
I1009 15:47:07.020718 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/output/LayerNorm/beta with shape [768]
I1009 15:47:07.026179 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:07.031653 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:07.037088 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/output/LayerNorm/gamma with shape [768]
I1009 15:47:07.042681 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:07.047634 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:07.052863 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/output/dense/bias with shape [768]
I1009 15:47:07.058085 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/output/dense/bias/adam_m with shape [768]
I1009 15:47:07.063718 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/output/dense/bias/adam_v with shape [768]
I1009 15:47:07.069069 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/output/dense/kernel with shape [3072, 768]
I1009 15:47:07.078701 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/output/dense/kernel/adam_m with shape [3072, 768]
I1009 15:47:07.088213 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_8/output/dense/kernel/adam_v with shape [3072, 768]
I1009 15:47:07.097591 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/beta with shape [768]
I1009 15:47:07.103389 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:07.109018 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:07.114469 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/gamma with shape [768]
I1009 15:47:07.119768 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:07.125015 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:07.130143 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/output/dense/bias with shape [768]
I1009 15:47:07.135096 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/output/dense/bias/adam_m with shape [768]
I1009 15:47:07.140460 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/output/dense/bias/adam_v with shape [768]
I1009 15:47:07.145726 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/output/dense/kernel with shape [768, 768]
I1009 15:47:07.151650 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/output/dense/kernel/adam_m with shape [768, 768]
I1009 15:47:07.157860 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/output/dense/kernel/adam_v with shape [768, 768]
I1009 15:47:07.163643 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/key/bias with shape [768]
I1009 15:47:07.168712 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/key/bias/adam_m with shape [768]
I1009 15:47:07.174322 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/key/bias/adam_v with shape [768]
I1009 15:47:07.179837 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/key/kernel with shape [768, 768]
I1009 15:47:07.185978 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/key/kernel/adam_m with shape [768, 768]
I1009 15:47:07.192014 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/key/kernel/adam_v with shape [768, 768]
I1009 15:47:07.198294 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/query/bias with shape [768]
I1009 15:47:07.203473 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/query/bias/adam_m with shape [768]
I1009 15:47:07.209064 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/query/bias/adam_v with shape [768]
I1009 15:47:07.214386 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/query/kernel with shape [768, 768]
I1009 15:47:07.220609 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/query/kernel/adam_m with shape [768, 768]
I1009 15:47:07.226944 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/query/kernel/adam_v with shape [768, 768]
I1009 15:47:07.233198 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/value/bias with shape [768]
I1009 15:47:07.238353 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/value/bias/adam_m with shape [768]
I1009 15:47:07.243831 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/value/bias/adam_v with shape [768]
I1009 15:47:07.249445 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/value/kernel with shape [768, 768]
I1009 15:47:07.255527 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/value/kernel/adam_m with shape [768, 768]
I1009 15:47:07.261869 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/attention/self/value/kernel/adam_v with shape [768, 768]
I1009 15:47:07.268050 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/intermediate/dense/bias with shape [3072]
I1009 15:47:07.273308 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/intermediate/dense/bias/adam_m with shape [3072]
I1009 15:47:07.278274 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/intermediate/dense/bias/adam_v with shape [3072]
I1009 15:47:07.283823 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/intermediate/dense/kernel with shape [768, 3072]
I1009 15:47:07.293444 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/intermediate/dense/kernel/adam_m with shape [768, 3072]
I1009 15:47:07.302655 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/intermediate/dense/kernel/adam_v with shape [768, 3072]
I1009 15:47:07.312628 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/output/LayerNorm/beta with shape [768]
I1009 15:47:07.318495 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/output/LayerNorm/beta/adam_m with shape [768]
I1009 15:47:07.323935 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/output/LayerNorm/beta/adam_v with shape [768]
I1009 15:47:07.329243 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/output/LayerNorm/gamma with shape [768]
I1009 15:47:07.334521 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/output/LayerNorm/gamma/adam_m with shape [768]
I1009 15:47:07.339932 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/output/LayerNorm/gamma/adam_v with shape [768]
I1009 15:47:07.345828 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/output/dense/bias with shape [768]
I1009 15:47:07.351069 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/output/dense/bias/adam_m with shape [768]
I1009 15:47:07.356699 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/output/dense/bias/adam_v with shape [768]
I1009 15:47:07.362353 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/output/dense/kernel with shape [3072, 768]
I1009 15:47:07.371929 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/output/dense/kernel/adam_m with shape [3072, 768]
I1009 15:47:07.381705 47631408520768 modeling_bert.py:71] Loading TF weight bert/encoder/layer_9/output/dense/kernel/adam_v with shape [3072, 768]
I1009 15:47:07.390963 47631408520768 modeling_bert.py:71] Loading TF weight bert/pooler/dense/bias with shape [768]
I1009 15:47:07.396645 47631408520768 modeling_bert.py:71] Loading TF weight bert/pooler/dense/bias/adam_m with shape [768]
I1009 15:47:07.401854 47631408520768 modeling_bert.py:71] Loading TF weight bert/pooler/dense/bias/adam_v with shape [768]
I1009 15:47:07.406987 47631408520768 modeling_bert.py:71] Loading TF weight bert/pooler/dense/kernel with shape [768, 768]
I1009 15:47:07.412913 47631408520768 modeling_bert.py:71] Loading TF weight bert/pooler/dense/kernel/adam_m with shape [768, 768]
I1009 15:47:07.419250 47631408520768 modeling_bert.py:71] Loading TF weight bert/pooler/dense/kernel/adam_v with shape [768, 768]
I1009 15:47:07.425417 47631408520768 modeling_bert.py:71] Loading TF weight global_step with shape []
I1009 15:47:07.430413 47631408520768 modeling_bert.py:71] Loading TF weight output_bias with shape [21]
I1009 15:47:07.435464 47631408520768 modeling_bert.py:71] Loading TF weight output_bias/adam_m with shape [21]
I1009 15:47:07.440396 47631408520768 modeling_bert.py:71] Loading TF weight output_bias/adam_v with shape [21]
I1009 15:47:07.445353 47631408520768 modeling_bert.py:71] Loading TF weight output_weights with shape [21, 768]
I1009 15:47:07.450436 47631408520768 modeling_bert.py:71] Loading TF weight output_weights/adam_m with shape [21, 768]
I1009 15:47:07.455591 47631408520768 modeling_bert.py:71] Loading TF weight output_weights/adam_v with shape [21, 768]
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-34-7cd6093dcd13> in <module>
     15 
     16 # Load weights from tf checkpoint
---> 17 load_tf_weights_in_bert(model, config, tf_checkpoint_path)
     18 
     19 # Save pytorch-model

~/py3.6/lib/python3.6/site-packages/transformers/modeling_bert.py in load_tf_weights_in_bert(model, config, tf_checkpoint_path)

~/py3.6/lib/python3.6/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    589                 return modules[name]
    590         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 591             type(self).__name__, name))
    592 
    593     def __setattr__(self, name, value):

AttributeError: 'LayerNorm' object has no attribute 'cls'
```

Comments:
Hi @chiyuzhang94 . What do you get if you remove everything related to `cls` in

```
                pointer = getattr(pointer, 'cls')
                pointer = getattr(pointer, 'bias')
            elif l[0] == 'output_weights':
                pointer = getattr(pointer, 'cls')
                pointer = getattr(pointer, 'weight')
```

?

> Hi @chiyuzhang94 . What do you get if you remove everything related to `cls` in
> 
> ```
>                 pointer = getattr(pointer, 'cls')
>                 pointer = getattr(pointer, 'bias')
>             elif l[0] == 'output_weights':
>                 pointer = getattr(pointer, 'cls')
>                 pointer = getattr(pointer, 'weight')
> ```
> 
> ?



Hi @rlouf 
When I removed `cls` parts, I got the following error: 
```
I1009 17:18:05.159223 47655893610048 modeling_bert.py:81] Skipping bert/encoder/layer_9/output/dense/bias/adam_m
I1009 17:18:05.159870 47655893610048 modeling_bert.py:81] Skipping bert/encoder/layer_9/output/dense/bias/adam_v
I1009 17:18:05.160566 47655893610048 modeling_bert.py:115] Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'output', 'dense', 'kernel']
I1009 17:18:05.161365 47655893610048 modeling_bert.py:81] Skipping bert/encoder/layer_9/output/dense/kernel/adam_m
I1009 17:18:05.162042 47655893610048 modeling_bert.py:81] Skipping bert/encoder/layer_9/output/dense/kernel/adam_v
I1009 17:18:05.162718 47655893610048 modeling_bert.py:115] Initialize PyTorch weight ['bert', 'pooler', 'dense', 'bias']
I1009 17:18:05.163373 47655893610048 modeling_bert.py:81] Skipping bert/pooler/dense/bias/adam_m
I1009 17:18:05.164020 47655893610048 modeling_bert.py:81] Skipping bert/pooler/dense/bias/adam_v
I1009 17:18:05.164690 47655893610048 modeling_bert.py:115] Initialize PyTorch weight ['bert', 'pooler', 'dense', 'kernel']
I1009 17:18:05.165374 47655893610048 modeling_bert.py:81] Skipping bert/pooler/dense/kernel/adam_m
I1009 17:18:05.166048 47655893610048 modeling_bert.py:81] Skipping bert/pooler/dense/kernel/adam_v
I1009 17:18:05.166695 47655893610048 modeling_bert.py:81] Skipping global_step
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-1-a2cbceaf2173> in <module>
     15 
     16 # Load weights from tf checkpoint
---> 17 load_tf_weights_in_bert(model, config, tf_checkpoint_path)
     18 
     19 # Save pytorch-model

~/py3.6/lib/python3.6/site-packages/transformers-2.1.0-py3.6.egg/transformers/modeling_bert.py in load_tf_weights_in_bert(model, config, tf_checkpoint_path)
     90                 pointer = getattr(pointer, 'weight')
     91             elif l[0] == 'output_bias' or l[0] == 'beta':
---> 92                 pointer = getattr(pointer, 'bias')
     93             elif l[0] == 'output_weights':
     94                 pointer = getattr(pointer, 'weight')

~/py3.6/lib/python3.6/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    589                 return modules[name]
    590         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 591             type(self).__name__, name))
    592 
    593     def __setattr__(self, name, value):
```

Hi @chiyuzhang94, as a side question which TensorFlow version did you use to train your bert model ?

Do you observe the same behavior by loading the .index file directly using:

```python
config = BertConfig.from_json_file('your/tf_model/config.json')
model = BertForSequenceClassification.from_pretrained('your/tf_model/xxxx.ckpt.index', from_tf=True, config=config)
```

Hi @mfuntowicz ,
I trained the model with tensorflow 1.12.0.

My am currently using tensorflow 1.13.1  and torch 1.2.0 for this converting task. 

If I use your suggestion code, it is also same issue:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-2-35a5c9b35e78> in <module>
     12 
     13 print("Building PyTorch model from configuration: {}".format(str(config)))
---> 14 model = BertForSequenceClassification.from_pretrained(tf_checkpoint_path, from_tf=True, config=config)
     15 # Load weights from tf checkpoint
     16 # load_tf_weights_in_bert(model, config, tf_checkpoint_path)

~/py3.6/lib/python3.6/site-packages/transformers-2.1.0-py3.6.egg/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    352             if resolved_archive_file.endswith('.index'):
    353                 # Load from a TensorFlow 1.X checkpoint - provided by original authors
--> 354                 model = cls.load_tf_weights(model, config, resolved_archive_file[:-6])  # Remove the '.index'
    355             else:
    356                 # Load from our TensorFlow 2.0 checkpoints

~/py3.6/lib/python3.6/site-packages/transformers-2.1.0-py3.6.egg/transformers/modeling_bert.py in load_tf_weights_in_bert(model, config, tf_checkpoint_path)
     90                 pointer = getattr(pointer, 'weight')
     91             elif l[0] == 'output_bias' or l[0] == 'beta':
---> 92                 pointer = getattr(pointer, 'bias')
     93             elif l[0] == 'output_weights':
     94                 pointer = getattr(pointer, 'weight')

~/py3.6/lib/python3.6/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    589                 return modules[name]
    590         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 591             type(self).__name__, name))
    592 
    593     def __setattr__(self, name, value):

AttributeError: 'BertForSequenceClassification' object has no attribute 'bias'
```

This happens because you are trying to load weights the functions wasn't designed for. Unfortunately we cannot support every possible file. You will have to modify `modeling_bert.py` manually to support your file. The part you need to modify is:

```
    for name, array in zip(names, arrays):
        name = name.split('/')
        # adam_v and adam_m are variables used in AdamWeightDecayOptimizer to calculated m and v
        # which are not required for using pretrained model
        if any(n in ["adam_v", "adam_m", "global_step"] for n in name):
            logger.info("Skipping {}".format("/".join(name)))
            continue
        pointer = model
        for m_name in name:
            if re.fullmatch(r'[A-Za-z]+_\d+', m_name):
                l = re.split(r'_(\d+)', m_name)
            else:
                l = [m_name]
            if l[0] == 'kernel' or l[0] == 'gamma':
                pointer = getattr(pointer, 'weight')
            elif l[0] == 'output_bias' or l[0] == 'beta':
                pointer = getattr(pointer, 'bias')
            elif l[0] == 'output_weights':
                pointer = getattr(pointer, 'weight')
            elif l[0] == 'squad':
                pointer = getattr(pointer, 'classifier')
            else:
                try:
                    pointer = getattr(pointer, l[0])
                except AttributeError:
                    logger.info("Skipping {}".format("/".join(name)))
                    continue
            if len(l) >= 2:
                num = int(l[1])
                pointer = pointer[num]
        if m_name[-11:] == '_embeddings':
            pointer = getattr(pointer, 'weight')
        elif m_name == 'kernel':
            array = np.transpose(array)
        try:
            assert pointer.shape == array.shape
        except AssertionError as e:
            e.args += (pointer.shape, array.shape)
            raise
        logger.info("Initialize PyTorch weight {}".format(name))
        pointer.data = torch.from_numpy(array)
```

Hi @rlouf ,
Thanks for your answer.  I think adding `pointer = getattr(pointer, 'cls')` to the two if-so section make sense. But I am wondering how I can deal with the question of 'LayerNorm' object has no attribute 'cls'. Could you please provide me any hint?

Separating out the conditional statement:
`elif l[0] == 'output_bias' or l[0] == 'beta':`
into two, while maintaining the original functionality in the beta conditional should work?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.