Error converting tensorflow checkpoints
Author: charlenesnb
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

I'm trying to convert a BERT Tensorflow checkpoint to hugging face model

## Details
```
!transformers-cli convert \
  --model_type bert \
  --tf_checkpoint C:\Users\sacl\Panasonic-AI\POC\pretraining\content\PatentBERT\model.ckpt-181172 \
  --config C:\Users\sacl\Panasonic-AI\POC\pretraining\content\PatentBERT\bert_config.json \
  --pytorch_dump_output C:\Users\sacl\Panasonic-AI\POC\pretraining\content\PatentBERT\pytorch_model.bin
```

Full traceback: 

> c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\tensorflow\python\framework\dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\tensorflow\python\framework\dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\tensorflow\python\framework\dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\tensorflow\python\framework\dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\tensorflow\python\framework\dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\tensorflow\python\framework\dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
Converting TensorFlow checkpoint from C:\Users\sacl\Panasonic-AI\POC\pretraining\content\PatentBERT\model.ckpt-181172
Loading TF weight bert/embeddings/LayerNorm/beta with shape [768]
Loading TF weight bert/embeddings/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/embeddings/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/embeddings/LayerNorm/gamma with shape [768]
Loading TF weight bert/embeddings/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/embeddings/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/embeddings/position_embeddings with shape [512, 768]
Loading TF weight bert/embeddings/position_embeddings/adam_m with shape [512, 768]
Loading TF weight bert/embeddings/position_embeddings/adam_v with shape [512, 768]
Loading TF weight bert/embeddings/token_type_embeddings with shape [2, 768]
Loading TF weight bert/embeddings/token_type_embeddings/adam_m with shape [2, 768]
Loading TF weight bert/embeddings/token_type_embeddings/adam_v with shape [2, 768]
Loading TF weight bert/embeddings/word_embeddings with shape [30522, 768]
Loading TF weight bert/embeddings/word_embeddings/adam_m with shape [30522, 768]
Loading TF weight bert/embeddings/word_embeddings/adam_v with shape [30522, 768]
Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_0/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_0/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_0/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_0/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/output/dense/kernel/adam_m with shape [768, 768]
  "vocab_size": 30522
}


Loading TF weight bert/encoder/layer_0/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_0/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_0/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_0/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_0/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_0/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_0/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_0/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_0/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_0/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_0/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_0/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_0/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_0/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_0/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_0/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_0/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_0/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_0/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_0/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_0/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_0/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_0/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_0/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_0/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_0/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_0/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_0/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_1/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_1/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_1/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_1/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_1/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_1/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_1/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_1/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_1/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_1/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_1/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_1/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_1/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_1/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_1/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_1/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_1/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_1/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_1/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_1/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_1/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_1/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_1/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_1/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_1/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_1/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_1/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_1/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_1/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_1/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_1/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_10/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_10/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_10/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_10/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_10/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_10/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_10/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_10/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_10/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_10/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_10/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_10/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_10/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_10/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_10/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_10/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_10/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_10/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_10/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_10/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_10/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_10/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_10/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_10/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_10/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_10/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_10/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_10/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_10/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_10/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_10/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_11/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_11/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_11/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_11/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_11/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_11/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_11/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_11/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_11/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_11/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_11/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_11/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_11/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_11/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_11/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_11/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_11/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_11/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_11/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_11/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_11/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_11/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_11/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_11/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_11/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_11/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_11/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_11/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_11/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_11/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_11/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_2/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_2/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_2/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_2/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_2/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_2/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_2/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_2/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_2/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_2/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_2/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_2/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_2/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_2/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_2/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_2/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_2/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_2/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_2/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_2/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_2/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_2/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_3/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_3/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_3/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_3/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_3/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_3/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_3/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_3/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_3/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_3/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_3/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_3/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_3/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_3/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_3/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_3/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_3/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_3/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_3/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_3/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_3/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_3/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_3/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_3/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_3/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_3/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_3/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_3/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_3/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_3/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_3/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_4/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_4/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_4/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_4/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_4/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_4/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_4/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_4/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_4/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_4/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_4/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_4/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_4/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_4/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_4/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_4/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_4/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_4/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_4/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_4/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_4/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_4/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_4/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_4/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_4/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_4/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_4/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_4/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_4/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_4/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_4/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_5/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_5/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_5/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_5/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_5/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_5/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_5/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_5/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_5/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_5/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_5/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_5/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_5/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_5/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_5/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_5/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_5/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_5/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_5/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_5/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_5/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_5/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_5/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_5/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_5/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_5/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_5/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_5/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_5/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_5/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_5/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_6/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_6/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_6/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_6/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_6/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_6/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_6/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_6/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_6/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_6/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_6/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_6/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_6/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_6/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_6/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_6/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_6/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_6/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_6/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_6/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_6/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_6/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_6/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_6/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_6/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_6/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_6/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_6/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_6/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_6/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_6/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_7/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_7/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_7/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_7/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_7/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_7/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_7/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_7/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_7/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_7/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_7/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_7/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_7/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_7/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_7/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_7/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_7/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_7/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_7/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_7/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_7/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_7/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_7/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_7/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_7/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_7/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_7/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_7/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_7/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_7/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_7/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_8/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_8/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_8/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_8/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_8/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_8/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_8/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_8/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_8/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_8/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_8/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_8/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_8/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_8/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_8/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_8/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_8/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_8/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_8/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_8/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_8/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_8/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_8/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_8/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_8/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_8/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_8/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_8/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_8/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_8/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_8/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_9/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_9/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_9/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_9/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_9/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_9/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_9/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_9/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_9/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_9/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_9/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_9/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_9/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_9/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_9/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_9/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_9/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_9/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_9/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_9/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_9/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_9/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_9/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_9/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_9/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_9/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_9/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_9/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_9/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_9/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_9/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/pooler/dense/bias with shape [768]
Loading TF weight bert/pooler/dense/bias/adam_m with shape [768]
Loading TF weight bert/pooler/dense/bias/adam_v with shape [768]
Loading TF weight bert/pooler/dense/kernel with shape [768, 768]
Loading TF weight bert/pooler/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/pooler/dense/kernel/adam_v with shape [768, 768]
Loading TF weight global_step with shape []
Loading TF weight output_bias with shape [656]
Loading TF weight output_bias/adam_m with shape [656]
Loading TF weight output_bias/adam_v with shape [656]
Loading TF weight output_weights with shape [656, 768]
Loading TF weight output_weights/adam_m with shape [656, 768]
Loading TF weight output_weights/adam_v with shape [656, 768]
Initialize PyTorch weight ['bert', 'embeddings', 'LayerNorm', 'beta']
Skipping bert/embeddings/LayerNorm/beta/adam_m
Skipping bert/embeddings/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'embeddings', 'LayerNorm', 'gamma']
Skipping bert/embeddings/LayerNorm/gamma/adam_m
Skipping bert/embeddings/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'embeddings', 'position_embeddings']
Skipping bert/embeddings/position_embeddings/adam_m
Skipping bert/embeddings/position_embeddings/adam_v
Initialize PyTorch weight ['bert', 'embeddings', 'token_type_embeddings']
Skipping bert/embeddings/token_type_embeddings/adam_m
Skipping bert/embeddings/token_type_embeddings/adam_v
Initialize PyTorch weight ['bert', 'embeddings', 'word_embeddings']
Skipping bert/embeddings/word_embeddings/adam_m
Skipping bert/embeddings/word_embeddings/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_0/attention/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_0/attention/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_0/attention/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_0/attention/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_0/attention/output/dense/bias/adam_m
Skipping bert/encoder/layer_0/attention/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_0/attention/output/dense/kernel/adam_m
Skipping bert/encoder/layer_0/attention/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'self', 'key', 'bias']
Skipping bert/encoder/layer_0/attention/self/key/bias/adam_m
Skipping bert/encoder/layer_0/attention/self/key/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'self', 'key', 'kernel']
Skipping bert/encoder/layer_0/attention/self/key/kernel/adam_m
Skipping bert/encoder/layer_0/attention/self/key/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'self', 'query', 'bias']
Skipping bert/encoder/layer_0/attention/self/query/bias/adam_m
Skipping bert/encoder/layer_0/attention/self/query/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'self', 'query', 'kernel']
Skipping bert/encoder/layer_0/attention/self/query/kernel/adam_m
Skipping bert/encoder/layer_0/attention/self/query/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'self', 'value', 'bias']
Skipping bert/encoder/layer_0/attention/self/value/bias/adam_m
Skipping bert/encoder/layer_0/attention/self/value/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'attention', 'self', 'value', 'kernel']
Skipping bert/encoder/layer_0/attention/self/value/kernel/adam_m
Skipping bert/encoder/layer_0/attention/self/value/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'intermediate', 'dense', 'bias']
Skipping bert/encoder/layer_0/intermediate/dense/bias/adam_m
Skipping bert/encoder/layer_0/intermediate/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'intermediate', 'dense', 'kernel']
Skipping bert/encoder/layer_0/intermediate/dense/kernel/adam_m
Skipping bert/encoder/layer_0/intermediate/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_0/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_0/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_0/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_0/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_0/output/dense/bias/adam_m
Skipping bert/encoder/layer_0/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_0', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_0/output/dense/kernel/adam_m
Skipping bert/encoder/layer_0/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_1/attention/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_1/attention/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_1/attention/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_1/attention/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_1/attention/output/dense/bias/adam_m
Skipping bert/encoder/layer_1/attention/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_1/attention/output/dense/kernel/adam_m
Skipping bert/encoder/layer_1/attention/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'self', 'key', 'bias']
Skipping bert/encoder/layer_1/attention/self/key/bias/adam_m
Skipping bert/encoder/layer_1/attention/self/key/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'self', 'key', 'kernel']
Skipping bert/encoder/layer_1/attention/self/key/kernel/adam_m
Skipping bert/encoder/layer_1/attention/self/key/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'self', 'query', 'bias']
Skipping bert/encoder/layer_1/attention/self/query/bias/adam_m
Skipping bert/encoder/layer_1/attention/self/query/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'self', 'query', 'kernel']
Skipping bert/encoder/layer_1/attention/self/query/kernel/adam_m
Skipping bert/encoder/layer_1/attention/self/query/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'self', 'value', 'bias']
Skipping bert/encoder/layer_1/attention/self/value/bias/adam_m
Skipping bert/encoder/layer_1/attention/self/value/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'attention', 'self', 'value', 'kernel']
Skipping bert/encoder/layer_1/attention/self/value/kernel/adam_m
Skipping bert/encoder/layer_1/attention/self/value/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'intermediate', 'dense', 'bias']
Skipping bert/encoder/layer_1/intermediate/dense/bias/adam_m
Skipping bert/encoder/layer_1/intermediate/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'intermediate', 'dense', 'kernel']
Skipping bert/encoder/layer_1/intermediate/dense/kernel/adam_m
Skipping bert/encoder/layer_1/intermediate/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_1/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_1/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_1/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_1/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_1/output/dense/bias/adam_m
Skipping bert/encoder/layer_1/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_1', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_1/output/dense/kernel/adam_m
Skipping bert/encoder/layer_1/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_10/attention/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_10/attention/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_10/attention/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_10/attention/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_10/attention/output/dense/bias/adam_m
Skipping bert/encoder/layer_10/attention/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_10/attention/output/dense/kernel/adam_m
Skipping bert/encoder/layer_10/attention/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'self', 'key', 'bias']
Skipping bert/encoder/layer_10/attention/self/key/bias/adam_m
Skipping bert/encoder/layer_10/attention/self/key/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'self', 'key', 'kernel']
Skipping bert/encoder/layer_10/attention/self/key/kernel/adam_m
Skipping bert/encoder/layer_10/attention/self/key/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'self', 'query', 'bias']
Skipping bert/encoder/layer_10/attention/self/query/bias/adam_m
Skipping bert/encoder/layer_10/attention/self/query/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'self', 'query', 'kernel']
Skipping bert/encoder/layer_10/attention/self/query/kernel/adam_m
Skipping bert/encoder/layer_10/attention/self/query/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'self', 'value', 'bias']
Skipping bert/encoder/layer_10/attention/self/value/bias/adam_m
Skipping bert/encoder/layer_10/attention/self/value/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'attention', 'self', 'value', 'kernel']
Skipping bert/encoder/layer_10/attention/self/value/kernel/adam_m
Skipping bert/encoder/layer_10/attention/self/value/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'intermediate', 'dense', 'bias']
Skipping bert/encoder/layer_10/intermediate/dense/bias/adam_m
Skipping bert/encoder/layer_10/intermediate/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'intermediate', 'dense', 'kernel']
Skipping bert/encoder/layer_10/intermediate/dense/kernel/adam_m
Skipping bert/encoder/layer_10/intermediate/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_10/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_10/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_10/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_10/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_10/output/dense/bias/adam_m
Skipping bert/encoder/layer_10/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_10', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_10/output/dense/kernel/adam_m
Skipping bert/encoder/layer_10/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_11/attention/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_11/attention/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_11/attention/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_11/attention/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_11/attention/output/dense/bias/adam_m
Skipping bert/encoder/layer_11/attention/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_11/attention/output/dense/kernel/adam_m
Skipping bert/encoder/layer_11/attention/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'self', 'key', 'bias']
Skipping bert/encoder/layer_11/attention/self/key/bias/adam_m
Skipping bert/encoder/layer_11/attention/self/key/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'self', 'key', 'kernel']
Skipping bert/encoder/layer_11/attention/self/key/kernel/adam_m
Skipping bert/encoder/layer_11/attention/self/key/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'self', 'query', 'bias']
Skipping bert/encoder/layer_11/attention/self/query/bias/adam_m
Skipping bert/encoder/layer_11/attention/self/query/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'self', 'query', 'kernel']
Skipping bert/encoder/layer_11/attention/self/query/kernel/adam_m
Skipping bert/encoder/layer_11/attention/self/query/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'self', 'value', 'bias']
Skipping bert/encoder/layer_11/attention/self/value/bias/adam_m
Skipping bert/encoder/layer_11/attention/self/value/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'attention', 'self', 'value', 'kernel']
Skipping bert/encoder/layer_11/attention/self/value/kernel/adam_m
Skipping bert/encoder/layer_11/attention/self/value/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'intermediate', 'dense', 'bias']
Skipping bert/encoder/layer_11/intermediate/dense/bias/adam_m
Skipping bert/encoder/layer_11/intermediate/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'intermediate', 'dense', 'kernel']
Skipping bert/encoder/layer_11/intermediate/dense/kernel/adam_m
Skipping bert/encoder/layer_11/intermediate/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_11/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_11/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_11/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_11/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_11/output/dense/bias/adam_m
Skipping bert/encoder/layer_11/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_11', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_11/output/dense/kernel/adam_m
Skipping bert/encoder/layer_11/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_2/attention/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_2/attention/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_2/attention/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_2/attention/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_2/attention/output/dense/bias/adam_m
Skipping bert/encoder/layer_2/attention/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_2/attention/output/dense/kernel/adam_m
Skipping bert/encoder/layer_2/attention/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'self', 'key', 'bias']
Skipping bert/encoder/layer_2/attention/self/key/bias/adam_m
Skipping bert/encoder/layer_2/attention/self/key/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'self', 'key', 'kernel']
Skipping bert/encoder/layer_2/attention/self/key/kernel/adam_m
Skipping bert/encoder/layer_2/attention/self/key/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'self', 'query', 'bias']
Skipping bert/encoder/layer_2/attention/self/query/bias/adam_m
Skipping bert/encoder/layer_2/attention/self/query/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'self', 'query', 'kernel']
Skipping bert/encoder/layer_2/attention/self/query/kernel/adam_m
Skipping bert/encoder/layer_2/attention/self/query/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'self', 'value', 'bias']
Skipping bert/encoder/layer_2/attention/self/value/bias/adam_m
Skipping bert/encoder/layer_2/attention/self/value/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'attention', 'self', 'value', 'kernel']
Skipping bert/encoder/layer_2/attention/self/value/kernel/adam_m
Skipping bert/encoder/layer_2/attention/self/value/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'intermediate', 'dense', 'bias']
Skipping bert/encoder/layer_2/intermediate/dense/bias/adam_m
Skipping bert/encoder/layer_2/intermediate/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'intermediate', 'dense', 'kernel']
Skipping bert/encoder/layer_2/intermediate/dense/kernel/adam_m
Skipping bert/encoder/layer_2/intermediate/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_2/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_2/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_2/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_2/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_2/output/dense/bias/adam_m
Skipping bert/encoder/layer_2/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_2', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_2/output/dense/kernel/adam_m
Skipping bert/encoder/layer_2/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_3/attention/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_3/attention/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_3/attention/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_3/attention/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_3/attention/output/dense/bias/adam_m
Skipping bert/encoder/layer_3/attention/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_3/attention/output/dense/kernel/adam_m
Skipping bert/encoder/layer_3/attention/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'self', 'key', 'bias']
Skipping bert/encoder/layer_3/attention/self/key/bias/adam_m
Skipping bert/encoder/layer_3/attention/self/key/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'self', 'key', 'kernel']
Skipping bert/encoder/layer_3/attention/self/key/kernel/adam_m
Skipping bert/encoder/layer_3/attention/self/key/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'self', 'query', 'bias']
Skipping bert/encoder/layer_3/attention/self/query/bias/adam_m
Skipping bert/encoder/layer_3/attention/self/query/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'self', 'query', 'kernel']
Skipping bert/encoder/layer_3/attention/self/query/kernel/adam_m
Skipping bert/encoder/layer_3/attention/self/query/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'self', 'value', 'bias']
Skipping bert/encoder/layer_3/attention/self/value/bias/adam_m
Skipping bert/encoder/layer_3/attention/self/value/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'attention', 'self', 'value', 'kernel']
Skipping bert/encoder/layer_3/attention/self/value/kernel/adam_m
Skipping bert/encoder/layer_3/attention/self/value/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'intermediate', 'dense', 'bias']
Skipping bert/encoder/layer_3/intermediate/dense/bias/adam_m
Skipping bert/encoder/layer_3/intermediate/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'intermediate', 'dense', 'kernel']
Skipping bert/encoder/layer_3/intermediate/dense/kernel/adam_m
Skipping bert/encoder/layer_3/intermediate/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_3/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_3/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_3/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_3/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_3/output/dense/bias/adam_m
Skipping bert/encoder/layer_3/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_3', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_3/output/dense/kernel/adam_m
Skipping bert/encoder/layer_3/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_4/attention/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_4/attention/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_4/attention/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_4/attention/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_4/attention/output/dense/bias/adam_m
Skipping bert/encoder/layer_4/attention/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_4/attention/output/dense/kernel/adam_m
Skipping bert/encoder/layer_4/attention/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'self', 'key', 'bias']
Skipping bert/encoder/layer_4/attention/self/key/bias/adam_m
Skipping bert/encoder/layer_4/attention/self/key/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'self', 'key', 'kernel']
Skipping bert/encoder/layer_4/attention/self/key/kernel/adam_m
Skipping bert/encoder/layer_4/attention/self/key/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'self', 'query', 'bias']
Skipping bert/encoder/layer_4/attention/self/query/bias/adam_m
Skipping bert/encoder/layer_4/attention/self/query/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'self', 'query', 'kernel']
Skipping bert/encoder/layer_4/attention/self/query/kernel/adam_m
Skipping bert/encoder/layer_4/attention/self/query/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'self', 'value', 'bias']
Skipping bert/encoder/layer_4/attention/self/value/bias/adam_m
Skipping bert/encoder/layer_4/attention/self/value/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'attention', 'self', 'value', 'kernel']
Skipping bert/encoder/layer_4/attention/self/value/kernel/adam_m
Skipping bert/encoder/layer_4/attention/self/value/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'intermediate', 'dense', 'bias']
Skipping bert/encoder/layer_4/intermediate/dense/bias/adam_m
Skipping bert/encoder/layer_4/intermediate/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'intermediate', 'dense', 'kernel']
Skipping bert/encoder/layer_4/intermediate/dense/kernel/adam_m
Skipping bert/encoder/layer_4/intermediate/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_4/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_4/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_4/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_4/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_4/output/dense/bias/adam_m
Skipping bert/encoder/layer_4/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_4', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_4/output/dense/kernel/adam_m
Skipping bert/encoder/layer_4/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_5/attention/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_5/attention/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_5/attention/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_5/attention/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_5/attention/output/dense/bias/adam_m
Skipping bert/encoder/layer_5/attention/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_5/attention/output/dense/kernel/adam_m
Skipping bert/encoder/layer_5/attention/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'self', 'key', 'bias']
Skipping bert/encoder/layer_5/attention/self/key/bias/adam_m
Skipping bert/encoder/layer_5/attention/self/key/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'self', 'key', 'kernel']
Skipping bert/encoder/layer_5/attention/self/key/kernel/adam_m
Skipping bert/encoder/layer_5/attention/self/key/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'self', 'query', 'bias']
Skipping bert/encoder/layer_5/attention/self/query/bias/adam_m
Skipping bert/encoder/layer_5/attention/self/query/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'self', 'query', 'kernel']
Skipping bert/encoder/layer_5/attention/self/query/kernel/adam_m
Skipping bert/encoder/layer_5/attention/self/query/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'self', 'value', 'bias']
Skipping bert/encoder/layer_5/attention/self/value/bias/adam_m
Skipping bert/encoder/layer_5/attention/self/value/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'attention', 'self', 'value', 'kernel']
Skipping bert/encoder/layer_5/attention/self/value/kernel/adam_m
Skipping bert/encoder/layer_5/attention/self/value/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'intermediate', 'dense', 'bias']
Skipping bert/encoder/layer_5/intermediate/dense/bias/adam_m
Skipping bert/encoder/layer_5/intermediate/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'intermediate', 'dense', 'kernel']
Skipping bert/encoder/layer_5/intermediate/dense/kernel/adam_m
Skipping bert/encoder/layer_5/intermediate/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_5/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_5/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_5/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_5/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_5/output/dense/bias/adam_m
Skipping bert/encoder/layer_5/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_5', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_5/output/dense/kernel/adam_m
Skipping bert/encoder/layer_5/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_6/attention/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_6/attention/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_6/attention/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_6/attention/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_6/attention/output/dense/bias/adam_m
Skipping bert/encoder/layer_6/attention/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_6/attention/output/dense/kernel/adam_m
Skipping bert/encoder/layer_6/attention/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'self', 'key', 'bias']
Skipping bert/encoder/layer_6/attention/self/key/bias/adam_m
Skipping bert/encoder/layer_6/attention/self/key/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'self', 'key', 'kernel']
Skipping bert/encoder/layer_6/attention/self/key/kernel/adam_m
Skipping bert/encoder/layer_6/attention/self/key/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'self', 'query', 'bias']
Skipping bert/encoder/layer_6/attention/self/query/bias/adam_m
Skipping bert/encoder/layer_6/attention/self/query/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'self', 'query', 'kernel']
Skipping bert/encoder/layer_6/attention/self/query/kernel/adam_m
Skipping bert/encoder/layer_6/attention/self/query/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'self', 'value', 'bias']
Skipping bert/encoder/layer_6/attention/self/value/bias/adam_m
Skipping bert/encoder/layer_6/attention/self/value/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'attention', 'self', 'value', 'kernel']
Skipping bert/encoder/layer_6/attention/self/value/kernel/adam_m
Skipping bert/encoder/layer_6/attention/self/value/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'intermediate', 'dense', 'bias']
Skipping bert/encoder/layer_6/intermediate/dense/bias/adam_m
Skipping bert/encoder/layer_6/intermediate/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'intermediate', 'dense', 'kernel']
Skipping bert/encoder/layer_6/intermediate/dense/kernel/adam_m
Skipping bert/encoder/layer_6/intermediate/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_6/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_6/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_6/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_6/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_6/output/dense/bias/adam_m
Skipping bert/encoder/layer_6/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_6', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_6/output/dense/kernel/adam_m
Skipping bert/encoder/layer_6/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_7/attention/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_7/attention/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_7/attention/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_7/attention/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_7/attention/output/dense/bias/adam_m
Skipping bert/encoder/layer_7/attention/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_7/attention/output/dense/kernel/adam_m
Skipping bert/encoder/layer_7/attention/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'self', 'key', 'bias']
Skipping bert/encoder/layer_7/attention/self/key/bias/adam_m
Skipping bert/encoder/layer_7/attention/self/key/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'self', 'key', 'kernel']
Skipping bert/encoder/layer_7/attention/self/key/kernel/adam_m
Skipping bert/encoder/layer_7/attention/self/key/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'self', 'query', 'bias']
Skipping bert/encoder/layer_7/attention/self/query/bias/adam_m
Skipping bert/encoder/layer_7/attention/self/query/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'self', 'query', 'kernel']
Skipping bert/encoder/layer_7/attention/self/query/kernel/adam_m
Skipping bert/encoder/layer_7/attention/self/query/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'self', 'value', 'bias']
Skipping bert/encoder/layer_7/attention/self/value/bias/adam_m
Skipping bert/encoder/layer_7/attention/self/value/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'attention', 'self', 'value', 'kernel']
Skipping bert/encoder/layer_7/attention/self/value/kernel/adam_m
Skipping bert/encoder/layer_7/attention/self/value/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'intermediate', 'dense', 'bias']
Skipping bert/encoder/layer_7/intermediate/dense/bias/adam_m
Skipping bert/encoder/layer_7/intermediate/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'intermediate', 'dense', 'kernel']
Skipping bert/encoder/layer_7/intermediate/dense/kernel/adam_m
Skipping bert/encoder/layer_7/intermediate/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_7/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_7/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_7/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_7/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_7/output/dense/bias/adam_m
Skipping bert/encoder/layer_7/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_7', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_7/output/dense/kernel/adam_m
Skipping bert/encoder/layer_7/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_8/attention/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_8/attention/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_8/attention/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_8/attention/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_8/attention/output/dense/bias/adam_m
Skipping bert/encoder/layer_8/attention/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_8/attention/output/dense/kernel/adam_m
Skipping bert/encoder/layer_8/attention/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'self', 'key', 'bias']
Skipping bert/encoder/layer_8/attention/self/key/bias/adam_m
Skipping bert/encoder/layer_8/attention/self/key/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'self', 'key', 'kernel']
Skipping bert/encoder/layer_8/attention/self/key/kernel/adam_m
Skipping bert/encoder/layer_8/attention/self/key/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'self', 'query', 'bias']
Skipping bert/encoder/layer_8/attention/self/query/bias/adam_m
Skipping bert/encoder/layer_8/attention/self/query/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'self', 'query', 'kernel']
Skipping bert/encoder/layer_8/attention/self/query/kernel/adam_m
Skipping bert/encoder/layer_8/attention/self/query/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'self', 'value', 'bias']
Skipping bert/encoder/layer_8/attention/self/value/bias/adam_m
Skipping bert/encoder/layer_8/attention/self/value/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'attention', 'self', 'value', 'kernel']
Skipping bert/encoder/layer_8/attention/self/value/kernel/adam_m
Skipping bert/encoder/layer_8/attention/self/value/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'intermediate', 'dense', 'bias']
Skipping bert/encoder/layer_8/intermediate/dense/bias/adam_m
Skipping bert/encoder/layer_8/intermediate/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'intermediate', 'dense', 'kernel']
Skipping bert/encoder/layer_8/intermediate/dense/kernel/adam_m
Skipping bert/encoder/layer_8/intermediate/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_8/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_8/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_8/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_8/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_8/output/dense/bias/adam_m
Skipping bert/encoder/layer_8/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_8', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_8/output/dense/kernel/adam_m
Skipping bert/encoder/layer_8/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_9/attention/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_9/attention/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_9/attention/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_9/attention/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_9/attention/output/dense/bias/adam_m
Skipping bert/encoder/layer_9/attention/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_9/attention/output/dense/kernel/adam_m
Skipping bert/encoder/layer_9/attention/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'self', 'key', 'bias']
Skipping bert/encoder/layer_9/attention/self/key/bias/adam_m
Skipping bert/encoder/layer_9/attention/self/key/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'self', 'key', 'kernel']
Skipping bert/encoder/layer_9/attention/self/key/kernel/adam_m
Skipping bert/encoder/layer_9/attention/self/key/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'self', 'query', 'bias']
Skipping bert/encoder/layer_9/attention/self/query/bias/adam_m
Skipping bert/encoder/layer_9/attention/self/query/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'self', 'query', 'kernel']
Skipping bert/encoder/layer_9/attention/self/query/kernel/adam_m
Skipping bert/encoder/layer_9/attention/self/query/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'self', 'value', 'bias']
Skipping bert/encoder/layer_9/attention/self/value/bias/adam_m
Skipping bert/encoder/layer_9/attention/self/value/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'attention', 'self', 'value', 'kernel']
Skipping bert/encoder/layer_9/attention/self/value/kernel/adam_m
Skipping bert/encoder/layer_9/attention/self/value/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'intermediate', 'dense', 'bias']
Skipping bert/encoder/layer_9/intermediate/dense/bias/adam_m
Skipping bert/encoder/layer_9/intermediate/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'intermediate', 'dense', 'kernel']
Skipping bert/encoder/layer_9/intermediate/dense/kernel/adam_m
Skipping bert/encoder/layer_9/intermediate/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'output', 'LayerNorm', 'beta']
Skipping bert/encoder/layer_9/output/LayerNorm/beta/adam_m
Skipping bert/encoder/layer_9/output/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'output', 'LayerNorm', 'gamma']
Skipping bert/encoder/layer_9/output/LayerNorm/gamma/adam_m
Skipping bert/encoder/layer_9/output/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'output', 'dense', 'bias']
Skipping bert/encoder/layer_9/output/dense/bias/adam_m
Skipping bert/encoder/layer_9/output/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'output', 'dense', 'kernel']
Skipping bert/encoder/layer_9/output/dense/kernel/adam_m
Skipping bert/encoder/layer_9/output/dense/kernel/adam_v
Initialize PyTorch weight ['bert', 'pooler', 'dense', 'bias']
Skipping bert/pooler/dense/bias/adam_m
Skipping bert/pooler/dense/bias/adam_v
Initialize PyTorch weight ['bert', 'pooler', 'dense', 'kernel']
Skipping bert/pooler/dense/kernel/adam_m
Skipping bert/pooler/dense/kernel/adam_v
Skipping global_step
Traceback (most recent call last):
  File "c:\programdata\anaconda3\envs\py37cuda10\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\programdata\anaconda3\envs\py37cuda10\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\ProgramData\Anaconda3\envs\py37cuda10\Scripts\transformers-cli.exe\__main__.py", line 7, in <module>
  File "c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\transformers\commands\transformers_cli.py", line 33, in main
    service.run()
  File "c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\transformers\commands\convert.py", line 91, in run
    convert_tf_checkpoint_to_pytorch(self._tf_checkpoint, self._config, self._pytorch_dump_output)
  File "c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\transformers\convert_bert_original_tf_checkpoint_to_pytorch.py", line 36, in convert_tf_checkpoint_to_pytorch
    load_tf_weights_in_bert(model, config, tf_checkpoint_path)
  File "c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\transformers\modeling_bert.py", line 135, in load_tf_weights_in_bert
    pointer = getattr(pointer, "bias")
  File "c:\programdata\anaconda3\envs\py37cuda10\lib\site-packages\torch\nn\modules\module.py", line 779, in __getattr__
    type(self).__name__, name))
torch.nn.modules.module.ModuleAttributeError: 'BertForPreTraining' object has no attribute 'bias'

Comments:
I ran into the same problem

> I ran into the same problem

but I didn't get the error

Hi @nikhilbyte @chainesanbuenaventura 

Any updates? I also have the same problem while converting TensorFlow model to PyTorch model?

Thanks


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.