Package: bert4tf
This repo contains aTensorFlow 2.0Kerasimplementation ofgoogle-research/bertwith support for loading of the originalpre-trained weights,
and producing activationsnumerically identicalto the one calculated by the original model.ALBERTandadapter-BERTare also supported by setting the corresponding
configuration parameters (shared_layer=True,embedding_sizeforALBERTandadapter_sizeforadapter-BERT). Setting both will result in an adapter-ALBERT
by sharing the BERT parameters across all layers while adapting every layer with layer specific adapter.The implementation is build from scratch using only basic tensorflow operations,
following the code ingoogle-research/bert/modeling.py(but skipping dead code and applying some simplifications). It also utilizeskpe/params-flowto reduce
common Keras boilerplate code (related to passing model and layer configuration arguments).bert4tfshould work with bothTensorFlow 2.0andTensorFlow 1.14or newer.NEWSLICENSEMIT. SeeLicense File.Installbert4tfis on the Python Package Index (PyPI):pip install bert4tfUsageBERT inbert4tfis implemented as a Keras layer. You could instantiate it like this:frombert4tfimportBertModelLayerl_bert=BertModelLayer(**BertModelLayer.Params(vocab_size=16000,# embedding paramsuse_token_type=True,use_position_embeddings=True,token_type_vocab_size=2,num_layers=12,# transformer encoder paramshidden_size=768,hidden_dropout=0.1,intermediate_size=4*768,intermediate_activation="gelu",adapter_size=None,# see arXiv:1902.00751 (adapter-BERT)shared_layer=False,# True for ALBERT (arXiv:1909.11942)embedding_size=None,# None for BERT, wordpiece embedding size for ALBERTname="bert"# any other Keras layer params))or by using thebert_config.jsonfrom apre-trained google model:importbert4tfmodel_dir=".models/uncased_L-12_H-768_A-12"bert_params=bert.params_from_pretrained_ckpt(model_dir)l_bert=bert4tf.BertModelLayer.from_params(bert_params,name="bert")now you can use the BERT layer in your Keras model like this:fromtensorflowimportkerasmax_seq_len=128l_input_ids=keras.layers.Input(shape=(max_seq_len,),dtype='int32')l_token_type_ids=keras.layers.Input(shape=(max_seq_len,),dtype='int32')# using the default token_type/segment id 0output=l_bert(l_input_ids)# output: [batch_size, max_seq_len, hidden_size]model=keras.Model(inputs=l_input_ids,outputs=output)model.build(input_shape=(None,max_seq_len))# provide a custom token_type/segment id as a layer inputoutput=l_bert([l_input_ids,l_token_type_ids])# [batch_size, max_seq_len, hidden_size]model=keras.Model(inputs=[l_input_ids,l_token_type_ids],outputs=output)model.build(input_shape=[(None,max_seq_len),(None,max_seq_len)])if you choose to useadapter-BERTby setting theadapter_sizeparameter,
you would also like to freeze all the original BERT layers by calling:l_bert.apply_adapter_freeze()and once the model has been build or compiled, the original pre-trained weights
can be loaded in the BERT layer:importbert4tfbert_ckpt_file=os.path.join(model_dir,"bert_model.ckpt")bert.load_stock_weights(l_bert,bert_ckpt_file)N.B.seetests/test_bert_activations.pyfor a complete example.FAQHow to use BERT with thegoogle-research/bertpre-trained weights?model_name="uncased_L-12_H-768_A-12"model_dir=bert4tf.fetch_google_bert_model(model_name,".models")model_ckpt=os.path.join(model_dir,"bert_model.ckpt")bert_params=bert4tf.params_from_pretrained_ckpt(model_dir)l_bert=bert4tf.BertModelLayer.from_params(bert_params,name="bert")# use in Keras Model here, and call model.build()bert.load_bert_weights(l_bert,model_ckpt)# should be called after model.build()How to use ALBERT with thegoogle-research/albertpre-trained weights?model_name="albert_base"model_dir=bert4tf.fetch_tfhub_albert_model(model_name,".models")model_params=bert4tf.albert_params(model_name)l_bert=bert.BertModelLayer.from_params(model_params,name="albert")# use in Keras Model here, and call model.build()bert.load_albert_weights(l_bert,albert_dir)# should be called after model.build()How to use ALBERT with thealbert_zhpre-trained weights?seetests/nonci/test_albert.py:model_name="albert_base"model_dir=bert4tf.fetch_brightmart_albert_model(model_name,".models")model_ckpt=os.path.join(model_dir,"albert_model.ckpt")bert_params=bert.params_from_pretrained_ckpt(model_dir)l_bert=bert.BertModelLayer.from_params(bert_params,name="bert")# use in a Keras Model here, and call model.build()bert.load_albert_weights(l_bert,model_ckpt)# should be called after model.build()How to tokenize the input for thegoogle-research/bertmodels?do_lower_case = not (model_name.find(“cased”) == 0 or model_name.find(“multi_cased”) == 0)
bert.bert_tokenization.validate_case_matches_checkpoint(do_lower_case, model_ckpt)
vocab_file = os.path.join(model_dir, “vocab.txt”)
tokenizer = bert4tf.bert_tokenization.FullTokenizer(vocab_file, do_lower_case)
tokens = tokenizer.tokenize(“Hello, BERT-World!”)
token_ids = tokenizer.convert_tokens_to_ids(tokens)How to tokenize the input for thegoogle-research/albertmodels?import sentencepiece as spmspm_model = os.path.join(model_dir, “assets”, “30k-clean.model”)
sp = spm.SentencePieceProcessor()
sp.load(spm_model)
do_lower_case = Trueprocessed_text = bert.albert_tokenization.preprocess_text(“Hello, World!”, lower=do_lower_case)
token_ids = bert4tf.albert_tokenization.encode_ids(sp, processed_text)Referrence
—
1. kpeResourcesBERT- BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingadapter-BERT- adapter-BERT: Parameter-Efficient Transfer Learning for NLPALBERT- ALBERT: A Lite BERT for Self-Supervised Learning of Language Representationsgoogle-research/bert- the originalBERTimplementationgoogle-research/albert- the originalALBERTimplementation by Googlekpe/params-flow- A Keras coding style for reducingKerasboilerplate code in custom layers by utilizingkpe/py-params