Package: bert-for-tf2-mod
This repo contains aTensorFlow 2.0Kerasimplementation ofgoogle-research/bertwith support for loading of the originalpre-trained weights,
and producing activationsnumerically identicalto the one calculated by the original model.ALBERTandadapter-BERTare also supported by setting the corresponding
configuration parameters (shared_layer=True,embedding_sizeforALBERTandadapter_sizeforadapter-BERT). Setting both will result in an adapter-ALBERT
by sharing the BERT parameters across all layers while adapting every layer with layer specific adapter.The implementation is build from scratch using only basic tensorflow operations,
following the code ingoogle-research/bert/modeling.py(but skipping dead code and applying some simplifications). It also utilizeskpe/params-flowto reduce
common Keras boilerplate code (related to passing model and layer configuration arguments).bert-for-tf2should work with bothTensorFlow 2.0andTensorFlow 1.14or newer.NEWS30.Jul.2020-VERBOSE=0env variable for suppressing stdout output.06.Apr.2020- using latestpy-paramsintroducingWithParamsbase forLayerandModel. See news inkpe/py-paramsfor how to update (_construct()signature has change and
requires callingsuper().__construct()).06.Jan.2020- support for loading the tar format weights fromgoogle-research/ALBERT.18.Nov.2019- ALBERT tokenization added (make sure to import asfrom bert import albert_tokenizationorfrom bert import bert_tokenization).08.Nov.2019- using v2 per default when loading theTFHub/albertweights ofgoogle-research/ALBERT.05.Nov.2019- minor ALBERT word embeddings refactoring (word_embeddings_2->word_embeddings_projector) and related parameter freezing fixes.04.Nov.2019- support for extra (task specific) token embeddings using negative token ids.29.Oct.2019- support for loading of the pre-trained ALBERT weights released bygoogle-research/ALBERTatTFHub/albert.11.Oct.2019- support for loading of the pre-trained ALBERT weights released bybrightmart/albert_zh ALBERT for Chinese.10.Oct.2019- support forALBERTthrough theshared_layer=Trueandembedding_size=128params.03.Sep.2019- walkthrough on fine tuning with adapter-BERT and storing the
fine tuned fraction of the weights in a separate checkpoint (seetests/test_adapter_finetune.py).02.Sep.2019- support for extending the token type embeddings of a pre-trained model
by returning the mismatched weights inload_stock_weights()(seetests/test_extend_segments.py).25.Jul.2019- there are now two colab notebooks underexamples/showing how to
fine-tune an IMDB Movie Reviews sentiment classifier from pre-trained BERT weights
using anadapter-BERTmodel architecture on a GPU or TPU in Google Colab.28.Jun.2019- v.0.3.0 supportsadapter-BERT(google-research/adapter-bert)
for “Parameter-Efficient Transfer Learning for NLP”, i.e. fine-tuning small overlay adapter
layers over BERT’s transformer encoders without changing the frozen BERT weights.LICENSEMIT. SeeLicense File.Installbert-for-tf2is on the Python Package Index (PyPI):pip install bert-for-tf2UsageBERT inbert-for-tf2is implemented as a Keras layer. You could instantiate it like this:frombertimportBertModelLayerl_bert=BertModelLayer(**BertModelLayer.Params(vocab_size=16000,# embedding paramsuse_token_type=True,use_position_embeddings=True,token_type_vocab_size=2,num_layers=12,# transformer encoder paramshidden_size=768,hidden_dropout=0.1,intermediate_size=4*768,intermediate_activation="gelu",adapter_size=None,# see arXiv:1902.00751 (adapter-BERT)shared_layer=False,# True for ALBERT (arXiv:1909.11942)embedding_size=None,# None for BERT, wordpiece embedding size for ALBERTname="bert"# any other Keras layer params))or by using thebert_config.jsonfrom apre-trained google model:importbertmodel_dir=".models/uncased_L-12_H-768_A-12"bert_params=bert.params_from_pretrained_ckpt(model_dir)l_bert=bert.BertModelLayer.from_params(bert_params,name="bert")now you can use the BERT layer in your Keras model like this:fromtensorflowimportkerasmax_seq_len=128l_input_ids=keras.layers.Input(shape=(max_seq_len,),dtype='int32')l_token_type_ids=keras.layers.Input(shape=(max_seq_len,),dtype='int32')# using the default token_type/segment id 0output=l_bert(l_input_ids)# output: [batch_size, max_seq_len, hidden_size]model=keras.Model(inputs=l_input_ids,outputs=output)model.build(input_shape=(None,max_seq_len))# provide a custom token_type/segment id as a layer inputoutput=l_bert([l_input_ids,l_token_type_ids])# [batch_size, max_seq_len, hidden_size]model=keras.Model(inputs=[l_input_ids,l_token_type_ids],outputs=output)model.build(input_shape=[(None,max_seq_len),(None,max_seq_len)])if you choose to useadapter-BERTby setting theadapter_sizeparameter,
you would also like to freeze all the original BERT layers by calling:l_bert.apply_adapter_freeze()and once the model has been build or compiled, the original pre-trained weights
can be loaded in the BERT layer:importbertbert_ckpt_file=os.path.join(model_dir,"bert_model.ckpt")bert.load_stock_weights(l_bert,bert_ckpt_file)N.B.seetests/test_bert_activations.pyfor a complete example.FAQIn all the examlpes bellow,please notethe line:# use in a Keras Model here, and call model.build()for a quick test, you can replace it with something like:model=keras.models.Sequential([keras.layers.InputLayer(input_shape=(128,)),l_bert,keras.layers.Lambda(lambdax:x[:,0,:]),keras.layers.Dense(2)])model.build(input_shape=(None,128))How to use BERT with thegoogle-research/bertpre-trained weights?model_name="uncased_L-12_H-768_A-12"model_dir=bert.fetch_google_bert_model(model_name,".models")model_ckpt=os.path.join(model_dir,"bert_model.ckpt")bert_params=bert.params_from_pretrained_ckpt(model_dir)l_bert=bert.BertModelLayer.from_params(bert_params,name="bert")# use in a Keras Model here, and call model.build()bert.load_bert_weights(l_bert,model_ckpt)# should be called after model.build()How to use ALBERT with thegoogle-research/ALBERTpre-trained weights (fetching from TFHub)?seetests/nonci/test_load_pretrained_weights.py:model_name="albert_base"model_dir=bert.fetch_tfhub_albert_model(model_name,".models")model_params=bert.albert_params(model_name)l_bert=bert.BertModelLayer.from_params(model_params,name="albert")# use in a Keras Model here, and call model.build()bert.load_albert_weights(l_bert,albert_dir)# should be called after model.build()How to use ALBERT with thegoogle-research/ALBERTpre-trained weights (non TFHub)?seetests/nonci/test_load_pretrained_weights.py:model_name="albert_base_v2"model_dir=bert.fetch_google_albert_model(model_name,".models")model_ckpt=os.path.join(albert_dir,"model.ckpt-best")model_params=bert.albert_params(model_dir)l_bert=bert.BertModelLayer.from_params(model_params,name="albert")# use in a Keras Model here, and call model.build()bert.load_albert_weights(l_bert,model_ckpt)# should be called after model.build()How to use ALBERT with thebrightmart/albert_zhpre-trained weights?seetests/nonci/test_albert.py:model_name="albert_base"model_dir=bert.fetch_brightmart_albert_model(model_name,".models")model_ckpt=os.path.join(model_dir,"albert_model.ckpt")bert_params=bert.params_from_pretrained_ckpt(model_dir)l_bert=bert.BertModelLayer.from_params(bert_params,name="bert")# use in a Keras Model here, and call model.build()bert.load_albert_weights(l_bert,model_ckpt)# should be called after model.build()How to tokenize the input for thegoogle-research/bertmodels?do_lower_case=not(model_name.find("cased")==0ormodel_name.find("multi_cased")==0)bert.bert_tokenization.validate_case_matches_checkpoint(do_lower_case,model_ckpt)vocab_file=os.path.join(model_dir,"vocab.txt")tokenizer=bert.bert_tokenization.FullTokenizer(vocab_file,do_lower_case)tokens=tokenizer.tokenize("Hello, BERT-World!")token_ids=tokenizer.convert_tokens_to_ids(tokens)How to tokenize the input forbrightmart/albert_zh?importparams_flowpf# fetch the vocab filealbert_zh_vocab_url="https://raw.githubusercontent.com/brightmart/albert_zh/master/albert_config/vocab.txt"vocab_file=pf.utils.fetch_url(albert_zh_vocab_url,model_dir)tokenizer=bert.albert_tokenization.FullTokenizer(vocab_file)tokens=tokenizer.tokenize("你好世界")token_ids=tokenizer.convert_tokens_to_ids(tokens)How to tokenize the input for thegoogle-research/ALBERTmodels?importsentencepieceasspmspm_model=os.path.join(model_dir,"assets","30k-clean.model")sp=spm.SentencePieceProcessor()sp.load(spm_model)do_lower_case=Trueprocessed_text=bert.albert_tokenization.preprocess_text("Hello, World!",lower=do_lower_case)token_ids=bert.albert_tokenization.encode_ids(sp,processed_text)How to tokenize the input for the Chinesegoogle-research/ALBERTmodels?importbertvocab_file=os.path.join(model_dir,"vocab.txt")tokenizer=bert.albert_tokenization.FullTokenizer(vocab_file=vocab_file)tokens=tokenizer.tokenize(u"你好世界")token_ids=tokenizer.convert_tokens_to_ids(tokens)ResourcesBERT- BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingadapter-BERT- adapter-BERT: Parameter-Efficient Transfer Learning for NLPALBERT- ALBERT: A Lite BERT for Self-Supervised Learning of Language Representationsgoogle-research/bert- the originalBERTimplementationgoogle-research/ALBERT- the originalALBERTimplementation by Googlegoogle-research/albert(old)- the old location of the originalALBERTimplementation by Googlebrightmart/albert_zh- pre-trainedALBERTweights for Chinesekpe/params-flow- A Keras coding style for reducingKerasboilerplate code in custom layers by utilizingkpe/py-params