Package: langml
LangML (LanguageModeL) is a Keras-based and TensorFlow-backend language model toolkit, which provides mainstream pre-trained language models, e.g., BERT/RoBERTa/ALBERT, and their downstream application models.OutlineOutlineFeaturesInstallationQuick StartSpecify the Keras variantLoad pretrained language modelsFinetune a modelUse langml-cli to train baseline modelsDocumentationReferenceFeaturesCommon and widely-used Keras layers: CRF, Transformer, Attentions: Additive, ScaledDot, MultiHead, GatedAttentionUnit, and so on.Pretrained Language Models: BERT, RoBERTa, ALBERT. Providing friendly designed interfaces and easy to implement downstream singleton, shared/unshared two-tower or multi-tower models.Tokenizers: WPTokenizer (wordpiece), SPTokenizer (sentencepiece)Baseline models: Text Classification, Named Entity Recognition, Contrastive Learning. It's no need to write any code, and just need to preprocess the data into a specific format and use the "langml-cli" to train various baseline models.Prompt-Based Tuning: PTuningInstallationYou can install or upgrade langml/langml-cli via the following command:pipinstall-UlangmlQuick StartSpecify the Keras variantUse pure Keras (default setting)exportTF_KERAS=0Use TensorFlow KerasexportTF_KERAS=1Load pretrained language modelsfromlangmlimportWPTokenizer,SPTokenizerfromlangmlimportload_bert,load_albert# load bert / roberta plmbert_model,bert=load_bert(config_path,checkpoint_path)# load albert plmalbert_model,albert=load_albert(config_path,checkpoint_path)# load wordpiece tokenizerwp_tokenizer=WPTokenizer(vocab_path,lowercase)# load sentencepiece tokenizersp_tokenizer=SPTokenizer(vocab_path,lowercase)Finetune a modelfromlangmlimportkeras,Lfromlangmlimportload_bertconfig_path='/path/to/bert_config.json'ckpt_path='/path/to/bert_model.ckpt'vocab_path='/path/to/vocab.txt'bert_model,bert_instance=load_bert(config_path,ckpt_path)# get CLS representationcls_output=L.Lambda(lambdax:x[:,0])(bert_model.output)output=L.Dense(2,activation='softmax',kernel_intializer=bert_instance.initializer)(cls_output)train_model=keras.Model(bert_model.input,cls_output)train_model.summary()train_model.compile(loss='categorical_crossentropy',optimizer=keras.optimizer.Adam(1e-5))Use langml-cli to train baseline modelsText Classification$langml-clibaselineclf--help
Usage:langmlbaselineclf[OPTIONS]COMMAND[ARGS]...classificationcommandlinetools

Options:--helpShowthismessageandexit.

Commands:bertbilstmtextcnnNamed Entity Recognition$langml-clibaselinener--help
Usage:langmlbaselinener[OPTIONS]COMMAND[ARGS]...nercommandlinetools

Options:--helpShowthismessageandexit.

Commands:bert-crflstm-crfContrastive Learning$langml-clibaselinecontrastive--help
Usage:langmlbaselinecontrastive[OPTIONS]COMMAND[ARGS]...contrastivelearningcommandlinetools

Options:--helpShowthismessageandexit.

Commands:simcseText Matching$langml-clibaselinematching--help
Usage:langmlbaselinematching[OPTIONS]COMMAND[ARGS]...textmatchingcommandlinetools

Options:--helpShowthismessageandexit.

Commands:sbertDocumentationPlease visit thelangml.readthedocs.ioto check the latest documentation.ReferenceThe implementation of pretrained language model is inspired byCyberZHG/keras-bertandbojone/bert4keras.