Package: bert4torch
Documentation|Torch4keras|Examples1. 下载安装安装稳定版pipinstallbert4torch安装最新版pipinstallgit+https://github.com/Tongjilibo/bert4torch注意事项：pip包的发布慢于git上的开发版本，git clone注意引用路径，注意权重是否需要转换测试用例：git clone https://github.com/Tongjilibo/bert4torch，修改example中的预训练模型文件路径和数据路径即可启动脚本自行训练：针对自己的数据，修改相应的数据处理代码块开发环境：原使用torch==1.10版本进行开发，现已切换到torch2.0开发，如其他版本遇到不适配，欢迎反馈2. 功能LLM模型: 加载chatglm、llama、 baichuan、ziya、bloom等开源大模型权重进行推理和微调核心功能：加载bert、roberta、albert、xlnet、nezha、bart、RoFormer、RoFormer_V2、ELECTRA、GPT、GPT2、T5、GAU-alpha、ERNIE等预训练权重继续进行finetune、并支持在bert基础上灵活定义自己模型丰富示例：包含llm、pretrain、sentence_classfication、sentence_embedding、sequence_labeling、relation_extraction、seq2seq、serving等多种解决方案实验验证：已在公开数据集实验验证，使用如下examples数据集易用trick：集成了常见的trick，即插即用其他特性：加载transformers库模型一起使用；调用方式简洁高效；有训练进度条动态展示；配合torchinfo打印参数量；默认Logger和Tensorboard简便记录训练过程；自定义fit过程，满足高阶需求训练过程：2022-10-28 23:16:10 - Start Training
2022-10-28 23:16:10 - Epoch: 1/2
5000/5000 [==============================] - 13s 3ms/step - loss: 0.1351 - acc: 0.9601
Evaluate: 100%|██████████████████████████████████████████████████| 2500/2500 [00:03<00:00, 798.09it/s] 
test_acc: 0.98045. best_test_acc: 0.98045

2022-10-28 23:16:27 - Epoch: 2/2
5000/5000 [==============================] - 13s 3ms/step - loss: 0.0465 - acc: 0.9862
Evaluate: 100%|██████████████████████████████████████████████████| 2500/2500 [00:03<00:00, 635.78it/s] 
test_acc: 0.98280. best_test_acc: 0.98280

2022-10-28 23:16:44 - Finish Training功能bert4torchtransformers备注训练进度条✅✅进度条打印loss和定义的metrics分布式训练dp/ddp✅✅torch自带dp/ddp各类callbacks✅✅日志/tensorboard/earlystop/wandb等大模型推理，stream/batch输出✅✅各个模型是通用的，无需单独维护脚本大模型微调✅✅lora依赖peft库，pv2自带丰富tricks✅❌对抗训练等tricks即插即用代码简洁易懂，自定义空间大✅❌代码复用度高, keras代码训练风格仓库的维护能力/影响力/使用量/兼容性❌✅目前仓库个人维护3. 快速上手Quick-Start快速上手教程，教程示例，实战示例bert4torch介绍(知乎)，bert4torch快速上手(知乎)，bert4torch又双叒叕更新啦(知乎)4. 版本历史更新日期bert4torchtorch4keras版本说明202402040.4.70.1.9修改save_pretrained用于保存文件夹, 增加GenerateSpeed用于统计token生成速度，修复t5在use_states=True时候的错误, 修改层次编码的bug, 增加deepseek_moe模型，修复generation并发错误，优化大模型耗时202401160.4.60.1.8bug修复，增加save_pretrained用于保存transformer格式的权重, 增加部分embedding模型202401110.4.50.1.7training时候不生成past_key_values, 增加streamlit的example, 修复句向量max时的bug,batch_generate合并到generate, 修改generation的默认参数名(兼容过去的参数名), 多轮对话中可保留past_key_values, 把attention中的mask补齐逻辑移到apply_embedding中, 增加uie的pipeline，增加PtuningV2Trainer更多版本5. 更新历史：更多历史6. 预训练权重预训练模型支持多种代码加载方式frombert4torch.modelsimportbuild_transformer_model# 1. 仅指定config_path: 从头初始化模型结构, 不加载预训练模型model=build_transformer_model('./model/bert4torch_config.json')# 2. 仅指定checkpoint_path:## 2.1 文件夹路径: 自动寻找路径下的*.bin/*.safetensors权重文件 + bert4torch_config.json/config.json文件model=build_transformer_model(checkpoint_path='./model')## 2.2 文件路径/列表: 文件路径即权重路径/列表, config会从同级目录下寻找model=build_transformer_model(checkpoint_path='./pytorch_model.bin')## 2.3 model_name: hf上预训练权重名称, 会自动下载hf权重以及bert4torch_config.json文件model=build_transformer_model(checkpoint_path='bert-base-chinese')# 3. 同时指定config_path和checkpoint_path(本地路径名或model_name排列组合):config_path='./model/bert4torch_config.json'# 或'bert-base-chinese'checkpoint_path='./model/pytorch_model.bin'# 或'bert-base-chinese'model=build_transformer_model(config_path,checkpoint_path)模型分类模型名称权重来源权重链接/checkpoint_pathconfig_pathbertbert-base-chinesegoogle-bertbert-base-chinesebert-base-chinesechinese_L-12_H-768_A-12谷歌github,tf,Tongjilibo/bert-chinese_L-12_H-768_A-12chinese-bert-wwm-extHFLgithub，hfl/chinese-bert-wwm-extchinese-bert-wwm-extbert-base-multilingual-casedgoogle-bertbert-base-multilingual-casedbert-base-multilingual-casedmacbertHFLgithub，hfl/chinese-macbert-base,hfl/chinese-macbert-largechinese-macbert-base,chinese-macbert-largewobert追一科技github，junnyu/wobert_chinese_base，junnyu/wobert_chinese_plus_basewobert_chinese_base,wobert_chinese_plus_baserobertachinese-roberta-wwm-extHFLgithub，hfl/chinese-roberta-wwm-ext,hfl/chinese-roberta-wwm-ext-largechinese-roberta-wwm-ext,chinese-roberta-wwm-ext-largeroberta-small/tiny追一科技github，Tongjilibo/chinese_roberta_L-4_H-312_A-12,Tongjilibo/chinese_roberta_L-6_H-384_A-12roberta-baseFacebookAIroberta-baseroberta-baseguwenbertethanytethanyt/guwenbert-baseguwenbert-basealbertalbertbrightmartgithub，torch,voidful/albert_chinese_tiny，voidful/albert_chinese_small,voidful/albert_chinese_base,voidful/albert_chinese_large,voidful/albert_chinese_xlarge,voidful/albert_chinese_xxlargealbert_chinese_tiny，albert_chinese_small,albert_chinese_base,albert_chinese_large,albert_chinese_xlarge,albert_chinese_xxlargenezhaNEZHA华为github，torch,sijunhe/nezha-cn-base,sijunhe/nezha-cn-large,sijunhe/nezha-base-wwm,sijunhe/nezha-large-wwmnezha-cn-base,nezha-cn-large,nezha-base-wwm,nezha-large-wwmnezha_gpt_dialogbojonegithub,Tongjilibo/nezha_gpt_dialogxlnetchinese-xlnetHFLgithub,hfl/chinese-xlnet-basechinese-xlnet-basetranformer_xlhuggingfacetransfo-xl/transfo-xl-wt103transfo-xl-wt103debertaErlangshen-DeBERTa-v2IDEAIDEA-CCNL/Erlangshen-DeBERTa-v2-97M-Chinese,IDEA-CCNL/Erlangshen-DeBERTa-v2-320M-Chinese,IDEA-CCNL/Erlangshen-DeBERTa-v2-710M-ChineseErlangshen-DeBERTa-v2-97M-Chinese,Erlangshen-DeBERTa-v2-320M-Chinese,Erlangshen-DeBERTa-v2-710M-ChineseelectraChinese-ELECTRAHFLgithub，hfl/chinese-electra-base-discriminatorchinese-electra-base-discriminatorernieernie百度文心paddle，nghuyong/ernie-1.0-base-zh,nghuyong/ernie-3.0-base-zhernie-1.0-base-zh,ernie-3.0-base-zhroformerroformer追一科技github，junnyu/roformer_chinese_baseroformer_chinese_baseroformer_v2追一科技github，junnyu/roformer_v2_chinese_char_baseroformer_v2_chinese_char_basesimbertsimbert追一科技github，Tongjilibo/simbert-chinese-base,Tongjilibo/simbert-chinese-small,Tongjilibo/simbert-chinese-tinysimbert_v2/roformer-sim追一科技github，junnyu/roformer_chinese_sim_char_base，junnyu/roformer_chinese_sim_char_ft_base，junnyu/roformer_chinese_sim_char_small，junnyu/roformer_chinese_sim_char_ft_smallroformer_chinese_sim_char_base,roformer_chinese_sim_char_ft_base,roformer_chinese_sim_char_small,roformer_chinese_sim_char_ft_smallgauGAU-alpha追一科技github,Tongjilibo/chinese_GAU-alpha-char_L-24_H-768uieuie百度github,torch,Tongjilibo/uie-basegptCDial-GPTthu-coaigithub,thu-coai/CDial-GPT_LCCC-base,thu-coai/CDial-GPT_LCCC-largeCDial-GPT_LCCC-base,CDial-GPT_LCCC-largecmp_lm(26亿)清华github,TsinghuaAI/CPM-GenerateCPM-Generatenezha_genhuawei_noahgithub,Tongjilibo/chinese_nezha_gpt_L-12_H-768_A-12gpt2-chinese-cluecorpussmallUERuer/gpt2-chinese-cluecorpussmallgpt2-chinese-cluecorpussmallgpt2-mlimcaspartf,torch,BaiduYun(84dh)gpt2-ml_15g_corpus,gpt2-ml_30g_corpusbartbart_base_chinese复旦fnlpgithub,v1.0,fnlp/bart-base-chinesebart-base-chinese,bart-base-chinese-v1.0t5t5UERuer/t5-small-chinese-cluecorpussmall,uer/t5-base-chinese-cluecorpussmallt5-base-chinese-cluecorpussmall,t5-small-chinese-cluecorpussmallmt5谷歌google/mt5-basemt5-baset5_pegasus追一科技github,Tongjilibo/chinese_t5_pegasus_small,Tongjilibo/chinese_t5_pegasus_basechatyuan v1&v2clue-aigithub,ClueAI/ChatYuan-large-v1,ClueAI/ChatYuan-large-v2ChatYuan-large-v1,ChatYuan-large-v2PromptCLUEclue-aigithub,ClueAI/PromptCLUE-basePromptCLUE-basechatglmchatglm-6bTHUDMgithub,THUDM/chatglm-6b,THUDM/chatglm-6b-int8,THUDM/chatglm-6b-int4,v0.1.0chatglm-6b,chatglm-6b-int8,chatglm-6b-int4,chatglm-6b-v0.1.0chatglm2-6bTHUDMgithub,THUDM/chatglm2-6b,THUDM/chatglm2-6b-int4,THUDM/chatglm2-6b-32kchatglm2-6b,chatglm2-6b-int4,chatglm2-6b-32kchatglm3-6bTHUDMgithub,THUDM/chatglm3-6b,THUDM/chatglm3-6b-32kchatglm3-6b,chatglm3-6b-32kllamallamametagithubllama-7b,llama-13bllama-2metagithub,meta-llama/Llama-2-7b-hf,meta-llama/Llama-2-7b-chat-hf,meta-llama/Llama-2-13b-hf,meta-llama/Llama-2-13b-chat-hfLlama-2-7b-hf,Llama-2-7b-chat-hf,Llama-2-13b-hf,Llama-2-13b-chat-hfchinese_llama_alpacaHFLgithubchinese_alpaca_plus_7b,chinese_llama_plus_7bBelle_llamaLianjiaTechgithub,BelleGroup/BELLE-LLaMA-7B-2M-enc合成说明、BELLE-LLaMA-7B-2M-encZiyaIDEA-CCNLIDEA-CCNL/Ziya-LLaMA-13B-v1,IDEA-CCNL/Ziya-LLaMA-13B-v1.1,IDEA-CCNL/Ziya-LLaMA-13B-Pretrain-v1Ziya-LLaMA-13B-v1,Ziya-LLaMA-13B-v1.1Baichuanbaichuan-incgithub,baichuan-inc/Baichuan-7B,baichuan-inc/Baichuan-13B-Base,baichuan-inc/Baichuan-13B-ChatBaichuan-7B,Baichuan-13B-Base,Baichuan-13B-ChatBaichuan2baichuan-incgithub,baichuan-inc/Baichuan2-7B-Base,baichuan-inc/Baichuan2-7B-Chat,baichuan-inc/Baichuan2-13B-Base,baichuan-inc/Baichuan2-13B-ChatBaichuan2-7B-Base,Baichuan2-7B-Chat,Baichuan2-13B-Base,Baichuan2-13B-Chatvicunalmsyslmsys/vicuna-7b-v1.5vicuna-7b-v1.5Yi01-aigithub,01-ai/Yi-6B,01-ai/Yi-6B-200KYi-6B,Yi-6B-200Kbloombloombigsciencebigscience/bloom-560m,bigscience/bloomz-560mbloom-560m,bloomz-560mQwenQwen阿里云github,Qwen/Qwen-1_8B,Qwen/Qwen-1_8B-Chat,Qwen/Qwen-7B,Qwen/Qwen-7B-ChatQwen-1_8B,Qwen-1_8B-Chat,Qwen-7B,Qwen-7B-ChatInternLMInternLM上海人工智能实验室github,internlm/internlm-chat-7b,internlm/internlm-7binternlm-7b,internlm-chat-7bFalconFalcontiiuaehf,tiiuae/falcon-rw-1b,tiiuae/falcon-7b,tiiuae/falcon-7b-instructfalcon-rw-1b,falcon-7b,falcon-7b-instructmoedeeoseek-moedeepseekgithub,deepseek-ai/deepseek-moe-16b-base,deepseek-ai/deepseek-moe-16b-chatdeepseek-moe-16b-base,deepseek-moe-16b-chatembeddingtext2vec-base-chineseshibing624shibing624/text2vec-base-chinesetext2vec-base-chinesem3emoka-aimoka-ai/m3e-basem3e-basebgeBAAIBAAI/bge-large-en-v1.5,BAAI/bge-large-zh-v1.5bge-large-en-v1.5,bge-large-zh-v1.5gtethenlperthenlper/gte-large-zh,thenlper/gte-base-zhgte-base-zh,gte-large-zh*注：高亮格式(如bert-base-chinese)的表示可直接build_transformer_model()联网下载国内镜像网站加速下载HF_ENDPOINT=https://hf-mirror.com python your_script.pyexport HF_ENDPOINT=https://hf-mirror.com后再执行python代码在python代码开头如下设置importosos.environ['HF_ENDPOINT']="https://hf-mirror.com"7. 鸣谢感谢苏神实现的bert4keras，本实现有不少地方参考了bert4keras的源码，在此衷心感谢大佬的无私奉献;其次感谢项目bert4pytorch，也是在该项目的指引下给了我用pytorch来复现bert4keras的想法和思路。8. 引用@misc{bert4torch,
  title={bert4torch},
  author={Bo Li},
  year={2022},
  howpublished={\url{https://github.com/Tongjilibo/bert4torch}},
}9. 其他Wechat & Star History Chart微信号微信群Star History Chart