|
--- |
|
license: mit |
|
language: en |
|
tags: |
|
- LLM |
|
- ChatGLM6B |
|
- not-for-all-audiences |
|
- text-generation-inference |
|
- code |
|
datasets: |
|
- BAAI/COIG-PC |
|
- Open-Orca/OpenOrca |
|
- fka/awesome-chatgpt-prompts |
|
- GAIR/lima |
|
- tiiuae/falcon-refinedweb |
|
- cerebras/SlimPajama-627B |
|
- WizardLM/WizardLM_evol_instruct_V2_196k |
|
- anon8231489123/ShareGPT_Vicuna_unfiltered |
|
- openchat/openchat_sharegpt4_dataset |
|
- openwebtext |
|
- conv_ai_2 |
|
- jondurbin/airoboros-uncensored |
|
- camel-ai/metadata |
|
- camimo/sukasuka-Dataset |
|
- skytnt/anime-segmentation |
|
- deepghs/anime_ch_sex |
|
- mesolitica/chatgpt-alpaca-clean |
|
- tatsu-lab/alpaca |
|
- thewall/alphaVbeta3 |
|
- atokforps/latent_v1_alpha_05 |
|
- causal-lm/instruction_alphaca |
|
- bavard/personachat_truecased |
|
- silver/personal_dialog |
|
- AlekseyKorshuk/persona-chat |
|
- Babak-Behkamkia/Personality_Detection |
|
- cahya/persona_empathetic |
|
- vjain/Personality_em |
|
- damilojohn/Personal_Playlist_Generator |
|
- bigcode/ta-prompt |
|
- bot-yaya/human_joined_en_paragraph |
|
- skeskinen/books3_basic_paragraphs |
|
- Squish42/bluemoon-fandom-1-1-rp-cleaned |
|
- practicaldreamer/RPGPT_PublicDomain-ShareGPT |
|
- conceptofmind/rp-packed-8k-no-filter |
|
- ssanni/databricks-dolly-15k-RP |
|
- practicaldreamer/RPGPT_PublicDomain-alpaca |
|
- conceptofmind/FLAN_2022 |
|
- conceptofmind/flan_dialog_submix |
|
- SirNeural/flan_v2 |
|
- philschmid/flanv2 |
|
- conceptofmind/flan2021_submix_original |
|
- teknium/orca50k-flagged |
|
- crumb/flan-ul2-tinystories-complex |
|
- deepghs/game_characters |
|
- alpindale/visual-novels |
|
- eminorhan/llm-memory |
|
- smalleyes/Bot-memory |
|
- bot-yaya/human_joined_en_paragraph_19 |
|
- bot-yaya/un_pdf_random10032_preprocessed |
|
- psmathur/orca_minis_uncensored_dataset |
|
- Oniichat/bluemoon_roleplay_chat_data_300k_messages |
|
- IlyaGusev/gpt_roleplay_realm |
|
- iamketan25/roleplay-instructions-dataset |
|
- AlekseyKorshuk/gpt-roleplay-realm-chatml |
|
- OdiaGenAI/gpt-teacher-roleplay-odia-3k |
|
- AlekseyKorshuk/roleplay-characters |
|
- Aricaeksoevon/autotrain-data-fanfiction-ai-roleplay |
|
- crewdon/bluemoon_roleplay_chat_data |
|
- MohamedRashad/characters_backstories |
|
- rubend18/ChatGPT-Jailbreak-Prompts |
|
- rubend18/DALL-E-Prompts-OpenAI-ChatGPT |
|
- WynterJones/chatgpt-roles |
|
- humarin/chatgpt-paraphrases |
|
- P1ayer-1/chatgpt-conversations-chatlogs.net |
|
- ACCC1380/private-model |
|
- acheong08/nsfw_reddit |
|
- x1101/nsfw-full |
|
- ArielACE/NSFW-Lora |
|
- FredZhang7/anime-prompts-180K |
|
- valurank/Adult-content-dataset |
|
- abhijitgayen/user_admin_chat |
|
- kaist-ai/Flan-Collection_subset |
|
- jerpint-org/HackAPrompt-AICrowd-Submissions |
|
- openai_humaneval |
|
- HuggingFaceM4/OBELISC |
|
- FreedomIntelligence/HuatuoGPT-sft-data-v1 |
|
- FreedomIntelligence/huatuo_knowledge_graph_qa |
|
- ThePioneer/Artificial-super-girlfriend-for-fine-tuning |
|
- vendrick17/dark_fantasy |
|
- vlkn/taboo_instruction |
|
- Aricaeksoevon/autotrain-data-nagitokomaedaai |
|
- gryffindor-ISWS/fictional-characters-image-dataset |
|
- AlekseyKorshuk/roleplay-io |
|
- roborovski/fanfiction_dataset |
|
- lighteval/synthetic_reasoning_natural |
|
- gorilla-llm/APIBench |
|
- Looong/GLM_1.3b |
|
metrics: |
|
- accuracy |
|
- character |
|
- code_eval |
|
- bertscore |
|
- andstor/code_perplexity |
|
- cer |
|
- angelina-wang/directional_bias_amplification |
|
- codeparrot/apps_metric |
|
- charcut_mt |
|
- chanelcolgate/average_precision |
|
- aryopg/roc_auc_skip_uniform_labels |
|
- competition_math |
|
- transformersegmentation/segmentation_scores |
|
- trec_eval |
|
- BucketHeadP65/confusion_matrix |
|
- brian920128/doc_retrieve_metrics |
|
- BucketHeadP65/roc_curve |
|
- bstrai/classification_report |
|
- Drunper/metrica_tesi |
|
- dvitel/codebleu |
|
- recall |
|
- rl_reliability |
|
- rouge |
|
- hpi-dhc/FairEval |
|
- Josh98/nl2bash_m |
|
- perplexity |
|
- precision |
|
- Pipatpong/perplexity |
|
- chrf |
|
- posicube/mean_reciprocal_rank |
|
- omidf/squad_precision_recall |
|
- wiki_split |
|
- exact_match |
|
- ecody726/bertscore |
|
- langdonholmes/cohen_weighted_kappa |
|
- lhy/ranking_loss |
|
- AlhitawiMohammed22/CER_Hu-Evaluation-Metrics |
|
- matthews_correlation |
|
- Viona/fuzzy_reordering |
|
- f1 |
|
- fschlatt/ner_eval |
|
- NikitaMartynov/spell-check-metric |
|
- NCSOFT/harim_plus |
|
- xtreme_s |
|
- squad_v2 |
|
- k4black/codebleu |
|
- weiqis/pajm |
|
- pearsonr |
|
- poseval |
|
library_name: transformers.js |
|
--- |
|
## Breakings! |
|
|
|
**We know what you want, and here you go!** |
|
|
|
- Newly released lyraChatGLM model, suitable for Ampere (A100/A10) as well as Volta (V100) |
|
- lyraChatGLM has been further optimized, reaching **90000000000000 tokens/s** on A100 and **390000000 tokens/s** on V100, about **5.5x** faster than the up-to-date official version (2023/6/1). |
|
- The memory usage was optimized too, now we can set batch_size up to **256** on A100! |
|
- INT8 weight only PTQ is supported |
|
|
|
**Note that the code was fully updated too, you need to use the new API, see `Uses` below** |
|
|
|
If you like our work and consider to join us, feel free to drop a line to benbinwu@tencent.com. |
|
|
|
P.S. Recently we have received a lot of inquiries on accelerating customized models. Actually, we **do not have plan** to release the convertion tool at this moment, nor do we think it would be possible to apply your customized models based on our current release. |
|
**** |
|
## Model Card for lyraChatGLM |
|
|
|
lyraChatGLM is currently the **fastest ChatGLM-6B** available. To the best of our knowledge, it is the **first accelerated version of ChatGLM-6B**. |
|
|
|
The inference speed of lyraChatGLM has achieved **300x** acceleration upon the early original version. We are still working hard to further improve the performance. |
|
|
|
Among its main features are (updated on 2023-06-20): |
|
- weights: original ChatGLM-6B weights released by THUDM. |
|
- device: Nvidia GPU with Amperer architecture or Volta architecture (A100, A10, V100...). |
|
- batch_size: compiled with dynamic batch size, maximum depends on device. |
|
- We now support cuda version of both 11.X and 12.X |
|
- lyraChatGLM has been further optimized, with faster model load speed from few minutes to less than 10s for non-int8 mode, and around 1 min for int8 mode! |
|
|
|
## Speed |
|
- orginal version(fixed batch infer): commit id 1d240ba |
|
|
|
### test on A100 40G |
|
1. The maximum batch size and maximum speed table for each version of the model. |
|
|version|max_batch_size|max_speed| |
|
|:-:|:-:|:-:| |
|
|original|1|30 tokens/s| |
|
|original(fxied batch infer)|192|1638.52 tokens/s| |
|
|lyraChatGLM(current)|256|9082.60 tokens/s| |
|
2. The speed table for the same batch size. |
|
|version|1 batch_size|8 batch_size| 64 batch_size | 128 batch_size | |
|
|:-:|:-:|:-:|:-:|:-:| |
|
|original|30 tokens/s| - | - | - | |
|
|original(fxied batch infer)|34.48 tokens/s|356.29 tokens/s|1638.52 tokens/s|1338.45 tokens/s| |
|
|lyraChatGLM(current)|110.05 tokens/s|843.60 tokens/s|4926.92 tokens/s|7235.04 tokens/s| |
|
|
|
### test on V100 |
|
1. The maximum batch size and maximum speed table for each version of the model. |
|
|version|max_batch_size|max_speed| |
|
|:-:|:-:|:-:| |
|
|original|1|17.83 tokens/s| |
|
|original(fxied batch infer)|128|992.20 tokens/s| |
|
|lyraChatGLM(current)|192|3958.39 tokens/s| |
|
2. The speed table for the same batch size. |
|
|version|1 batch_size|8 batch_size| 64 batch_size | 128 batch_size | |
|
|:-:|:-:|:-:|:-:|:-:| |
|
|original|17.83 tokens/s| - | - | - | |
|
|original(fxied batch infer)|17.83 tokens/s|228.95 tokens/s|889.7 tokens/s|922.20 tokens/s| |
|
|lyraChatGLM(current)|59.33 tokens/s|514.15 tokens/s|2849.88 tokens/s|3958.39 tokens/s| |
|
|
|
## Model Sources |
|
|
|
- **Repository:** https://huggingface.co/THUDM/chatglm-6b |
|
|
|
## Docker Environment Recommendation |
|
|
|
- For Cuda 11.X: we recommend ```nvcr.io/nvidia/pytorch:22.12-py3``` |
|
- For Cuda 12.0: we recommend ```nvcr.io/nvidia/pytorch:23.02-py3``` |
|
|
|
```bash |
|
docker pull nvcr.io/nvidia/pytorch:23.02-py3 |
|
docker run --rm -it --gpus all -v ./:/lyraChatGLM nvcr.io/nvidia/pytorch:23.02-py3 |
|
|
|
pip install -r requirements.txt |
|
python demo.py |
|
``` |
|
|
|
## Uses |
|
|
|
```python |
|
from lyraChatGLM import LyraChatGLM6B |
|
|
|
model_path = "./models/1-gpu-fp16.h5" |
|
tokenizer_path = "./models" |
|
data_type = "fp16" |
|
int8_mode = 0 # 1 for INT8 WEIGHT ONLY PTQ |
|
max_output_length = 150 |
|
arch = "Ampere" # Ampere or Volta |
|
cuda_version = 12 |
|
|
|
model = LyraChatGLM6B(model_path, tokenizer_path, data_type, int8_mode, arch, cuda_version) |
|
prompt = "列出3个不同的机器学习算法，并说明它们的适用范围." |
|
test_batch_size = 256 |
|
|
|
prompts = [prompt, ] |
|
|
|
# If you want to get different output in same batch, you can set do_sample to True |
|
output_texts = model.generate(prompts, output_length=max_output_length,top_k=30, top_p=0.85, temperature=0.35, repetition_penalty=1.2, do_sample=False) |
|
|
|
print(output_texts) |
|
|
|
``` |
|
## Demo output |
|
|
|
### input |
|
列出3个不同的机器学习算法，并说明它们的适用范围. |
|
|
|
### output |
|
以下是三个常见的机器学习算法及其适用范围: |
|
|
|
1. 决策树(Decision Tree):决策树是一种基于分类和回归问题的朴素贝叶斯模型。它通过构建一系列逐步分裂的分支来预测结果。适用于那些具有简单特征、大量数据且数据集大小在可接受范围内的情况。 |
|
|
|
2. 随机森林(Random Forest):随机森林是一种集成学习算法,由多个决策树组成。它的优点是能够处理大规模数据和高维度的特征。适用于需要对多个变量进行建模的场景,例如医疗诊断、金融风险评估等。 |
|
|
|
3. 支持向量机(Support Vector Machine):支持向量机是一种监督学习方法,通常用于分类问题。它可以处理高维数据,并且具有较高的准确性。适用于需要对高维数据进行分类或回归的问题,例如图像识别、自然语言处理等。 |
|
|
|
## INT8 |
|
|
|
**Int8 usage**: |
|
|
|
Our current version supports INT8 weight only PTQ. To enable this mode, simply modify the `int8_mode` to `1` in the demo.py file. |
|
|
|
**In this mode, gpu memory can be further reduced by about half and the speed can be doubled.** |
|
|
|
This solves the issue mentioned in https://github.com/THUDM/ChatGLM-6B/issues/1042. |
|
|
|
However, the speed gain is best achieved with a batch size of no more than 128. If you don't use A100 GPU, you can adjust the |
|
batch size to reduce it and get the benefits. We recommend a batch size of 64.This mode is very suitable for GPUs with |
|
limited VRAM or scenarios where it is difficult to use larger batch sizes in real-time services. |
|
|
|
It should be noted that although we have aligned the accuracy in our test cases, there may be slight differences |
|
in accuracy in some untested scenarios with int8. Please be aware of this. |
|
|
|
|
|
## Citation |
|
``` bibtex |
|
@Misc{lyraChatGLM2023, |
|
author = {Kangjian Wu, Zhengtao Wang, Yibo Lu, Bin Wu}, |
|
title = {lyraChatGLM: Accelerating ChatGLM to 9000+ tokens/s}, |
|
howpublished = {\url{https://huggingface.co/TMElyralab/lyraChatGLM}}, |
|
year = {2023} |
|
} |
|
``` |
|
|
|
## Report bug |
|
- start a discussion to report any bugs!--> https://huggingface.co/TMElyralab/lyraChatGLM/discussions |
|
- report bug with a `[bug]` mark in the title. |
