Commit
•
201dd8a
1
Parent(s):
b71c38e
Update README.md
Browse files
README.md
CHANGED
@@ -34,9 +34,11 @@ This official repository unveils the TransNormerLLM3 model along with its open-s
|
|
34 |
* Join to dicussion: [discord](https://discord.gg/JEU3nTcWKC) <<<>>> [wechat group](https://github.com/OpenNLPLab/TransnormerLLM/blob/main/images/contact_me_qr.png)
|
35 |
|
36 |
> --23.12.25-- startup: [WeChat - 预训练启航](https://mp.weixin.qq.com/s/YjUY-uy89WkF75_-rBTuKw) <<<>>> [Twitter - Pre-training Commences ](https://twitter.com/opennlplab/status/1739568669502611825) <<<>>> [YouTube Recording](https://t.co/wk7svS4o5r) <<<>>> [bilibili 回放](https://www.bilibili.com/video/BV11j411J7Dy)
|
37 |
-
> --24.01.02-- first week review: [WeChat - 第一周概览](https://mp.weixin.qq.com/s/zwGnZZI3itNPoxzzXkuU2w) <<<>>> [Twitter -
|
38 |
-
> --24.01.09-- second week review: [WeChat - 第二周概览](https://mp.weixin.qq.com/s/6D0qi-0aBier05OKuHfPEA) <<<>>> [Twitter -
|
39 |
-
> --24.01.15-- third week review: [WeChat - 第三周概览](https://mp.weixin.qq.com/s/EQg8evZ2cNtAk4HruwCXPA) <<<>>> [Twitter -
|
|
|
|
|
40 |
|
41 |
|
42 |
# Released Weights
|
@@ -47,6 +49,8 @@ This official repository unveils the TransNormerLLM3 model along with its open-s
|
|
47 |
| **15B** | 100B | 🤗[step26000](https://huggingface.co/OpenNLPLab/TransNormerLLM3-15B-Intermediate-Checkpoints/tree/step26000-100Btokens) | 🤖 | 🐯 |
|
48 |
| **15B** | 150B | 🤗[step39000](https://huggingface.co/OpenNLPLab/TransNormerLLM3-15B-Intermediate-Checkpoints/tree/step39000-150Btokens) | 🤖 | 🐯 |
|
49 |
| **15B** | 200B | 🤗[step52000](https://huggingface.co/OpenNLPLab/TransNormerLLM3-15B-Intermediate-Checkpoints/tree/step52000-200Btokens) | 🤖 | 🐯 |
|
|
|
|
|
50 |
|
51 |
|
52 |
```python
|
@@ -67,6 +71,9 @@ The evaluations of all models are conducted using the official settings and the
|
|
67 |
| **TransNormerLLM3-15B** | 15 | 0.20 | 52.05 | 74.48 | 64.72 | 62.75 | 66.16 | 35.15 | 36.80 | 27.25 | 30.80 |
|
68 |
| **TransNormerLLM3-15B** | 15 | 0.25 | 66.70 | 76.50 | 66.51 | 64.80 | 66.84 | 36.18 | 39.40 | 30.87 | 36.10 |
|
69 |
| **TransNormerLLM3-15B** | 15 | 0.30 | 67.00 | 76.50 | 67.17 | 64.40 | 66.29 | 36.77 | 38.80 | 33.99 | 37.60 |
|
|
|
|
|
|
|
70 |
|
71 |
> **P**: parameter size (billion). **T**: tokens (trillion). **BoolQ**: acc. **PIQA**: acc. **HellaSwag**: acc_norm. **WinoGrande**: acc. **ARC-easy**: acc. **ARC-challenge**: acc_norm. **OpenBookQA**: acc_norm. **MMLU**: 5-shot acc. **C-Eval**: 5-shot acc.
|
72 |
|
@@ -89,11 +96,13 @@ Our project is developed based on the following open source projects:
|
|
89 |
## Citation
|
90 |
If you wish to cite our work, please use the following reference:
|
91 |
```
|
92 |
-
@
|
93 |
-
|
94 |
-
|
95 |
-
|
96 |
-
|
|
|
|
|
97 |
}
|
98 |
|
99 |
@misc{qin2024lightning,
|
|
|
34 |
* Join to dicussion: [discord](https://discord.gg/JEU3nTcWKC) <<<>>> [wechat group](https://github.com/OpenNLPLab/TransnormerLLM/blob/main/images/contact_me_qr.png)
|
35 |
|
36 |
> --23.12.25-- startup: [WeChat - 预训练启航](https://mp.weixin.qq.com/s/YjUY-uy89WkF75_-rBTuKw) <<<>>> [Twitter - Pre-training Commences ](https://twitter.com/opennlplab/status/1739568669502611825) <<<>>> [YouTube Recording](https://t.co/wk7svS4o5r) <<<>>> [bilibili 回放](https://www.bilibili.com/video/BV11j411J7Dy)
|
37 |
+
> --24.01.02-- first week review: [WeChat - 第一周概览](https://mp.weixin.qq.com/s/zwGnZZI3itNPoxzzXkuU2w) <<<>>> [Twitter - Week 1 Review](https://twitter.com/opennlplab/status/1742187694078501038)
|
38 |
+
> --24.01.09-- second week review: [WeChat - 第二周概览](https://mp.weixin.qq.com/s/6D0qi-0aBier05OKuHfPEA) <<<>>> [Twitter - Week 2 Review](https://twitter.com/opennlplab/status/1744720007299523063)
|
39 |
+
> --24.01.15-- third week review: [WeChat - 第三周概览](https://mp.weixin.qq.com/s/EQg8evZ2cNtAk4HruwCXPA) <<<>>> [Twitter - Week 3 Review](https://twitter.com/opennlplab/status/1746920293069910190)
|
40 |
+
> --24.01.23-- third week review: [WeChat - 第四周概览](https://mp.weixin.qq.com/s/l7LrFGQKkPU38exUtSF4cw) <<<>>> [Twitter - Week 4 Review](https://twitter.com/opennlplab/status/1749821039360840001)
|
41 |
+
> --24.01.30-- third week review: [WeChat - 第五周概览](https://mp.weixin.qq.com/s/OgtQIb749IbX6y5C01bLFg) <<<>>> [Twitter - Week 5 Review](https://twitter.com/opennlplab/status/1752366090754425283)
|
42 |
|
43 |
|
44 |
# Released Weights
|
|
|
49 |
| **15B** | 100B | 🤗[step26000](https://huggingface.co/OpenNLPLab/TransNormerLLM3-15B-Intermediate-Checkpoints/tree/step26000-100Btokens) | 🤖 | 🐯 |
|
50 |
| **15B** | 150B | 🤗[step39000](https://huggingface.co/OpenNLPLab/TransNormerLLM3-15B-Intermediate-Checkpoints/tree/step39000-150Btokens) | 🤖 | 🐯 |
|
51 |
| **15B** | 200B | 🤗[step52000](https://huggingface.co/OpenNLPLab/TransNormerLLM3-15B-Intermediate-Checkpoints/tree/step52000-200Btokens) | 🤖 | 🐯 |
|
52 |
+
| **15B** | 250B | 🤗[step65000](https://huggingface.co/OpenNLPLab/TransNormerLLM3-15B-Intermediate-Checkpoints/tree/step65000-250Btokens) | 🤖 | 🐯 |
|
53 |
+
| **15B** | 300B | 🤗[step78000](https://huggingface.co/OpenNLPLab/TransNormerLLM3-15B-Intermediate-Checkpoints/tree/step78000-300Btokens) | 🤖 | 🐯 |
|
54 |
|
55 |
|
56 |
```python
|
|
|
71 |
| **TransNormerLLM3-15B** | 15 | 0.20 | 52.05 | 74.48 | 64.72 | 62.75 | 66.16 | 35.15 | 36.80 | 27.25 | 30.80 |
|
72 |
| **TransNormerLLM3-15B** | 15 | 0.25 | 66.70 | 76.50 | 66.51 | 64.80 | 66.84 | 36.18 | 39.40 | 30.87 | 36.10 |
|
73 |
| **TransNormerLLM3-15B** | 15 | 0.30 | 67.00 | 76.50 | 67.17 | 64.40 | 66.29 | 36.77 | 38.80 | 33.99 | 37.60 |
|
74 |
+
| **TransNormerLLM3-15B** | 15 | 0.35 | 65.78 | 75.46 | 67.88 | 66.54 | 67.34 | 38.57 | 39.60 | 36.02 | 39.20 |
|
75 |
+
| **TransNormerLLM3-15B** | 15 | 0.40 | 67.34 | 75.24 | 68.51 | 66.22 | 68.94 | 40.10 | 39.20 | 41.10 | 39.01 |
|
76 |
+
|
77 |
|
78 |
> **P**: parameter size (billion). **T**: tokens (trillion). **BoolQ**: acc. **PIQA**: acc. **HellaSwag**: acc_norm. **WinoGrande**: acc. **ARC-easy**: acc. **ARC-challenge**: acc_norm. **OpenBookQA**: acc_norm. **MMLU**: 5-shot acc. **C-Eval**: 5-shot acc.
|
79 |
|
|
|
96 |
## Citation
|
97 |
If you wish to cite our work, please use the following reference:
|
98 |
```
|
99 |
+
@misc{qin2024transnormerllm,
|
100 |
+
title={TransNormerLLM: A Faster and Better Large Language Model with Improved TransNormer},
|
101 |
+
author={Zhen Qin and Dong Li and Weigao Sun and Weixuan Sun and Xuyang Shen and Xiaodong Han and Yunshen Wei and Baohong Lv and Xiao Luo and Yu Qiao and Yiran Zhong},
|
102 |
+
year={2024},
|
103 |
+
eprint={2307.14995},
|
104 |
+
archivePrefix={arXiv},
|
105 |
+
primaryClass={cs.CL}
|
106 |
}
|
107 |
|
108 |
@misc{qin2024lightning,
|