Hugging Face's logo

MediaTek-Research
/
Breeze-7B-Instruct-v0_1

Text Generation
Transformers
PyTorch
Safetensors
Chinese
English
mistral
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
5
YC-Chen commited on
Commit
b2afa7e
•
1 Parent(s): c1eb6f3

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +43 -46
README.md CHANGED
@@ -28,46 +28,42 @@ and is comparable with Mistral-7B-Instruct-v0.1 on MMLU and MT-Bench in English.
28
 
29
  ## Base Model Performance
30
 
31
- | Models | | TMMLU+ (ACC) | DRCD (EM) | MMLU (ACC) |
32
- |---------------------------------------------|--------|--------------|-----------|------------|
33
- | | |TC, Knowledge |TC, Reasoning|EN, Knowledge|
34
- | | | 5 shot | 3 shot | 5 shot |
35
- | 01-ai/Yi-6B | 6B | 49.63 | 76.61 | 65.35 |
36
- | MediaTek-Research/Breeze-7B-Base-v0.1 | 7B | 40.35 | 81.13 | 61.63 |
37
- | mistralai/Mistral-7B-v0.1 | 7B | 36.93 | 79.27 | 64.89 |
38
- | Qwen/Qwen-7B | 7B | 42.84 | 0.0 * | 61.00 |
39
- | Qwen/Qwen-14B | 14B | 51.30 | 16.95 * | 68.83 |
40
- | 01-ai/Yi-34B | 34B | 63.10 | 84.57 | 77.42 |
 
41
 
42
  \* Few-shot learning cannot effectively guide the model to generate the proper answer.
43
 
44
- | Category ACC of TMMLU+ | STEM | Social Science | Humanities | Other |
45
  |-----------------------------------------------------|--------------|----------------|------------|------------|
46
- | 01-ai/Yi-6B | 41.14 | 57.77 | 50.22 | 49.39 |
47
- | MediaTek-Research/Breeze-7B-Base-v0.1 | 35.74 | 46.08 | 40.29 | 39.27 |
48
- | mistralai/Mistral-7B-v0.1 | 33.01 | 42.23 | 35.86 | 37.63 |
49
- | Qwen/Qwen-7B | 28.25 | 47.80 | 43.14 | 42.17 |
50
- | Qwen/Qwen-14B | 46.51 | 58.20 | 51.12 | 49.38 |
51
- | 01-ai/Yi-34B | 56.03 | 73.06 | 61.12 | 62.19 |
52
 
53
 
54
  ## Inference Performance
55
 
56
- | Models | Speed (char/sec) |Estimated Max Input Length (TC Char)|
57
  |--------------------------------------------------------------------|-------------------|--------------------------|
58
- | MediaTek-Research/Breeze-7B-Base-v0.1 | 59.57 | 10.1k |
59
- | mistralai/Mistral-7B-v0.1 | 45.31 | 6.3k |
60
- | yentinglin/Taiwan-LLM-7B-v2.1-base | 16.23 | 2.6k |
61
- | yentinglin/Taiwan-LLM-13B-v2.0-base | 19.61 | 2.6k |
62
- | 01-ai/Yi-6B | 62.08 | 4.4k |
63
- | 01-ai/Yi-34B | 15.18 | 4.4k |
64
- | Qwen/Qwen-7B | 55.00 | 9.7k |
65
- | Qwen/Qwen-14B | 51.12 | 9.7k |
66
-
67
- \*Test Input: 201802最高法院民事裁判書(first 1500 characters)
68
- \*Test Prompt: 請重新寫出這篇文章
69
- \*Output Token: 1000 (We set max_tokens=400 for Qwen/Qwen-7B)
70
- \*Using vllm, TP=2 on 2 A6000
71
 
72
  ## Chat Model Performance
73
 
@@ -75,27 +71,28 @@ and is comparable with Mistral-7B-Instruct-v0.1 on MMLU and MT-Bench in English.
75
  |--------------------------------------------|--------|--------------|--------------|-----------|---------------------|------------|------------|------------------|
76
  | | |TC, Knowledge |TC, Knowledge |TC, Reasoning|TC, Chat |EN, Knowledge|EN, Knowledge|EN, Chat |
77
  | | | 0 shot | 5 shot | 3 shot | 0 shot | 0 shot | 5 shot | 0 shot |
78
- | 01-ai/Yi-6B-Chat | 6B | 44.79 | | | 5.0 | 59.45 | | 6.0 |
79
- | MediaTek-Research/Breeze-7B-Instruct-v0.1 | 7B | 41.61 | | | 5.7 | 63.26 | | 7.1 |
80
- | MediaTek-Research/Breeze-7B-Instruct-64k-v0.1| 7B | 40.99 | | | | 63.68 | | 7.1 |
81
- | yentinglin/Taiwan-LLM-7B-v2.1-chat | 7B | 28.08 | | | 4.2 | 42.72 | | -* |
82
- | Qwen/Qwen-7B-Chat | 7B | 40.02 | | | 5.4 | 55.94 | | 6.2 |
83
- | yentinglin/Taiwan-LLM-13B-v2.0-chat | 13B | 29.47 | | | 5.0 | 50.50 | | -* |
84
- | Qwen/Qwen-14B-Chat | 14B | 48.41 | | | 6.4 | 64.91 | | 7.2 |
85
- | 01-ai/Yi-34B-Chat | 34B | 54.87 | | | 6.9 | 71.04 | | 7.6 |
86
  | gpt-3.5-turbo-1106 | | | | 76.30 | 7.1 | | | 7.9 |
 
 
 
 
 
 
 
 
 
87
 
88
  \* Taiwan-LLM models responds to multi-turn questions (English) in Traditional Chinese.
89
 
90
  | Category ACC of TMMLU+ | STEM | Social Science | Humanities | Other |
91
  |-----------------------------------------------------|--------------|----------------|------------|------------|
92
- | 01-ai/Yi-6B-Chat | 26.28 | 33.48 | 29.48 | 27.62 |
93
- | MediaTek-Research/Breeze-7B-Instruct-v0.1 | 37.45 | 48.35 | 40.26 | 40.44 |
94
- | yentinglin/Taiwan-LLM-7B-v2.1-chat | 26.53 | 29.47 | 26.11 | 26.90 |
95
- | Qwen/Qwen-7B-Chat | 32.89 | 44.26 | 38.21 | 37.83 |
96
- | yentinglin/Taiwan-LLM-13B-v2.0-chat | 29.68 | 37.13 | 30.31 | 30.55 |
97
- | Qwen/Qwen-14B-Chat | 46.51 | 58.20 | 51.12 | 49.38 |
98
- | 01-ai/Yi-34B-Chat | 46.36 | 65.02 | 52.84 | 52.21 |
99
  | gpt-3.5-turbo-1106 | | | | |
100
 
101
  ## Examples
 
28
 
29
  ## Base Model Performance
30
 
31
+ | Models | | TMMLU+ (ACC) | DRCD (EM) | MMLU (ACC) |
32
+ |----------------------------------------------|--------|--------------|-------------|------------|
33
+ | | |TC, Knowledge |TC, Reasoning|EN, Knowledge|
34
+ | | | 5 shot | 3 shot | 5 shot |
35
+ | [Yi-34B](https://huggingface.co/01-ai/Yi-34B)| 34B | 63.10 | 84.57 | 77.42 |
36
+ | [Qwen-14B](https://huggingface.co/01-ai/Qwen/Qwen-14B)| 14B | 51.30 | 16.95 * | 68.83 |
37
+ | [Yi-6B](https://huggingface.co/01-ai/Yi-6B) | 6B | 49.63 | 76.61 | 65.35 |
38
+ | [Qwen-7B](https://huggingface.co/01-ai/Qwen/Qwen-7B)| 7B | 42.84 | 0.0 * | 61.00 |
39
+ | [Breeze-7B-Base-v0.1](https://huggingface.co/MediaTek-Research/Breeze-7B-Base-v0.1) | 7B | 40.35 | 81.13 | 61.63 |
40
+ | [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)| 7B | 36.93 | 79.27 | 64.89 |
41
+
42
 
43
  \* Few-shot learning cannot effectively guide the model to generate the proper answer.
44
 
45
+ | Category ACC of TMMLU+ (5 shot) | STEM | Social Science | Humanities | Other |
46
  |-----------------------------------------------------|--------------|----------------|------------|------------|
47
+ | Yi-34B | 56.03 | 73.06 | 61.12 | 62.19 |
48
+ | Qwen-14B | 46.51 | 58.20 | 51.12 | 49.38 |
49
+ | Yi-6B | 41.14 | 57.77 | 50.22 | 49.39 |
50
+ | Qwen-7B | 28.25 | 47.80 | 43.14 | 42.17 |
51
+ | Breeze-7B-Base-v0.1 | 35.74 | 46.08 | 40.29 | 39.27 |
52
+ | Mistral-7B-v0.1 | 33.01 | 42.23 | 35.86 | 37.63 |
53
 
54
 
55
  ## Inference Performance
56
 
57
+ | Models | Speed (char/sec) |Max Input Length (TC Char)|
58
  |--------------------------------------------------------------------|-------------------|--------------------------|
59
+ | Breeze-7B-Base-v0.1 | | |
60
+ | Mistral-7B-v0.1 | | |
61
+ | Taiwan-LLM-7B-v2.1-base | | |
62
+ | Taiwan-LLM-13B-v2.0-base | | |
63
+ | Yi-6B | | |
64
+ | Yi-34B | | |
65
+ | Qwen-7B | | |
66
+ | Qwen-14B | | |
 
 
 
 
 
67
 
68
  ## Chat Model Performance
69
 
 
71
  |--------------------------------------------|--------|--------------|--------------|-----------|---------------------|------------|------------|------------------|
72
  | | |TC, Knowledge |TC, Knowledge |TC, Reasoning|TC, Chat |EN, Knowledge|EN, Knowledge|EN, Chat |
73
  | | | 0 shot | 5 shot | 3 shot | 0 shot | 0 shot | 5 shot | 0 shot |
 
 
 
 
 
 
 
 
74
  | gpt-3.5-turbo-1106 | | | | 76.30 | 7.1 | | | 7.9 |
75
+ | [Yi-34B-Chat](https://huggingface.co/01-ai/Yi-34B-Chat)| 34B | 54.87 | | | 6.9 | 71.04 | | 7.6 |
76
+ | [Qwen-14B-Chat](https://huggingface.co/Qwen/Qwen-14B-Chat)| 14B | 48.41 | | | 6.4 | 64.91 | | 7.2 |
77
+ | [Yi-6B](https://huggingface.co/01-ai/Yi-6B)| 6B | 44.79 | | | 5.0 | 59.45 | | 6.0 |
78
+ | [Breeze-7B-Instruct-v0.1](https://huggingface.co/MediaTek-Research/Breeze-7B-Instruct-v0.1)| 7B | 41.61 | | | 5.7 | 63.26 | | 7.1 |
79
+ | [Breeze-7B-Instruct-64k-v0.1](https://huggingface.co/MediaTek-Research/Breeze-7B-Instruct-64k-v0.1)| 7B | 40.99 | | | | 63.68 | | 7.1 |
80
+ | [Qwen-7B-Chat](https://huggingface.co/Qwen/Qwen-7B-Chat) | 7B | 40.02 | | | 5.4 | 55.94 | | 6.2 |
81
+ | [Taiwan-LLM-13B-v2.0-chat](https://huggingface.co/yentinglin/Taiwan-LLM-13B-v2.0-chat) | 13B | 29.47 | | | 5.0 | 50.50 | | -* |
82
+ | [Taiwan-LLM-7B-v2.1-chat](https://huggingface.co/yentinglin/Taiwan-LLM-7B-v2.1-chat) | 7B | 28.08 | | | 4.2 | 42.72 | | -* |
83
+
84
 
85
  \* Taiwan-LLM models responds to multi-turn questions (English) in Traditional Chinese.
86
 
87
  | Category ACC of TMMLU+ | STEM | Social Science | Humanities | Other |
88
  |-----------------------------------------------------|--------------|----------------|------------|------------|
89
+ | Yi-6B-Chat | 26.28 | 33.48 | 29.48 | 27.62 |
90
+ | Breeze-7B-Instruct-v0.1 | 37.45 | 48.35 | 40.26 | 40.44 |
91
+ | Taiwan-LLM-7B-v2.1-chat | 26.53 | 29.47 | 26.11 | 26.90 |
92
+ | Qwen-7B-Chat | 32.89 | 44.26 | 38.21 | 37.83 |
93
+ | Taiwan-LLM-13B-v2.0-chat | 29.68 | 37.13 | 30.31 | 30.55 |
94
+ | Qwen-14B-Chat | 46.51 | 58.20 | 51.12 | 49.38 |
95
+ | Yi-34B-Chat | 46.36 | 65.02 | 52.84 | 52.21 |
96
  | gpt-3.5-turbo-1106 | | | | |
97
 
98
  ## Examples