Hugging Face's logo

RichardErkhov
/
MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf

GGUF
Inference Endpoints
conversational
Model card Files Community
RichardErkhov commited on
Commit
f6dde6f
•
1 Parent(s): fee7305

uploaded readme

Browse files
Files changed (1) hide show
  1. README.md +273 -0
README.md ADDED
@@ -0,0 +1,273 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Quantization made by Richard Erkhov.
2
+
3
+ [Github](https://github.com/RichardErkhov)
4
+
5
+ [Discord](https://discord.gg/pvy7H8DZMG)
6
+
7
+ [Request more models](https://github.com/RichardErkhov/quant_request)
8
+
9
+
10
+ Breeze-7B-Instruct-v0_1 - GGUF
11
+ - Model creator: https://huggingface.co/MediaTek-Research/
12
+ - Original model: https://huggingface.co/MediaTek-Research/Breeze-7B-Instruct-v0_1/
13
+
14
+
15
+ | Name | Quant method | Size |
16
+ | ---- | ---- | ---- |
17
+ | [Breeze-7B-Instruct-v0_1.Q2_K.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q2_K.gguf) | Q2_K | 2.66GB |
18
+ | [Breeze-7B-Instruct-v0_1.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.IQ3_XS.gguf) | IQ3_XS | 2.95GB |
19
+ | [Breeze-7B-Instruct-v0_1.IQ3_S.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.IQ3_S.gguf) | IQ3_S | 3.11GB |
20
+ | [Breeze-7B-Instruct-v0_1.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q3_K_S.gguf) | Q3_K_S | 3.09GB |
21
+ | [Breeze-7B-Instruct-v0_1.IQ3_M.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.IQ3_M.gguf) | IQ3_M | 3.2GB |
22
+ | [Breeze-7B-Instruct-v0_1.Q3_K.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q3_K.gguf) | Q3_K | 3.42GB |
23
+ | [Breeze-7B-Instruct-v0_1.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q3_K_M.gguf) | Q3_K_M | 3.42GB |
24
+ | [Breeze-7B-Instruct-v0_1.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q3_K_L.gguf) | Q3_K_L | 3.7GB |
25
+ | [Breeze-7B-Instruct-v0_1.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.IQ4_XS.gguf) | IQ4_XS | 3.83GB |
26
+ | [Breeze-7B-Instruct-v0_1.Q4_0.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q4_0.gguf) | Q4_0 | 3.99GB |
27
+ | [Breeze-7B-Instruct-v0_1.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.IQ4_NL.gguf) | IQ4_NL | 4.03GB |
28
+ | [Breeze-7B-Instruct-v0_1.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q4_K_S.gguf) | Q4_K_S | 4.01GB |
29
+ | [Breeze-7B-Instruct-v0_1.Q4_K.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q4_K.gguf) | Q4_K | 4.23GB |
30
+ | [Breeze-7B-Instruct-v0_1.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q4_K_M.gguf) | Q4_K_M | 4.23GB |
31
+ | [Breeze-7B-Instruct-v0_1.Q4_1.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q4_1.gguf) | Q4_1 | 4.41GB |
32
+ | [Breeze-7B-Instruct-v0_1.Q5_0.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q5_0.gguf) | Q5_0 | 4.83GB |
33
+ | [Breeze-7B-Instruct-v0_1.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q5_K_S.gguf) | Q5_K_S | 4.83GB |
34
+ | [Breeze-7B-Instruct-v0_1.Q5_K.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q5_K.gguf) | Q5_K | 4.95GB |
35
+ | [Breeze-7B-Instruct-v0_1.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q5_K_M.gguf) | Q5_K_M | 4.95GB |
36
+ | [Breeze-7B-Instruct-v0_1.Q5_1.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q5_1.gguf) | Q5_1 | 5.25GB |
37
+ | [Breeze-7B-Instruct-v0_1.Q6_K.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q6_K.gguf) | Q6_K | 5.72GB |
38
+ | [Breeze-7B-Instruct-v0_1.Q8_0.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q8_0.gguf) | Q8_0 | 7.41GB |
39
+
40
+
41
+
42
+
43
+ Original model description:
44
+ ---
45
+ pipeline_tag: text-generation
46
+ license: apache-2.0
47
+ language:
48
+ - zh
49
+ - en
50
+ ---
51
+
52
+ # Model Card for MediaTek Research Breeze-7B-Instruct-v0_1
53
+
54
+ MediaTek Research Breeze-7B (hereinafter referred to as Breeze-7B) is a language model family that builds on top of [Mistral-7B](https://huggingface.co/mistralai/Mistral-7B-v0.1), specifically intended for Traditional Chinese use.
55
+
56
+ [Breeze-7B-Base](https://huggingface.co/MediaTek-Research/Breeze-7B-Base-v0_1) is the base model for the Breeze-7B series.
57
+ It is suitable for use if you have substantial fine-tuning data to tune it for your specific use case.
58
+
59
+ [Breeze-7B-Instruct](https://huggingface.co/MediaTek-Research/Breeze-7B-Instruct-v0_1) derives from the base model Breeze-7B-Base, making the resulting model amenable to be used as-is for commonly seen tasks.
60
+
61
+ [Breeze-7B-Instruct-64k](https://huggingface.co/MediaTek-Research/Breeze-7B-Instruct-64k-v0_1) is a slightly modified version of
62
+ Breeze-7B-Instruct to enable a 64k-token context length. Roughly speaking, that is equivalent to 88k Traditional Chinese characters.
63
+
64
+ *Update (Feb. 21st, 2024): Breeze-7B-Instruct-64k-v0_1 has been temporarily removed from Hugging Face due to its actual performance in long context tests not meeting expectations.*
65
+
66
+ *Update (Mar. 7th, 2024): The current release version of Breeze-7B is v1.0. See [Breeze-7B-Instruct-v1_0](https://huggingface.co/MediaTek-Research/Breeze-7B-Instruct-v1_0).*
67
+
68
+ The current release version of Breeze-7B is v0.1.
69
+
70
+ Practicality-wise:
71
+ - Breeze-7B-Base expands the original vocabulary with additional 30,000 Traditional Chinese tokens. With the expanded vocabulary, everything else being equal, Breeze-7B operates at twice the inference speed for Traditional Chinese to Mistral-7B and Llama 7B. [See [Inference Performance](#inference-performance).]
72
+ - Breeze-7B-Instruct can be used as is for common tasks such as Q&A, RAG, multi-round chat, and summarization.
73
+ - In particular, Breeze-7B-Instruct-64k can perform tasks at a document level, not a chapter level.
74
+
75
+
76
+ Performance-wise:
77
+ - Breeze-7B-Instruct demonstrates impressive performance in benchmarks for Traditional Chinese and English, when compared to similar sized open-source contemporaries such as Taiwan-LLM-7B/13B-chat, QWen-7B-Chat, and Yi-6B-Chat. [See [Chat Model Performance](#chat-model-performance).]
78
+
79
+
80
+ *A project by the members (in alphabetical order): Chan-Jan Hsu 許湛然, Chang-Le Liu 劉昶樂, Feng-Ting Liao 廖峰挺, Po-Chun Hsu 許博竣, Yi-Chang Chen 陳宜昌, and the supervisor Da-Shan Shiu 許大山.*
81
+
82
+ ## Features
83
+
84
+ - Breeze-7B-Base-v0_1
85
+ - Expanding the vocabulary dictionary size from 32k to 62k to better support Traditional Chinese
86
+ - 8k-token context length
87
+ - Breeze-7B-Instruct-v0_1
88
+ - Expanding the vocabulary dictionary size from 32k to 62k to better support Traditional Chinese
89
+ - 8k-token context length
90
+ - Multi-turn dialogue (without special handling for harmfulness)
91
+ - Breeze-7B-Instruct-64k-v0_1
92
+ - Expanding the vocabulary dictionary size from 32k to 62k to better support Traditional Chinese
93
+ - 64k-token context length
94
+ - Multi-turn dialogue (without special handling for harmfulness)
95
+
96
+ ## Model Details
97
+
98
+ - Breeze-7B-Base-v0_1
99
+ - Finetuned from: [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)
100
+ - Model type: Causal decoder-only transformer language model
101
+ - Language: English and Traditional Chinese (zh-tw)
102
+ - Breeze-7B-Instruct-v0_1
103
+ - Finetuned from: [MediaTek-Research/Breeze-7B-Base-v0_1](https://huggingface.co/MediaTek-Research/Breeze-7B-Base-v0_1)
104
+ - Model type: Causal decoder-only transformer language model
105
+ - Language: English and Traditional Chinese (zh-tw)
106
+ - Breeze-7B-Instruct-64k-v0_1
107
+ - Finetuned from: [MediaTek-Research/Breeze-7B-Instruct-v0_1](https://huggingface.co/MediaTek-Research/Breeze-7B-Instruct-v0_1)
108
+ - Model type: Causal decoder-only transformer language model
109
+ - Language: English and Traditional Chinese (zh-tw)
110
+
111
+ ## Base Model Performance
112
+
113
+ **TMMLU+**, **DRCD**, and **Table** source from [MediaTek-Research/TCEval-v2](https://huggingface.co/datasets/MediaTek-Research/TCEval-v2).
114
+ [MediaTek-Research/TCEval-v2](https://huggingface.co/datasets/MediaTek-Research/TCEval-v2) derives from [TCEval-v1](https://github.com/mtkresearch/MR-Models/tree/main/TC-Eval)
115
+ and [ikala/tmmluplus](https://huggingface.co/datasets/ikala/tmmluplus). **MMLU** sources from [hails/mmlu_no_train](https://huggingface.co/datasets/hails/mmlu_no_train).
116
+ We use the code revised from [EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) to evaluate **TMMLU+**, **DRCD**, **Table**, and **MMLU**. All choice problems adapt the selection by the log-likelihood.
117
+
118
+
119
+ | Models | |↑ TMMLU+ (ACC) | DRCD (EM) | Table (ACC) | MMLU (ACC) |
120
+ |----------------------------------------------|--------|--------------|-------------|-------------|------------|
121
+ | | |TC, Knowledge |TC, Reasoning|TC, Reasoning|EN, Knowledge|
122
+ | | | 5 shot | 3 shot | 5 shot | 5 shot |
123
+ | [Yi-34B](https://huggingface.co/01-ai/Yi-34B)| 34B | 63.10 | 84.57 | 49.31 | 77.42 |
124
+ | [Qwen-14B](https://huggingface.co/01-ai/Qwen/Qwen-14B)| 14B | 51.30 | 16.95 * | 50.69 | 68.83 |
125
+ | [Yi-6B](https://huggingface.co/01-ai/Yi-6B) | 6B | 49.63 | 76.61 | 34.72 | 65.35 |
126
+ | [Qwen-7B](https://huggingface.co/01-ai/Qwen/Qwen-7B)| 7B | 42.84 | 0.0 * | 39.58 | 61.00 |
127
+ | [**Breeze-7B-Base-v0_1**](https://huggingface.co/MediaTek-Research/Breeze-7B-Base-v0_1) | 7B | 40.35 | 81.13 | 28.47 | 61.63 |
128
+ | [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)| 7B | 36.93 | 79.27 | 27.78 | 64.89 |
129
+
130
+
131
+ \* Few-shot learning cannot effectively guide the model to generate the proper answer.
132
+
133
+
134
+ ## Chat Model Performance
135
+
136
+ **TMMLU+**, **DRCD**, **Table**, and **MT-Bench-tw** source from [MediaTek-Research/TCEval-v2](https://huggingface.co/datasets/MediaTek-Research/TCEval-v2).
137
+ [MediaTek-Research/TCEval-v2](https://huggingface.co/datasets/MediaTek-Research/TCEval-v2) derives from [TCEval-v1](https://github.com/mtkresearch/MR-Models/tree/main/TC-Eval)
138
+ and [ikala/tmmluplus](https://huggingface.co/datasets/ikala/tmmluplus). **MMLU** sources from [hails/mmlu_no_train](https://huggingface.co/datasets/hails/mmlu_no_train).
139
+ **MT-Bench** source from [lmsys/mt_bench_human_judgments](https://huggingface.co/datasets/lmsys/mt_bench_human_judgments).
140
+ We use the code revised from [EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) to evaluate **TMMLU+**, **DRCD**, **Table**, and **MMLU**. All choice problems adapt the selection by the log-likelihood.
141
+ We use the code revised from [fastchat llm_judge](https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge) (GPT4 as judge) to evaluate **MT-Bench-tw** and **MT-Bench**.
142
+
143
+
144
+ | Models | |↑ MT-Bench-tw (Score)| TMMLU+ (ACC) | TMMLU+ (ACC) | DRCD (EM) | Table (ACC) | MT-Bench (Score) | MMLU (ACC) | MMLU (ACC) |
145
+ |---------------------------------------------------------------------------------------------------------|--------|--------------------|--------------|--------------|-------------|-------------|------------------|-------------|-------------|
146
+ | | |TC, Chat |TC, Knowledge |TC, Knowledge |TC, Reasoning|TC, Reasoning|EN, Chat |EN, Knowledge|EN, Knowledge|
147
+ | | |0 shot | 0 shot | 5 shot | 3 shot | 0 shot |0 shot | 0 shot | 5 shot |
148
+ | [gpt-3.5-turbo](https://openai.com) | |7.1 | 43.56 | | | 45.14 |7.9 | 67.09 | |
149
+ | [Yi-34B-Chat](https://huggingface.co/01-ai/Yi-34B-Chat) | 34B |6.9 | 54.87 | | | 36.81 |7.6 | 71.04 | |
150
+ | [Qwen-14B-Chat](https://huggingface.co/Qwen/Qwen-14B-Chat) | 14B |6.4 | 48.41 | | | 41.67 |7.2 | 64.91 | |
151
+ | [**Breeze-7B-Instruct-v0_1**](https://huggingface.co/MediaTek-Research/Breeze-7B-Instruct-v0_1) | 7B |5.7 | 41.61 | | | 45.83 |7.1 | 63.26 | |
152
+ | [**Breeze-7B-Instruct-64k-v0_1**](https://huggingface.co/MediaTek-Research/Breeze-7B-Instruct-64k-v0_1) | 7B |5.5 | 40.99 | | | 36.11 |7.1 | 63.68 | |
153
+ | [Qwen-7B-Chat](https://huggingface.co/Qwen/Qwen-7B-Chat) | 7B |5.4 | 40.02 | | | 33.33 |6.2 | 55.94 | |
154
+ | [Yi-6B-Chat](https://huggingface.co/01-ai/Yi-6B-Chat) | 6B |5.0 | 44.79 | | | 25.69 |6.0 | 59.45 | |
155
+ | [Taiwan-LLM-13B-v2.0-chat](https://huggingface.co/yentinglin/Taiwan-LLM-13B-v2.0-chat) | 13B |5.0 | 29.47 | | | 23.61 |-* | 50.50 | |
156
+ | [Taiwan-LLM-7B-v2.1-chat](https://huggingface.co/yentinglin/Taiwan-LLM-7B-v2.1-chat) | 7B |4.2 | 28.08 | | | 31.25 | -* | 42.72 | |
157
+
158
+ \* Taiwan-LLM models responds to multi-turn questions (English) in Traditional Chinese.
159
+
160
+
161
+ | Details on MT-Bench-tw (0 shot):<br/>Models | STEM |Extraction|Reasoning| Math | Coding | Roleplay| Writing |Humanities|↑ AVG |
162
+ |-----------------------------------------------------|---------|---------|---------|---------|---------|---------|---------|---------|---------|
163
+ | gpt-3.5-turbo | 7.8 | 6.1 | 5.1 | 6.4 | 6.2 | 8.7 | 7.4 | 9.3 | 7.1 |
164
+ | Yi-34B-Chat | 9.0 | 4.8 | 5.7 | 4.0 | 4.7 | 8.5 | 8.7 | 9.8 | 6.9 |
165
+ | Qwen-14B-Chat | 7.6 | 5.7 | 4.5 | 4.2 | 5.3 | 7.5 | 7.3 | 9.1 | 6.4 |
166
+ | **Breeze-7B-Instruct-v0_1** | 6.5 | 5.6 | 3.9 | 3.6 | 4.3 | 6.9 | 5.7 | 9.3 | 5.7 |
167
+ | **Breeze-7B-Instruct-64k-v0_1** | 6.1 | 5.3 | 3.7 | 2.9 | 4.2 | 7.0 | 6.7 | 8.3 | 5.5 |
168
+ | Qwen-7B-Chat | 6.6 | 4.5 | 4.8 | 2.9 | 3.6 | 6.2 | 6.8 | 8.2 | 5.4 |
169
+ | Yi-6B-Chat | 7.3 | 2.7 | 3.1 | 3.3 | 2.3 | 7.2 | 5.2 | 8.8 | 5.0 |
170
+ | Taiwan-LLM-13B-v2.0-chat | 6.1 | 3.4 | 4.1 | 2.3 | 3.1 | 7.4 | 6.6 | 6.8 | 5.0 |
171
+ | Taiwan-LLM-7B-v2.1-chat | 5.2 | 2.6 | 2.3 | 1.2 | 3.4 | 6.6 | 5.7 | 6.8 | 4.2 |
172
+
173
+
174
+ | Details on TMMLU+ (0 shot):<br/>Model | STEM | Social Science | Humanities | Other | ↑ AVG |
175
+ |-----------------------------------------------------|--------------|----------------|------------|------------|---------|
176
+ | Yi-34B-Chat | 47.65 | 64.25 | 52.73 | 54.91 | 54.87 |
177
+ | Qwen-14B-Chat | 43.83 | 55.00 | 48.55 | 46.22 | 48.41 |
178
+ | Yi-6B-Chat | 37.80 | 51.74 | 45.36 | 44.25 | 44.79 |
179
+ | gpt-3.5-turbo | 41.58 | 48.52 | 40.96 | 43.18 | 43.56 |
180
+ | **Breeze-7B-Instruct-v0_1** | 37.41 | 46.81 | 42.06 | 40.16 | 41.61 |
181
+ | **Breeze-7B-Instruct-64k-v0_1** | 37.88 | 46.35 | 40.31 | 39.40 | 40.99 |
182
+ | Qwen-7B-Chat | 35.44 | 46.22 | 38.35 | 40.06 | 40.02 |
183
+ | Taiwan-LLM-13B-v2.0-chat | 27.74 | 33.69 | 27.03 | 29.43 | 29.47 |
184
+ | Taiwan-LLM-7B-v2.1-chat | 25.58 | 31.76 | 27.36 | 27.61 | 28.08 |
185
+
186
+
187
+
188
+ ## Inference Performance
189
+ In this test, we use the first 700 characters of the [web article](https://health.udn.com/health/story/5976/7699252?from=udn_ch1005_main_index) as the input and ask the model to write the same article again.
190
+ All inferences run on 2 RTX A6000 GPUs (using `vllm`, with a tensor-parallel size of 2).
191
+
192
+ | Models | ↓ Inference Time (sec)|Estimated Max Input Length (Char)|
193
+ |--------------------------------------------------------------------|-------------------|--------------------------|
194
+ | Yi-6B-Chat | 10.62 | 5.2k |
195
+ | **Breeze-7B-Instruct-v0_1** | 10.74 | 11.1k |
196
+ | **Breeze-7B-Instruct-64k-v0_1** | 10.74 | 88.8k |
197
+ | Qwen-7B-Chat | 10.86 | 9.8k |
198
+ | Qwen-14B-Chat | 18.89 | 9.8k |
199
+ | Mistral-7B-v0.1-Instruct | 20.48 | 5.1k |
200
+ | Taiwan-LLM-7B-v2.1-chat | 26.26 | 2.2k |
201
+ | Taiwan-LLM-13B-v2.0-chat | 36.80 | 2.2k |
202
+ | Yi-34B-Chat | 43.71 | 4.5k |
203
+
204
+ ## Long-context Performance
205
+
206
+ TBD
207
+
208
+ ## Use in Transformers
209
+
210
+ First install direct dependencies:
211
+ ```
212
+ pip install transformers torch accelerate
213
+ ```
214
+ If you want faster inference using flash-attention2, you need to install these dependencies:
215
+ ```bash
216
+ pip install packaging ninja
217
+ pip install flash-attn
218
+ ```
219
+ Then load the model in transformers:
220
+ ```python
221
+ from transformers import AutoModelForCausalLM, AutoTokenizer
222
+ import torch
223
+
224
+ model = AutoModelForCausalLM.from_pretrained(
225
+ "MediaTek-Research/Breeze-7B-Instruct-v0_1",
226
+ device_map="auto",
227
+ torch_dtype=torch.bfloat16,
228
+ attn_implementation="flash_attention_2" # optional
229
+ )
230
+ ```
231
+
232
+ The structure of the query is
233
+ ```txt
234
+ <s>SYS_PROMPT [INST] QUERY1 [/INST] RESPONSE1 [INST] QUERY2 [/INST]
235
+ ```
236
+ where `SYS_PROMPT`, `QUERY1`, `RESPONSE1`, and `QUERY2` can be provided by the user.
237
+
238
+ The suggested default `SYS_PROMPT` is
239
+ ```txt
240
+ You are a helpful AI assistant built by MediaTek Research. The user you are helping speaks Traditional Chinese and comes from Taiwan.
241
+ ```
242
+
243
+ We also integrate `chat_template` into [tokenizer_config.json](tokenizer_config.json), so you can `apply_chat_template` to get the prompt.
244
+
245
+ ```python
246
+ >>> from transformers import AutoTokenizer
247
+ >>> tokenizer = AutoTokenizer.from_pretrained("MediaTek-Research/Breeze-7B-Instruct-v0_1")
248
+ >>> chat = [
249
+ ... {"role": "user", "content": "你好，請問你可以完成什麼任務？"},
250
+ ... {"role": "assistant", "content": "你好，我可以幫助您解決各種問題、提供資訊和協助您完成許多不同的任務。例如：回答技術問題、提供建議、翻譯文字、尋找資料或協助您安排行程等。請告訴我如何能幫助您。"},
251
+ ... {"role": "user", "content": "太棒了！"},
252
+ ... ]
253
+ >>> tokenizer.apply_chat_template(chat, tokenize=False)
254
+ "<s>You are a helpful AI assistant built by MediaTek Research. The user you are helping speaks Traditional Chinese and comes from Taiwan. [INST] 你好，請問你可以完成什麼任務？ [/INST] 你好，我可以幫助您解決各種問題、提供資訊和協助您完成許多不同的任務。例如：回答技術問題、提供建議、翻譯文字、尋找資料或協助您安排行程等。請告訴我如何能幫助您。 [INST] 太棒了！ [/INST] "
255
+ # Tokenized results
256
+ # ['▁', '你好', '，', '請問', '你', '可以', '完成', '什麼', '任務', '？']
257
+ # ['▁', '你好', '，', '我', '可以', '幫助', '您', '解決', '各種', '問題', '、', '提供', '資訊', '和', '協助', '您', '完成', '許多', '不同', '的', '任務', '。', '例如', '：', '回答', '技術', '問題', '、', '提供', '建議', '、', '翻譯', '文字', '、', '尋找', '資料', '或', '協助', '您', '安排', '行程', '等', '。', '請', '告訴', '我', '如何', '能', '幫助', '您', '。']
258
+ # ['▁', '太', '棒', '了', '！']
259
+ ```
260
+
261
+ ## Citation
262
+
263
+ ```
264
+ @article{MediaTek-Research2024breeze7b,
265
+ title={Breeze-7B Technical Report},
266
+ author={Chan-Jan Hsu and Chang-Le Liu and Feng-Ting Liao and Po-Chun Hsu and Yi-Chang Chen and Da-Shan Shiu},
267
+ year={2024},
268
+ eprint={2403.02712},
269
+ archivePrefix={arXiv},
270
+ primaryClass={cs.CL}
271
+ }
272
+ ```
273
+