LoneStriker commited on
Commit
3569cae
1 Parent(s): 85ab52e

Upload 10 files

Browse files
README.md CHANGED
@@ -6,6 +6,7 @@ tags:
6
  - C-RLFT
7
  datasets:
8
  - openchat/openchat_sharegpt4_dataset
 
9
  - imone/OpenOrca_FLAN
10
  - LDJnr/LessWrong-Amplify-Instruct
11
  - LDJnr/Pure-Dove
@@ -18,40 +19,67 @@ datasets:
18
  library_name: transformers
19
  pipeline_tag: text-generation
20
  ---
21
-
22
- # OpenChat (1210 Version): Advancing Open-source Language Models with Mixed-Quality Data
23
-
24
  <div align="center">
25
  <img src="https://raw.githubusercontent.com/imoneoi/openchat/master/assets/logo_new.png" style="width: 65%">
 
26
  </div>
27
 
28
- <p align="center">
29
- <a href="https://github.com/imoneoi/openchat">GitHub Repo</a> •
30
- <a href="https://openchat.team">Online Demo</a>
31
- <a href="https://discord.gg/pQjnXvNKHY">Discord</a>
32
- <a href="https://twitter.com/imonenext">Twitter</a>
33
- <a href="https://huggingface.co/openchat">Huggingface</a> •
34
- <a href="https://arxiv.org/pdf/2309.11235.pdf">Paper</a>
 
 
 
 
 
 
 
 
 
 
35
  </p>
36
 
37
- **🔥 **
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
38
 
39
- | Model | HumanEval+ |
40
- |-----------------------------|------------|
41
- | GPT-3.5 (December 2023) | 64.6 |
42
- | **OpenChat 3.5 1210** | **63.4** |
43
- | GPT-3.5 (March 2023) | 64.6 |
44
- | OpenHermes 2.5 | 41.5 |
45
 
46
- <div align="center" style="justify-content: center; align-items: center; "'>
47
- <img src="https://github.com/alpayariyak/openchat/blob/master/assets/3.5-benchmarks.png?raw=true" style="width: 100%; border-radius: 0.5em">
48
- </div>
49
 
50
- OpenChat is an innovative library of open-source language models, fine-tuned with [C-RLFT](https://arxiv.org/pdf/2309.11235.pdf) - a strategy inspired by offline reinforcement learning. Our models learn from mixed-quality data without preference labels, delivering exceptional performance on par with ChatGPT, even with a 7B model. Despite our simple approach, we are committed to developing a high-performance, commercially viable, open-source large language model, and we continue to make significant strides toward this vision.
 
 
 
 
 
 
51
 
52
- [![DOI](https://zenodo.org/badge/645397533.svg)](https://zenodo.org/badge/latestdoi/645397533)
53
 
54
- ## Usage
 
 
55
 
56
  To use this model, we highly recommend installing the OpenChat package by following the [installation guide](https://github.com/imoneoi/openchat#installation) in our repository and using the OpenChat OpenAI-compatible API server by running the serving command from the table below. The server is optimized for high-throughput deployment using [vLLM](https://github.com/vllm-project/vllm) and can run on a consumer GPU with 24GB RAM. To enable tensor parallelism, append `--tensor-parallel-size N` to the serving command.
57
 
@@ -59,10 +87,14 @@ Once started, the server listens at `localhost:18888` for requests and is compat
59
 
60
  If you want to deploy the server as an online service, you can use `--api-keys sk-KEY1 sk-KEY2 ...` to specify allowed API keys and `--disable-log-requests --disable-log-stats --log-file openchat.log` for logging only to a file. For security purposes, we recommend using an [HTTPS gateway](https://fastapi.tiangolo.com/es/deployment/concepts/#security-https) in front of the server.
61
 
 
 
 
 
62
  <details>
63
  <summary>Example request (click to expand)</summary>
64
 
65
- Default Mode (Chat & Coding)
66
 
67
  ```bash
68
  curl http://localhost:18888/v1/chat/completions \
@@ -73,7 +105,7 @@ curl http://localhost:18888/v1/chat/completions \
73
  }'
74
  ```
75
 
76
- Mathematical Reasoning Mode
77
 
78
  ```bash
79
  curl http://localhost:18888/v1/chat/completions \
@@ -87,25 +119,23 @@ curl http://localhost:18888/v1/chat/completions \
87
 
88
  </details>
89
 
90
- | Model | Size | Context | Weights | Serving |
91
- |-------------------|------|---------|------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------|
92
- | OpenChat 3.5 1210 | 7B | 8192 | [Huggingface](https://huggingface.co/openchat/openchat_3.5_1210) | `python -m ochat.serving.openai_api_server --model openchat/openchat_3.5_1210 --engine-use-ray --worker-use-ray` |
93
-
94
  ### Conversation templates
95
 
96
- Default Mode (GPT4 Correct)
97
 
98
  ```
99
  GPT4 Correct User: Hello<|end_of_turn|>GPT4 Correct Assistant: Hi<|end_of_turn|>GPT4 Correct User: How are you today?<|end_of_turn|>GPT4 Correct Assistant:
100
  ```
101
 
102
- Mathematical Reasoning Mode
103
 
104
  ```
105
  Math Correct User: 10.3 − 7988.8133=<|end_of_turn|>Math Correct Assistant:
106
  ```
107
 
108
- The default (GPT4 Correct) template is also available as the integrated `tokenizer.chat_template`,
 
 
109
  which can be used instead of manually specifying the template:
110
 
111
  ```python
@@ -118,16 +148,41 @@ tokens = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
118
  assert tokens == [1, 420, 6316, 28781, 3198, 3123, 1247, 28747, 22557, 32000, 420, 6316, 28781, 3198, 3123, 21631, 28747, 15359, 32000, 420, 6316, 28781, 3198, 3123, 1247, 28747, 1602, 460, 368, 3154, 28804, 32000, 420, 6316, 28781, 3198, 3123, 21631, 28747]
119
  ```
120
 
121
- ## Comparison with [X.AI Grok models](https://x.ai/)
122
-
123
- | | License | # Param | Average | MMLU | HumanEval | MATH | GSM8k |
124
- |-------------------|-------------|---------|----------|------|-----------|----------|----------|
125
- | OpenChat 3.5 1210 | Apache-2.0 | **7B** | **60.1** | 65.3 | **68.9** | **28.9** | **77.3** |
126
- | OpenChat 3.5 | Apache-2.0 | **7B** | 56.4 | 64.3 | 55.5 | 28.6 | **77.3** |
127
- | Grok-0 | Proprietary | 33B | 44.5 | 65.7 | 39.7 | 15.7 | 56.8 |
128
- | Grok-1 | Proprietary | ???B | 55.8 | 73 | 63.2 | 23.9 | 62.9 |
129
 
130
- ## <a id="benchmarks"></a> Benchmarks
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
131
 
132
  | Model | # Params | Average | MT-Bench | HumanEval | BBH MC | AGIEval | TruthfulQA | MMLU | GSM8K | BBH CoT |
133
  |--------------------|----------|----------|--------------|-----------------|----------|----------|---------------|--------------|--------------|-------------|
@@ -139,9 +194,9 @@ assert tokens == [1, 420, 6316, 28781, 3198, 3123, 1247, 28747, 22557, 32000, 42
139
  | OpenOrca Mistral | 7B | 52.7 | 6.86 | 38.4 | 49.4 | 42.9 | 45.9 | 59.3 | 59.1 | 58.1 |
140
  | Zephyr-β^ | 7B | 34.6 | 7.34 | 22.0 | 40.6 | 39.0 | 40.8 | 39.8 | 5.1 | 16.0 |
141
  | Mistral | 7B | - | 6.84 | 30.5 | 39.0 | 38.0 | - | 60.1 | 52.2 | - |
142
- | Open-source SOTA** | 13B-70B | 61.4 | 7.71 | 73.2 | 49.7 | 41.7 | 62.3 | 63.7 | 82.3 | 41.4 |
143
- | | | | WizardLM 70B | WizardCoder 34B | Orca 13B | Orca 13B | Platypus2 70B | WizardLM 70B | MetaMath 70B | Flan-T5 11B |
144
 
 
 
145
  *: ChatGPT (March) results are from [GPT-4 Technical Report](https://arxiv.org/abs/2303.08774), [Chain-of-Thought Hub](https://github.com/FranxYao/chain-of-thought-hub), and our evaluation. Please note that ChatGPT is not a fixed baseline and evolves rapidly over time.
146
 
147
  ^: Zephyr-β often fails to follow few-shot CoT instructions, likely because it was aligned with only chat data but not trained on few-shot data.
@@ -149,15 +204,65 @@ assert tokens == [1, 420, 6316, 28781, 3198, 3123, 1247, 28747, 22557, 32000, 42
149
  **: Mistral and Open-source SOTA results are taken from reported results in instruction-tuned model papers and official repositories.
150
 
151
  All models are evaluated in chat mode (e.g. with the respective conversation template applied). All zero-shot benchmarks follow the same setting as in the AGIEval paper and Orca paper. CoT tasks use the same configuration as Chain-of-Thought Hub, HumanEval is evaluated with EvalPlus, and MT-bench is run using FastChat. To reproduce our results, follow the instructions in [our repository](https://github.com/imoneoi/openchat/#benchmarks).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
152
 
153
- ## Limitations
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
154
 
155
  **Foundation Model Limitations**
156
  Despite its advanced capabilities, OpenChat is still bound by the limitations inherent in its foundation models. These limitations may impact the model's performance in areas such as:
157
 
158
- - Complex reasoning
159
- - Mathematical and arithmetic tasks
160
- - Programming and coding challenges
161
 
162
  **Hallucination of Non-existent Information**
163
  OpenChat may sometimes generate information that does not exist or is not accurate, also known as "hallucination". Users should be aware of this possibility and verify any critical information obtained from the model.
@@ -165,24 +270,31 @@ OpenChat may sometimes generate information that does not exist or is not accura
165
  **Safety**
166
  OpenChat may sometimes generate harmful, hate speech, biased responses, or answer unsafe questions. It's crucial to apply additional AI safety measures in use cases that require safe and moderated responses.
167
 
168
- ## License
 
 
169
 
170
  Our OpenChat 3.5 code and models are distributed under the Apache License 2.0.
171
 
172
- ## Dataset Details
 
 
173
 
174
  OpenChat 3.5 was trained with C-RLFT on a collection of publicly available high-quality instruction data, with a custom processing pipeline. We detail some notable subsets included here:
175
 
176
- - [OpenChat ShareGPT](https://huggingface.co/datasets/openchat/openchat_sharegpt4_dataset)
177
- - [Open-Orca with FLAN answers](https://huggingface.co/datasets/imone/OpenOrca_FLAN)
178
- - Capybara [1](https://huggingface.co/datasets/LDJnr/Pure-Dove) [2](https://huggingface.co/datasets/LDJnr/Verified-Camel) [3](https://huggingface.co/datasets/LDJnr/LessWrong-Amplify-Instruct)
179
- - [GOAT](https://huggingface.co/datasets/tiedong/goat)
180
- - [Glaive](https://huggingface.co/datasets/glaiveai/glaive-code-assistant)
181
- - [MetaMathQA](https://huggingface.co/datasets/meta-math/MetaMathQA)
182
- - [MathInstruct](https://huggingface.co/datasets/TIGER-Lab/MathInstruct)
183
- - [OpenAssistant](https://huggingface.co/datasets/OpenAssistant/oasst_top1_2023-08-25)
 
184
 
185
- ## Citation
 
 
186
 
187
  ```
188
  @article{wang2023openchat,
@@ -193,10 +305,12 @@ OpenChat 3.5 was trained with C-RLFT on a collection of publicly available high-
193
  }
194
  ```
195
 
196
- ## Acknowledgements
 
 
197
 
198
  We extend our heartfelt gratitude to AutoMeta and caesus from Alignment Lab AI, LDJ and Teknium from Nous Research, alpin and TearGosling from Pygmalion AI for their substantial contributions to data collection and model training.
199
 
200
  Special thanks go to Changling Liu from GPT Desk Pte. Ltd., Qiying Yu at Tsinghua University, Baochang Ma, and Hao Wan from 01.AI company for their generous provision of resources. We are also deeply grateful to Jianxiong Li and Peng Li at Tsinghua University for their insightful discussions.
201
 
202
- Furthermore, we appreciate the developers behind the following projects for their significant contributions to our research: [Mistral](https://mistral.ai/), [Chain-of-Thought Hub](https://github.com/FranxYao/chain-of-thought-hub), [Llama 2](https://ai.meta.com/llama/), [Self-Instruct](https://arxiv.org/abs/2212.10560), [FastChat (Vicuna)](https://github.com/lm-sys/FastChat), [Alpaca](https://github.com/tatsu-lab/stanford_alpaca.git), and [StarCoder](https://github.com/bigcode-project/starcoder). Their work has been instrumental in driving our research forward.
 
6
  - C-RLFT
7
  datasets:
8
  - openchat/openchat_sharegpt4_dataset
9
+ - kaist-ai/Feedback-Collection
10
  - imone/OpenOrca_FLAN
11
  - LDJnr/LessWrong-Amplify-Instruct
12
  - LDJnr/Pure-Dove
 
19
  library_name: transformers
20
  pipeline_tag: text-generation
21
  ---
 
 
 
22
  <div align="center">
23
  <img src="https://raw.githubusercontent.com/imoneoi/openchat/master/assets/logo_new.png" style="width: 65%">
24
+ <h1>Advancing Open-source Language Models with Mixed-Quality Data</h1>
25
  </div>
26
 
27
+ <p align="center" style="margin-top: 0px;">
28
+ <a href="https://openchat.team">
29
+ <img src="https://github.com/alpayariyak/openchat/blob/master/assets/logo_nobg.png?raw=true" alt="OpenChat Logo" style="width:20px; vertical-align: middle; display: inline-block; margin-right: 5px; margin-left: 10px; margin-top: 0px; margin-bottom: 0px;"/>
30
+ <span class="link-text" style=" margin-right: 5px;">Online Demo</span>
31
+ </a> |
32
+ <a href="https://github.com/imoneoi/openchat">
33
+ <img src="https://camo.githubusercontent.com/4133dc1cd4511d4a292b84ce10e52e4ed92569fb2a8165381c9c47be5edc2796/68747470733a2f2f6564656e742e6769746875622e696f2f537570657254696e7949636f6e732f696d616765732f706e672f6769746875622e706e67" alt="GitHub Logo" style="width:20px; vertical-align: middle; display: inline-block; margin-right: 5px; margin-left: 5px; margin-top: 0px; margin-bottom: 0px;"/>
34
+ <span class="link-text" style=" margin-right: 5px;">GitHub</span>
35
+ </a> |
36
+ <a href="https://arxiv.org/pdf/2309.11235.pdf">
37
+ <img src="https://github.com/alpayariyak/openchat/blob/master/assets/arxiv-logomark-small-square-border.png?raw=true" alt="ArXiv Logo" style="width:20px; vertical-align: middle; display: inline-block; margin-right: 5px; margin-left: 5px; margin-top: 0px; margin-bottom: 0px;"/>
38
+ <span class="link-text" style="margin-right: 5px;">Paper</span>
39
+ </a> |
40
+ <a href="https://discord.gg/pQjnXvNKHY">
41
+ <img src="https://cloud.githubusercontent.com/assets/6291467/26705903/96c2d66e-477c-11e7-9f4e-f3c0efe96c9a.png" alt="Discord Logo" style="width:20px; vertical-align: middle; display: inline-block; margin-right: 5px; margin-left: 5px; margin-top: 0px; margin-bottom: 0px;"/>
42
+ <span class="link-text">Discord</span>
43
+ </a>
44
  </p>
45
 
46
+ <hr>
47
+ <div style="background-color: white; padding: 0.7em; border-radius: 0.5em; color: black; display: flex; flex-direction: column; justify-content: center; text-align: center; ont-size: 0.5em;">
48
+ <a href="https://huggingface.co/openchat/openchat_3.5" style="text-decoration: none; color: black;">
49
+ <span style="font-size: 1.7em; font-family: 'Helvetica'; letter-spacing: 0.1em; font-weight: bold; color: black;">OPENCHAT</span><span style="font-size: 1.8em; font-family: 'Helvetica'; color: #3c72db; ">3.5</span>
50
+ <span style="font-size: 0.7em; font-family: 'Helvetica'; color: white; vertical-align: top; background-color:red; border-radius: 6em; padding: 0.066em 0.4em; letter-spacing: 0.1em; font-weight: bold;">1210</span>
51
+ <span style="font-size: 0.85em; font-family: 'Helvetica'; color: black;">
52
+ <br> 🏆 The Overall Best Performing Open Source 7B Model 🏆
53
+ <br> 🤖 Outperforms <span style="font-weight: bold;">ChatGPT</span> (March) and <span style="font-weight: bold;">Grok-1</span> 🤖
54
+ <br> 🚀<span style="font-size: 1em; font-family: 'Helvetica'; color: black; font-weight: bold;">15</span>-point improvement in Coding over <span style="font-size: 0.9em;
55
+ font-family: 'Helvetica'; color: black; font-weight: bold;">OpenChat-3.5🚀</span>
56
+ <br><br><span style="font-size: 1em; font-family: 'Helvetica'; color: #3c72db; font-weight: bold;">New Features</span>
57
+ <br> 💡 2 Modes: Coding + Generalist, Mathematical Reasoning 💡
58
+ <br> 🧑‍⚖️ Experimental support for Evaluator and Feedback capabilities 🧑‍⚖️
59
+ </span>
60
+ </a>
61
+ </div>
62
 
63
+ <div style="display: flex; justify-content: center; align-items: center">
64
+ <img src="https://github.com/alpayariyak/openchat/blob/master/assets/1210bench.png?raw=true" style="width: 100%; border-radius: 1em">
65
+ </div>
 
 
 
66
 
67
+ <div>
68
+ <h3> Table of Contents</h3>
69
+ </div>
70
 
71
+ 1. [Usage](#usage)
72
+ 2. [Benchmarks](#benchmarks)
73
+ 3. [Limitations](#limitations)
74
+ 4. [License](#license)
75
+ 5. [Dataset Details](#dataset-details)
76
+ 6. [Citation](#citation)
77
+ 7. [Acknowledgements](#acknowledgements)
78
 
 
79
 
80
+ <div align="center">
81
+ <h2> Usage </h2>
82
+ </div>
83
 
84
  To use this model, we highly recommend installing the OpenChat package by following the [installation guide](https://github.com/imoneoi/openchat#installation) in our repository and using the OpenChat OpenAI-compatible API server by running the serving command from the table below. The server is optimized for high-throughput deployment using [vLLM](https://github.com/vllm-project/vllm) and can run on a consumer GPU with 24GB RAM. To enable tensor parallelism, append `--tensor-parallel-size N` to the serving command.
85
 
 
87
 
88
  If you want to deploy the server as an online service, you can use `--api-keys sk-KEY1 sk-KEY2 ...` to specify allowed API keys and `--disable-log-requests --disable-log-stats --log-file openchat.log` for logging only to a file. For security purposes, we recommend using an [HTTPS gateway](https://fastapi.tiangolo.com/es/deployment/concepts/#security-https) in front of the server.
89
 
90
+ | Model | Size | Context | Weights | Serving |
91
+ |-------------------|------|---------|------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------|
92
+ | OpenChat 3.5 1210 | 7B | 8192 | [Huggingface](https://huggingface.co/openchat/openchat_3.5_1210) | `python -m ochat.serving.openai_api_server --model openchat/openchat_3.5_1210 --engine-use-ray --worker-use-ray` |
93
+
94
  <details>
95
  <summary>Example request (click to expand)</summary>
96
 
97
+ 💡 **Default Mode (GPT4 Correct)**: Best for coding, chat and general tasks
98
 
99
  ```bash
100
  curl http://localhost:18888/v1/chat/completions \
 
105
  }'
106
  ```
107
 
108
+ 🧮 **Mathematical Reasoning Mode**: Tailored for solving math problems
109
 
110
  ```bash
111
  curl http://localhost:18888/v1/chat/completions \
 
119
 
120
  </details>
121
 
 
 
 
 
122
  ### Conversation templates
123
 
124
+ 💡 **Default Mode (GPT4 Correct)**: Best for coding, chat and general tasks
125
 
126
  ```
127
  GPT4 Correct User: Hello<|end_of_turn|>GPT4 Correct Assistant: Hi<|end_of_turn|>GPT4 Correct User: How are you today?<|end_of_turn|>GPT4 Correct Assistant:
128
  ```
129
 
130
+ 🧮 **Mathematical Reasoning Mode**: Tailored for solving math problems
131
 
132
  ```
133
  Math Correct User: 10.3 − 7988.8133=<|end_of_turn|>Math Correct Assistant:
134
  ```
135
 
136
+ ⚠️ **Notice:** Remember to set `<|end_of_turn|>` as end of generation token.
137
+
138
+ The default (GPT4 Correct) template is also available as the integrated `tokenizer.chat_template`,
139
  which can be used instead of manually specifying the template:
140
 
141
  ```python
 
148
  assert tokens == [1, 420, 6316, 28781, 3198, 3123, 1247, 28747, 22557, 32000, 420, 6316, 28781, 3198, 3123, 21631, 28747, 15359, 32000, 420, 6316, 28781, 3198, 3123, 1247, 28747, 1602, 460, 368, 3154, 28804, 32000, 420, 6316, 28781, 3198, 3123, 21631, 28747]
149
  ```
150
 
151
+ <div align="center">
152
+ <h2> (Experimental) Evaluator / Feedback Capabilities </h2>
153
+ </div>
154
+ We've included evaluator capabilities in this release to advance open-source models as evaluators. You can use `Default Mode (GPT4 Correct)` with the following prompt (same as [Prometheus](https://huggingface.co/datasets/kaist-ai/Feedback-Collection)) to evaluate a response.
 
 
 
 
155
 
156
+ ```
157
+ ###Task Description:
158
+ An instruction (might include an Input inside it), a response to evaluate, a reference answer that gets a score of 5, and a score rubric representing a evaluation criteria are given.
159
+ 1. Write a detailed feedback that assess the quality of the response strictly based on the given score rubric, not evaluating in general.
160
+ 2. After writing a feedback, write a score that is an integer between 1 and 5. You should refer to the score rubric.
161
+ 3. The output format should look as follows: "Feedback: (write a feedback for criteria) [RESULT] (an integer number between 1 and 5)"
162
+ 4. Please do not generate any other opening, closing, and explanations.
163
+
164
+ ###The instruction to evaluate:
165
+ {orig_instruction}
166
+
167
+ ###Response to evaluate:
168
+ {orig_response}
169
+
170
+ ###Reference Answer (Score 5):
171
+ {orig_reference_answer}
172
+
173
+ ###Score Rubrics:
174
+ [{orig_criteria}]
175
+ Score 1: {orig_score1_description}
176
+ Score 2: {orig_score2_description}
177
+ Score 3: {orig_score3_description}
178
+ Score 4: {orig_score4_description}
179
+ Score 5: {orig_score5_description}
180
+
181
+ ###Feedback:
182
+ ```
183
+ <div align="center">
184
+ <h2> Benchmarks </h2>
185
+ </div>
186
 
187
  | Model | # Params | Average | MT-Bench | HumanEval | BBH MC | AGIEval | TruthfulQA | MMLU | GSM8K | BBH CoT |
188
  |--------------------|----------|----------|--------------|-----------------|----------|----------|---------------|--------------|--------------|-------------|
 
194
  | OpenOrca Mistral | 7B | 52.7 | 6.86 | 38.4 | 49.4 | 42.9 | 45.9 | 59.3 | 59.1 | 58.1 |
195
  | Zephyr-β^ | 7B | 34.6 | 7.34 | 22.0 | 40.6 | 39.0 | 40.8 | 39.8 | 5.1 | 16.0 |
196
  | Mistral | 7B | - | 6.84 | 30.5 | 39.0 | 38.0 | - | 60.1 | 52.2 | - |
 
 
197
 
198
+ <details>
199
+ <summary>Evaluation Details(click to expand)</summary>
200
  *: ChatGPT (March) results are from [GPT-4 Technical Report](https://arxiv.org/abs/2303.08774), [Chain-of-Thought Hub](https://github.com/FranxYao/chain-of-thought-hub), and our evaluation. Please note that ChatGPT is not a fixed baseline and evolves rapidly over time.
201
 
202
  ^: Zephyr-β often fails to follow few-shot CoT instructions, likely because it was aligned with only chat data but not trained on few-shot data.
 
204
  **: Mistral and Open-source SOTA results are taken from reported results in instruction-tuned model papers and official repositories.
205
 
206
  All models are evaluated in chat mode (e.g. with the respective conversation template applied). All zero-shot benchmarks follow the same setting as in the AGIEval paper and Orca paper. CoT tasks use the same configuration as Chain-of-Thought Hub, HumanEval is evaluated with EvalPlus, and MT-bench is run using FastChat. To reproduce our results, follow the instructions in [our repository](https://github.com/imoneoi/openchat/#benchmarks).
207
+ </details>
208
+ <div>
209
+ <h3>HumanEval+</h3>
210
+ </div>
211
+
212
+ | Model | Size | HumanEval+ pass@1 |
213
+ |-----------------------------|----------|------------|
214
+ | ChatGPT (December 12, 2023) | - | 64.6 |
215
+ | WizardCoder-Python-34B-V1.0 | 34B | 64.6 |
216
+ | **OpenChat 3.5 (Dec 10)** | **7B** | **63.4** |
217
+ | OpenHermes 2.5 | 7B | 41.5 |
218
+
219
+ <div>
220
+ <h3>OpenChat-3.5-1210 vs. Grok</h3>
221
+ </div>
222
+
223
+ | | License | # Param | Average | MMLU | HumanEval | MATH | GSM8k |
224
+ |-------------------|-------------|---------|----------|------|-----------|----------|----------|
225
+ | OpenChat 3.5 1210 | Apache-2.0 | **7B** | **60.1** | 65.3 | **68.9** | **28.9** | **77.3** |
226
+ | OpenChat 3.5 | Apache-2.0 | **7B** | 56.4 | 64.3 | 55.5 | 28.6 | **77.3** |
227
+ | Grok-0 | Proprietary | 33B | 44.5 | 65.7 | 39.7 | 15.7 | 56.8 |
228
+ | Grok-1 | Proprietary | ???B | 55.8 | 73 | 63.2 | 23.9 | 62.9 |
229
+
230
+ *: Grok results are reported by [X.AI](https://x.ai/).
231
+
232
+ <div align="center">
233
+ <h2> 中文评估结果 / Chinese Evaluations </h2>
234
+ </div>
235
+
236
+ ⚠️ Note that this model was not explicitly trained in Chinese (only < 0.1% of the data is in Chinese). 请注意本模型没有针对性训练中文(中文数据占比小于0.1%)。
237
 
238
+ <div>
239
+ <h3>Multi-Level Multi-Discipline Chinese Evaluation Suite (CEVAL)</h3>
240
+ <div>
241
+
242
+ | Model | Avg | STEM | Social Science | Humanities | Others |
243
+ |----------|-------|-------|----------------|------------|--------|
244
+ | ChatGPT | 54.4 | 52.9 | 61.8 | 50.9 | 53.6 |
245
+ | OpenChat | 47.29 | 45.22 | 52.49 | 48.52 | 45.08 |
246
+
247
+ <div>
248
+ <h3>Massive Multitask Language Understanding in Chinese (CMMLU, 5-shot)</h3>
249
+ </div>
250
+
251
+ | Models | STEM | Humanities | SocialSciences | Other | ChinaSpecific | Avg |
252
+ |----------|-------|------------|----------------|-------|---------------|-------|
253
+ | ChatGPT | 47.81 | 55.68 | 56.5 | 62.66 | 50.69 | 55.51 |
254
+ | OpenChat | 38.7 | 45.99 | 48.32 | 50.23 | 43.27 | 45.85 |
255
+
256
+ <div align="center">
257
+ <h2> Limitations </h2>
258
+ </div>
259
 
260
  **Foundation Model Limitations**
261
  Despite its advanced capabilities, OpenChat is still bound by the limitations inherent in its foundation models. These limitations may impact the model's performance in areas such as:
262
 
263
+ - Complex reasoning
264
+ - Mathematical and arithmetic tasks
265
+ - Programming and coding challenges
266
 
267
  **Hallucination of Non-existent Information**
268
  OpenChat may sometimes generate information that does not exist or is not accurate, also known as "hallucination". Users should be aware of this possibility and verify any critical information obtained from the model.
 
270
  **Safety**
271
  OpenChat may sometimes generate harmful, hate speech, biased responses, or answer unsafe questions. It's crucial to apply additional AI safety measures in use cases that require safe and moderated responses.
272
 
273
+ <div align="center">
274
+ <h2> License </h2>
275
+ </div>
276
 
277
  Our OpenChat 3.5 code and models are distributed under the Apache License 2.0.
278
 
279
+ <div align="center">
280
+ <h2> Dataset Details </h2>
281
+ </div>
282
 
283
  OpenChat 3.5 was trained with C-RLFT on a collection of publicly available high-quality instruction data, with a custom processing pipeline. We detail some notable subsets included here:
284
 
285
+ - [OpenChat ShareGPT](https://huggingface.co/datasets/openchat/openchat_sharegpt4_dataset)
286
+ - [Open-Orca with FLAN answers](https://huggingface.co/datasets/imone/OpenOrca_FLAN)
287
+ - [Feedback-Collection](https://huggingface.co/datasets/kaist-ai/Feedback-Collection)
288
+ - Capybara [1](https://huggingface.co/datasets/LDJnr/Pure-Dove) [2](https://huggingface.co/datasets/LDJnr/Verified-Camel) [3](https://huggingface.co/datasets/LDJnr/LessWrong-Amplify-Instruct)
289
+ - [GOAT](https://huggingface.co/datasets/tiedong/goat)
290
+ - [Glaive](https://huggingface.co/datasets/glaiveai/glaive-code-assistant)
291
+ - [MetaMathQA](https://huggingface.co/datasets/meta-math/MetaMathQA)
292
+ - [MathInstruct](https://huggingface.co/datasets/TIGER-Lab/MathInstruct)
293
+ - [OpenAssistant](https://huggingface.co/datasets/OpenAssistant/oasst_top1_2023-08-25)
294
 
295
+ <div align="center">
296
+ <h2> Citation </h2>
297
+ </div>
298
 
299
  ```
300
  @article{wang2023openchat,
 
305
  }
306
  ```
307
 
308
+ <div align="center">
309
+ <h2> Acknowledgments </h2>
310
+ </div>
311
 
312
  We extend our heartfelt gratitude to AutoMeta and caesus from Alignment Lab AI, LDJ and Teknium from Nous Research, alpin and TearGosling from Pygmalion AI for their substantial contributions to data collection and model training.
313
 
314
  Special thanks go to Changling Liu from GPT Desk Pte. Ltd., Qiying Yu at Tsinghua University, Baochang Ma, and Hao Wan from 01.AI company for their generous provision of resources. We are also deeply grateful to Jianxiong Li and Peng Li at Tsinghua University for their insightful discussions.
315
 
316
+ Furthermore, we appreciate the developers behind the following projects for their significant contributions to our research: [Mistral](https://mistral.ai/), [Chain-of-Thought Hub](https://github.com/FranxYao/chain-of-thought-hub), [Llama 2](https://ai.meta.com/llama/), [Self-Instruct](https://arxiv.org/abs/2212.10560), [FastChat (Vicuna)](https://github.com/lm-sys/FastChat), [Alpaca](https://github.com/tatsu-lab/stanford_alpaca.git), and [StarCoder](https://github.com/bigcode-project/starcoder). Their work has been instrumental in driving our research forward.
config.json CHANGED
@@ -4,7 +4,7 @@
4
  "MistralForCausalLM"
5
  ],
6
  "bos_token_id": 1,
7
- "eos_token_id": 2,
8
  "hidden_act": "silu",
9
  "hidden_size": 4096,
10
  "initializer_range": 0.02,
 
4
  "MistralForCausalLM"
5
  ],
6
  "bos_token_id": 1,
7
+ "eos_token_id": 32000,
8
  "hidden_act": "silu",
9
  "hidden_size": 4096,
10
  "initializer_range": 0.02,
generation_config.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 1,
4
+ "eos_token_id": 32000,
5
+ "max_length": 8192,
6
+ "pad_token_id": 0,
7
+ "temperature": 0.5,
8
+ "transformers_version": "4.35.2"
9
+ }
huggingface-metadata.txt ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ url: https://huggingface.co/openchat/openchat-3.5-1210
2
+ branch: main
3
+ download date: 2023-12-18 02:14:41
4
+ sha256sum:
5
+ c65531d6a5a9bb8345aa117528f3f4105b0bf357eab647519c19b4dca952e89b model-00001-of-00003.safetensors
6
+ 26b18d5d207afd2257427cfa2a7f5f041bea575b77113855a25faae88d82c408 model-00002-of-00003.safetensors
7
+ 3ff9fa33b10323cb7604dfa9da1d0fac1af5f2aa485744031b5f43d0d8885563 model-00003-of-00003.safetensors
8
+ dadfd56d766715c61d2ef780a525ab43b8e6da4de6865bda3d95fdef5e134055 tokenizer.model
special_tokens_map.json CHANGED
@@ -11,7 +11,7 @@
11
  "single_word": false
12
  },
13
  "eos_token": {
14
- "content": "</s>",
15
  "lstrip": false,
16
  "normalized": false,
17
  "rstrip": false,
 
11
  "single_word": false
12
  },
13
  "eos_token": {
14
+ "content": "<|end_of_turn|>",
15
  "lstrip": false,
16
  "normalized": false,
17
  "rstrip": false,
tokenizer_config.json CHANGED
@@ -48,8 +48,9 @@
48
  "<|pad_0|>"
49
  ],
50
  "bos_token": "<s>",
 
51
  "clean_up_tokenization_spaces": false,
52
- "eos_token": "</s>",
53
  "legacy": true,
54
  "model_max_length": 1000000000000000019884624838656,
55
  "pad_token": null,
 
48
  "<|pad_0|>"
49
  ],
50
  "bos_token": "<s>",
51
+ "chat_template": "{{ bos_token }}{% for message in messages %}{{ 'GPT4 Correct ' + message['role'].title() + ': ' + message['content'] + '<|end_of_turn|>'}}{% endfor %}{% if add_generation_prompt %}{{ 'GPT4 Correct Assistant:' }}{% endif %}",
52
  "clean_up_tokenization_spaces": false,
53
+ "eos_token": "<|end_of_turn|>",
54
  "legacy": true,
55
  "model_max_length": 1000000000000000019884624838656,
56
  "pad_token": null,