File size: 3,499 Bytes
1eaf4ea 7b62db0 1eaf4ea 7b62db0 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 |
---
language:
- en
tags:
- llama
---
# OpenChat: Less is More for Open-source Models
OpenChat is a series of open-source language models fine-tuned on very little diverse and high-quality multi-round conversations. The [dataset](https://huggingface.co/datasets/openchat/openchat_sharegpt4_dataset) contains only ~6K GPT-4 conversations filtered from the 90K ShareGPT conversations.
Generic models:
- OpenChat: based on LLaMA-13B (2048 context length)
- **105.7%** of ChatGPT score on Vicuna GPT-4 evaluation
- **80.87%** Win-rate on AlpacaEval
- **🚀 Only used 6K data for finetuning!!!**
- OpenChat-8192: based on LLaMA-13B (extended to 8192 context length)
- **106.6%** of ChatGPT score on Vicuna GPT-4 evaluation
Code models:
- OpenCoderPlus: based on StarCoderPlus (native 8192 context length)
- **102.5%** of ChatGPT score on Vicuna GPT-4 evaluation
- **78.70%** Win-rate on AlpacaEval
**NOTE:** Please load the pretrained models using *bfloat16*
## Conversation Template
The conversation template **involves concatenating tokens**.
Besides base model vocabulary, an end-of-turn token `<|end_of_turn|>` is added, with id `eot_token_id`.
```python
# OpenChat
[bos_token_id] + tokenize("Human: ") + tokenize(user_question) + [eot_token_id] + tokenize("Assistant: ")
# OpenCoder
tokenize("User:") + tokenize(user_question) + [eot_token_id] + tokenize("Assistant:")
```
*Hint: In BPE, `tokenize(A) + tokenize(B)` does not always equals to `tokenize(A + B)`*
Following is the code for generating the conversation templates:
```python
@dataclass
class ModelConfig:
# Prompt
system: Optional[str]
role_prefix: dict
ai_role: str
eot_token: str
bos_token: Optional[str] = None
# Get template
def generate_conversation_template(self, tokenize_fn, tokenize_special_fn, message_list):
tokens = []
masks = []
# begin of sentence (bos)
if self.bos_token:
t = tokenize_special_fn(self.bos_token)
tokens.append(t)
masks.append(False)
# System
if self.system:
t = tokenize_fn(self.system) + [tokenize_special_fn(self.eot_token)]
tokens.extend(t)
masks.extend([False] * len(t))
# Messages
for idx, message in enumerate(message_list):
# Prefix
t = tokenize_fn(self.role_prefix[message["from"]])
tokens.extend(t)
masks.extend([False] * len(t))
# Message
if "value" in message:
t = tokenize_fn(message["value"]) + [tokenize_special_fn(self.eot_token)]
tokens.extend(t)
masks.extend([message["from"] == self.ai_role] * len(t))
else:
assert idx == len(message_list) - 1, "Empty message for completion must be on the last."
return tokens, masks
MODEL_CONFIG_MAP = {
# OpenChat / OpenChat-8192
"openchat": ModelConfig(
# Prompt
system=None,
role_prefix={
"human": "Human: ",
"gpt": "Assistant: "
},
ai_role="gpt",
eot_token="<|end_of_turn|>",
bos_token="<s>",
),
# OpenCoder / OpenCoderPlus
"opencoder": ModelConfig(
# Prompt
system=None,
role_prefix={
"human": "User:",
"gpt": "Assistant:"
},
ai_role="gpt",
eot_token="<|end_of_turn|>",
bos_token=None,
)
}
``` |