metadata
language:
- en
tags:
- llama
OpenChat: Less is More for Open-source Models
OpenChat is a series of open-source language models fine-tuned on very little diverse and high-quality multi-round conversations. The dataset contains only ~6K GPT-4 conversations filtered from the 90K ShareGPT conversations.
Generic models:
- OpenChat: based on LLaMA-13B (2048 context length)
- 105.7% of ChatGPT score on Vicuna GPT-4 evaluation
- 80.87% Win-rate on AlpacaEval
- 🚀 Only used 6K data for finetuning!!!
- OpenChat-8192: based on LLaMA-13B (extended to 8192 context length)
- 106.6% of ChatGPT score on Vicuna GPT-4 evaluation
Code models:
- OpenCoderPlus: based on StarCoderPlus (native 8192 context length)
- 102.5% of ChatGPT score on Vicuna GPT-4 evaluation
- 78.70% Win-rate on AlpacaEval
NOTE: Please load the pretrained models using bfloat16
Conversation Template
The conversation template involves concatenating tokens.
Besides base model vocabulary, an end-of-turn token <|end_of_turn|>
is added, with id eot_token_id
.
# OpenChat
[bos_token_id] + tokenize("Human: ") + tokenize(user_question) + [eot_token_id] + tokenize("Assistant: ")
# OpenCoder
tokenize("User:") + tokenize(user_question) + [eot_token_id] + tokenize("Assistant:")
Hint: In BPE, tokenize(A) + tokenize(B)
does not always equals to tokenize(A + B)
Following is the code for generating the conversation templates:
@dataclass
class ModelConfig:
# Prompt
system: Optional[str]
role_prefix: dict
ai_role: str
eot_token: str
bos_token: Optional[str] = None
# Get template
def generate_conversation_template(self, tokenize_fn, tokenize_special_fn, message_list):
tokens = []
masks = []
# begin of sentence (bos)
if self.bos_token:
t = tokenize_special_fn(self.bos_token)
tokens.append(t)
masks.append(False)
# System
if self.system:
t = tokenize_fn(self.system) + [tokenize_special_fn(self.eot_token)]
tokens.extend(t)
masks.extend([False] * len(t))
# Messages
for idx, message in enumerate(message_list):
# Prefix
t = tokenize_fn(self.role_prefix[message["from"]])
tokens.extend(t)
masks.extend([False] * len(t))
# Message
if "value" in message:
t = tokenize_fn(message["value"]) + [tokenize_special_fn(self.eot_token)]
tokens.extend(t)
masks.extend([message["from"] == self.ai_role] * len(t))
else:
assert idx == len(message_list) - 1, "Empty message for completion must be on the last."
return tokens, masks
MODEL_CONFIG_MAP = {
# OpenChat / OpenChat-8192
"openchat": ModelConfig(
# Prompt
system=None,
role_prefix={
"human": "Human: ",
"gpt": "Assistant: "
},
ai_role="gpt",
eot_token="<|end_of_turn|>",
bos_token="<s>",
),
# OpenCoder / OpenCoderPlus
"opencoder": ModelConfig(
# Prompt
system=None,
role_prefix={
"human": "User:",
"gpt": "Assistant:"
},
ai_role="gpt",
eot_token="<|end_of_turn|>",
bos_token=None,
)
}