library_name: transformers
tags:
- SkillTree
- llama2
license: llama2
SkillTree Model Collection
Applying a skill to your model with SkillTree is akin to unlocking a new ability in a video game's skill tree. Just as you would enhance your character's capabilities by selecting and activating specific skills, you can augment your model's abilities by integrating specialized skills. Follow these steps to imbue your model with new prowess, enhancing its performance and versatility in a straightforward and intuitive manner.
What is SkillTree?
SkillTree represents a set of model weights derived from further pre-training or fine-tuning Large Language Models (LLMs) to extract specific capabilities, such as code generation or chatting abilities. These extracted "skills" can be combined with a specific LLM base model to enhance its capabilities. The concept is inspired by ChatVector, aiming to modularize and transfer distinct skills across models.
SkillTree Details
- Functionality Status: Operational / Not Operational / Not Verified
- Base Model: meta-llama/Llama-2-7b-hf
- Skill Model: codellama/CodeLlama-7b-hf
- Enhanced Model(optional): HachiML/Swallow-7b-hf-CodeSkill
- Skill type: Coding
Uses
Limitation
- Model Architecture: Llama2
- Model Size: 6.74B
- Compatible Models[optional]:
How to Apply Skill (Example)
# Import Library
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
# Load the target model to be applied skill
tokenizer = AutoTokenizer.from_pretrained(
"tokyotech-llm/Swallow-7b-hf"
)
model = AutoModelForCausalLM.from_pretrained(
"tokyotech-llm/Swallow-7b-hf",
torch_dtype=torch.bfloat16,
device_map="auto",
)
# Load SkillTree
skill_tree = AutoModelForCausalLM.from_pretrained(
"HachiML/SkillTree-llama2-7b-hf-Code",
torch_dtype=torch.bfloat16,
device_map="auto",
)
# Apply the skill to the target model
def apply_skill(model, skill_tree):
# excluded object
skip_layers = ["model.embed_tokens.weight", "model.norm.weight", "lm_head.weight"]
# apply skill
for k, v in model.state_dict().items():
# layernorm is also excluded
if (k in skip_layers) or ("layernorm" in k):
continue
vector = skill_tree.state_dict()[k]
new_v = v + vector.to(v.device)
v.copy_(new_v)
return model
model = apply_skill(model, skill_tree)
# Push to hub
model_name = "HachiML/Swallow-7b-hf-CodeSkill"
tokenizer.save_pretrained(f"./models/{model_name}", repo_id=model_name, push_to_hub=True)
model.save_pretrained(f"./models/{model_name}", repo_id=model_name, push_to_hub=True)