README.md

---
library_name: transformers
tags:
- SkillTree
- llama2
license: llama2
---

# SkillTree Model Collection

Applying a skill to your model with SkillTree is akin to unlocking a new ability in a video game's skill tree. Just as you would enhance your character's capabilities by selecting and activating specific skills, you can augment your model's abilities by integrating specialized skills. Follow these steps to imbue your model with new prowess, enhancing its performance and versatility in a straightforward and intuitive manner.

## What is SkillTree?

SkillTree represents a set of model weights derived from further pre-training or fine-tuning Large Language Models (LLMs) to extract specific capabilities, such as code generation or chatting abilities. These extracted "skills" can be combined with a specific LLM base model to enhance its capabilities. The concept is inspired by [ChatVector](https://arxiv.org/abs/2310.04799), aiming to modularize and transfer distinct skills across models.

## SkillTree Details

- **Functionality Status:** Operational / **Not Operational** / Not Verified
- **Base Model:** [meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf)
- **Skill Model:** [codellama/CodeLlama-7b-hf](https://huggingface.co/codellama/CodeLlama-7b-hf)
- **Enhanced Model(optional):** [HachiML/Swallow-7b-hf-CodeSkill](https://huggingface.co/HachiML/Swallow-7b-hf-CodeSkill)
- **Skill type:** Coding

## Uses

### Limitation

- **Model Architecture:** Llama2
- **Model Size:** 6.74B
- **Compatible Models[optional]:**

### How to Apply Skill (Example)

```python
# Import Library
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Load the target model to be applied skill
tokenizer = AutoTokenizer.from_pretrained(
    "tokyotech-llm/Swallow-7b-hf"
)
model = AutoModelForCausalLM.from_pretrained(
    "tokyotech-llm/Swallow-7b-hf",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

# Load SkillTree
skill_tree = AutoModelForCausalLM.from_pretrained(
    "HachiML/SkillTree-llama2-7b-hf-Code",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

# Apply the skill to the target model
def apply_skill(model, skill_tree):
    # excluded object
    skip_layers = ["model.embed_tokens.weight", "model.norm.weight", "lm_head.weight"]
    # apply skill
    for k, v in model.state_dict().items():
        # layernorm is also excluded
        if (k in skip_layers) or ("layernorm" in k):
            continue
        vector = skill_tree.state_dict()[k]
        new_v = v + vector.to(v.device)
        v.copy_(new_v)
    return model

model = apply_skill(model, skill_tree)

# Push to hub
model_name = "HachiML/Swallow-7b-hf-CodeSkill"
tokenizer.save_pretrained(f"./models/{model_name}", repo_id=model_name, push_to_hub=True)
model.save_pretrained(f"./models/{model_name}", repo_id=model_name, push_to_hub=True)
```