Falcon3-3B-Instruct-Conceptizer

Falcon3-3B-Instruct-Conceptizer is built using artificial intelligence technology from the Technology Innovation Institute.

LoRA adapters for Falcon3-3B-Instruct from the paper Conceptizer: Next-Token Prediction Is Not All You Need (Haoran Zheng and Chen Shani).

Conceptizer adds a fixed vocabulary of units of meaning alongside the model's tokenizer, so post-training rewards different wordings of the same meaning, such as safety, protection and security, not only the exact next word. The concepts enter only the training loss; the adapters generate ordinary text.

In the paper, Conceptizer gives the lowest held-out perplexity at concept-target positions against all three baselines and improves knowledge and sentiment tasks, while staying comparable on commonsense.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

tokenizer = AutoTokenizer.from_pretrained("tiiuae/Falcon3-3B-Instruct")
model = AutoModelForCausalLM.from_pretrained("tiiuae/Falcon3-3B-Instruct", dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "FlyinGodzilla/Falcon3-3B-Instruct-Conceptizer")

The paper used transformers 5.0 and peft 0.19. The repository holds the paper's three training seeds: seed 0 loads by default, and seeds 37 and 42 load with subfolder="seed37" or subfolder="seed42".

Training

Rank-16 LoRA post-training on FineWeb-Edu text for 50,000 steps, with next-token prediction, the concept loss and a retention loss that keeps the model close to the base model; the paper has the details. To train your own model with Conceptizer, the Conceptizer repository has the code, the concept vocabulary, the training data and the commands.

License

Distributed under the TII Falcon License; use must comply with TII's Acceptable Use Policy.