Falcot922/tinyllama-sentiment-distilled

TinyLlama 1.1B, distilled from Mistral 7B, IMDB sentiment

LoRA adapter for TinyLlama-1.1B-Chat-v1.0, trained by knowledge distillation from the Mistral 7B teacher. Its control, trained on hard labels only, is tinyllama-sentiment-hard-labels. Part of mistral-optimisation.

Results

1,000 IMDB test reviews (shuffled with seed 42, first 400 characters), answer read from the class-token logits.

Model	Accuracy (95% Wilson CI)	Reviews / s (T4)	Peak VRAM
Teacher: Mistral 7B NF4 + QLoRA	91.6% (89.7 to 93.2)	0.87	4.1 GiB
TinyLlama, no adapter	65.4% (62.4 to 68.3)	24.4	2.1 GiB
TinyLlama, hard labels	89.7% (87.7 to 91.4)	24.9	2.2 GiB
TinyLlama, distilled	91.3% (89.4 to 92.9)	24.9	2.2 GiB

This adapter: 91.3% (89.4 to 92.9). Distillation beats hard labels by 1.6 points (paired test, p = 0.033); an earlier run of the same code gave 1.1 points (p = 0.14), so the gain is small. The distilled student is statistically indistinguishable from the teacher (p = 0.81) and 29 times faster.

Training

• Base: TinyLlama/TinyLlama-1.1B-Chat-v1.0 in fp16
• LoRA r = 16, alpha = 32, on all attention and MLP projections
• 1,000 IMDB train reviews, 3 epochs, lr 2e-4, effective batch 16
• Loss: 0.7 x KL divergence to the teacher's class distribution at temperature 2 (scaled by T²) + 0.3 x cross-entropy on the true label

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.float16, device_map="auto")
model = PeftModel.from_pretrained(model, "Falcot922/tinyllama-sentiment-distilled").eval()

prompt = ("Analyze the sentiment of this movie review and respond with exactly one word: "
          "positive or negative.\n\nReview: {text}\nSentiment:")
text = prompt.format(text="A wonderful, moving film.")
n = len(tok(text)["input_ids"])
pos, neg = (tok(text + w)["input_ids"][n] for w in (" positive", " negative"))   # first token of each answer
with torch.no_grad():
    last = model(**tok(text, return_tensors="pt").to(model.device)).logits[0, -1]
print("positive" if last[pos] > last[neg] else "negative")