Iltaf/sst2-lora-distilbert

SST-2 LoRA adapter on distilbert-base-uncased-finetuned-sst-2-english

A LoRA adapter fine-tuned on a subset of SST-2 (binary sentiment) using free Google Colab hardware. This is the deliverable for the E3 project: fine-tune a small open model with LoRA, measure a baseline before training, train, measure again, and report the change honestly even if small.

What this adapter is

• Base model: distilbert-base-uncased-finetuned-sst-2-english (already fine-tuned on full SST-2 by HuggingFace).
• Method: LoRA via peft - r=8, alpha=16, dropout=0.05, targets q_lin, v_lin, and (via modules_to_save) the classifier and pre_classifier heads.
• Training data: 8,000 rows sampled from nyu-mll/glue SST-2 train, seeded with SEED=42.
• Epochs: 3 - Batch: 32 - LR: 2e-4 - Precision: fp16
• Hardware: free Google Colab T4 (15 GB)
• Runtime: ~36 seconds of training (750 steps).

Results (honest before / after)

Same evaluation code, same tokenizer, same collator, same test/val splits for both the baseline and the LoRA-trained model. The only thing that changed is the model weights.

Split	n	Baseline acc	After acc	Delta acc	Baseline F1	After F1	Delta F1
test	72	0.9028	0.9028	+0.0000	0.9019	0.9019	+0.0000
val	800	0.9113	0.9062	-0.0050	0.9112	0.9061	-0.0050
full SST-2 val	872	0.9060	0.9060	+0.0000	0.9058	0.9058	+0.0000

Headline

The LoRA adapter did not improve over the frozen base checkpoint. On the 72-row test set and on the full 872-row SST-2 validation set, predictions are identical. On the 800-row in-training validation set, the adapter is 0.50 points worse (accuracy and macro-F1).

Training loss fell from 0.050 (epoch 1) to 0.037 (epoch 3) while validation loss did not improve - the classic signature of mild overfitting on a small training subset with an already-strong base model. There was very little room for a small adapter to improve on a checkpoint that was already fine-tuned on the full SST-2.

This is a negative result and is reported as such. It is the intended finding of the E3 exercise: the lesson is the method (baseline -> train -> measure -> report), not the magnitude of the delta.

Training curve

Epoch	Train loss	Val loss	Val accuracy	Val F1
1	0.0505	0.3336	0.9063	0.9061
2	0.0380	0.3589	0.9050	0.9049
3	0.0368	0.3361	0.9038	0.9036

The checkpoint reported above is epoch 1, selected by load_best_model_at_end=True on validation macro-F1.

Limitations (read these before using the adapter)

1. The test set has 72 rows. One flipped prediction is +/-1.39% accuracy. Treat any single test number as noisy.
2. The training set is 8,000 rows, about 12% of SST-2 train. This was a deliberate scope limit to fit free Colab.
3. The base model was already fine-tuned on full SST-2, so the headroom for improvement was small from the start.
4. No hyperparameter search. r=8, alpha=16, lr=2e-4, 3 epochs were chosen as standard defaults, not tuned.
5. modules_to_save includes the classifier head, so part of the (non-)improvement is due to head training, not purely LoRA on the attention projections.

Reproduce

from transformers import AutoTokenizer, AutoModelForSequenceClassification
from peft import PeftModel

BASE = "distilbert-base-uncased-finetuned-sst-2-english"
ADAPTER = "Iltaf/sst2-lora-distilbert"

tok = AutoTokenizer.from_pretrained(ADAPTER)
base = AutoModelForSequenceClassification.from_pretrained(BASE)
model = PeftModel.from_pretrained(base, ADAPTER)

inputs = tok("this movie was a delight", return_tensors="pt")
logits = model(**inputs).logits
print("positive" if logits.argmax(-1).item() == 1 else "negative")

Provenance

• Notebook, seeds, and full trace available on request.
• SEED = 42 throughout (data split, training, and evaluation).
• Dataset: nyu-mll/glue, config sst2 (bare glue alias is deprecated in datasets>=4).