distilbert-fin-sentiment

DistilBERT fine-tuned for 3-class financial sentiment classification (negative / neutral / positive), trained on the Financial PhraseBank dataset. Built as part of MarketPulse, a portfolio project comparing a fine-tuned transformer against a TF-IDF + Logistic Regression baseline. Full build log: see the project's docs/learning-log/.

Intended use

Classifies short financial news sentences/headlines by sentiment, from the viewpoint of an investor (e.g. "costs fell" is scored positive, not negative, despite the negative-sounding word). Intended for portfolio, research, and non-commercial demo use — not for production financial decision-making without further validation, and not for commercial use (see License below).

Training data

Financial PhraseBank (sentences_50agree subset), ~4,846 sentences annotated by 16 finance professionals/researchers. Licensed CC-BY-NC-SA-3.0 (non-commercial, share-alike) — this model inherits that restriction. Loaded from the raw archive directly, since the dataset's datasets-library loading script is deprecated. Cleaned before training: 6 exact duplicate sentences collapsed to one copy each, and 2 sentences with genuinely conflicting labels across duplicate occurrences were dropped entirely (kept ambiguous ground truth out rather than arbitrarily picking one label). Final training set: 4,836 sentences, split 70/15/15 (train/validation/test), stratified, seed=42.

Evaluation results (test set, n=726, measured — not estimated)

Model	Accuracy	Weighted F1	Macro F1
TF-IDF + Logistic Regression (baseline)	0.7631	0.7636	0.7393
This model (DistilBERT, fine-tuned)	0.8471	0.8475	0.8385

Per-class F1: negative 0.862, neutral 0.879, positive 0.775.

The dominant error mode is neutral↔positive confusion (see the project's confusion matrix) — negative-class sentences are classified most reliably.

Latency (single request, CPU — not GPU): ~14.9 ms/request, vs ~0.33 ms/request for the baseline (45x slower in relative terms, though still fast in absolute terms for a synchronous API).

Limitations

• Class imbalance: training data is ~59% neutral, 28% positive, 12% negative. Minority-class (negative) metrics are noisier on small eval splits as a result.
• Label noise: annotator agreement was only ≥50% for this subset; 2 sentences with directly conflicting labels were found and excluded during cleaning, which is evidence this dataset has genuine, not just theoretical, annotation disagreement.
• Distribution shift: trained on ~2013-era Nordic financial news sentences. Performance on modern text, tweets, or non-Finnish-sourced financial news is not validated.
• Short-text bias: trained on single sentences (median ~28 tokens); performance on long documents is not validated.
• Uncalibrated confidence: softmax scores are not true probabilities — don't treat them as calibrated uncertainty estimates.

Example usage

from transformers import pipeline

clf = pipeline("text-classification", model="HamimSaad/distilbert-fin-sentiment", top_k=None)
clf("The company reported a sharp drop in quarterly profit.")

License

Model weights: released under the same restrictions as the training data inherits, CC-BY-NC-SA-3.0 (non-commercial, share-alike, attribution required). Contact the Financial PhraseBank authors (Malo, Sinha, Takala, Korhonen, Wallenius) for commercial licensing of the underlying data.