finbert-deberta-large-sentiment

A larger, more accurate sibling of finbert-minilm-sentiment: 90.2% on finance tweets with transformers 4.x (+3.7 points; 89.5% on transformers 5.x), 99.4% on PhraseBank test, at 13x the parameters and about 19x the CPU latency.

finbert-deberta-large-sentiment classifies short financial text into negative / neutral / positive. It is microsoft/deberta-v3-large (435M parameters) fine-tuned with the same data mix as the compact model (v3): Financial PhraseBank, finance tweets and press-release headlines. Use it when accuracy matters more than speed. For CPU serving at scale the 33.4M-parameter compact model is still the better trade (see Size and speed).

Results (held-out sets)

None of these sets was used for training or for choosing the model. Differences are against v3 (9mark9/finbert-minilm-sentiment), with paired bootstrap 95% intervals (2,000 resamples of the same rows for both models). All numbers are for this repository's weights loaded in fp32.

Library versions matter (verified 2026-10-31). I re-scored the published weights on a separate ZeroGPU Space with training/common.py (fp32, 96 tokens). With the pinned environment in training/requirements.txt (torch 2.8.0, transformers 4.57.6) every number below reproduces exactly: PhraseBank 99.41, tweets 90.16, headlines 78.28, SEntFiN 79.74. With transformers 5.x, the newest release when I checked, the same weights score 89.53% on tweets (+3.10 over v3, 95% CI [+1.80, +4.44]) and the other three sets are unchanged. Install transformers>=4.51,<5 to get the numbers reported here. I did not isolate which library change causes the difference.

Set	n	v3 accuracy	This model	Change (points) [95% CI]	v3 macro-F1	This model	Change [95% CI]
PhraseBank AllAgree test (15%)	340	97.94	99.41	+1.47 [+0.00, +3.24]	96.43	99.28	+2.85 [+0.28, +5.97]
Finance tweets, validation file	2,388	86.43	90.16	+3.73 [+2.47, +5.07]	83.00	87.93	+4.94 [+3.36, +6.57]
Press-release headlines (JB), test	267	75.28	78.28	+3.00 [-2.25, +8.24]	59.04	61.37	+2.33 [-9.45, +14.79]
SEntFiN headlines, test (one label per headline)	933	75.99	79.74	+3.75 [+1.29, +6.22]	76.27	80.30	+4.03 [+1.53, +6.51]

Per-class precision / recall (negative, neutral, positive):

Set	v3 precision	v3 recall	This model precision	This model recall
PhraseBank	0.953 / 1.000 / 0.944	0.911 / 0.990 / 0.988	0.978 / 1.000 / 0.989	1.000 / 0.990 / 1.000
Tweets	0.725 / 0.946 / 0.752	0.850 / 0.870 / 0.857	0.831 / 0.947 / 0.819	0.879 / 0.908 / 0.897
JB headlines	0.667 / 0.764 / 0.733	0.167 / 0.853 / 0.667	0.667 / 0.794 / 0.766	0.167 / 0.865 / 0.727
SEntFiN	0.850 / 0.676 / 0.811	0.722 / 0.825 / 0.723	0.891 / 0.696 / 0.866	0.792 / 0.836 / 0.762

Datasets and splits: PhraseBank Sentences_AllAgree, stratified 70/15/15 with train_test_split(random_state=42) (test carved first). Tweets: zeroshot/twitter-financial-news-sentiment sent_valid.csv (labels 0 bearish, 1 bullish, 2 neutral). JB: Jean-Baptiste/financial_news_sentiment test split, headline text (0 negative, 1 neutral, 2 positive). SEntFiN: tasksource/sentfin test split (revision d6294b7), headlines where every entity has the same label. All revisions are pinned in training/common.py.

Release rule

Fixed before training started: release a new repository only if tweet accuracy beats v3 by at least 2.0 points with a paired 95% interval entirely above zero, and PhraseBank accuracy is not more than 1.0 point worse. Result: +3.73 [+2.47, +5.07] on tweets and +1.47 on PhraseBank, so it passes.

How this model was chosen

Every candidate was ranked by the mean macro-F1 over three validation slices carved from the training files (PhraseBank validation split, 10% of the tweet training file, 10% of the headline training file). The held-out sets above were scored for every candidate but not used to choose.

One change was made during the search, and it is stated here. Once this model was trained, a probability-average ensemble containing it ranked slightly higher on validation (by 0.0005). A tie-break rule was added at that point, after this model's own held-out scores had been seen, but before roberta-large had finished and before any ensemble containing this model had been scored on the held-out sets. The rule: an ensemble wins only if it leads the best single model on validation by at least 0.005, because it costs at least 1.5x the CPU time for a gap inside validation noise. The final top ensemble (this model + roberta-large + v3, 0.8772) led this model (0.8751) by 0.0021, so the single model was kept. For transparency, that ensemble scored 99.12 / 90.12 / 79.03 / 80.92 on the four sets (tweets +3.69 [+2.51, +4.94]), about the same as this model.

All candidates

Accuracy on the held-out sets; validation = mean macro-F1 over the three validation slices. ens_* = average of class probabilities. Runs marked (Mac) were trained on an Apple GPU in fp32, the others on a ZeroGPU Space with bf16 autocast.

Candidate	Params	Validation	PhraseBank	Tweets	JB	SEntFiN	Tweets vs v3 [95% CI]
v3 (finbert-minilm-sentiment)	33.4M	0.8239	97.94	86.43	75.28	75.99	-
deberta-v3-base, seed 42	184.4M	0.8650	99.12	88.61	80.15	79.96	+2.18 [+0.96, +3.48]
deberta-v3-base, seed 1	184.4M	0.8486	99.41	87.48	77.90	79.31	+1.05 [-0.21, +2.35]
deberta-v3-base, seed 2	184.4M	0.8500	98.82	86.64	74.91	80.39	+0.21 [-1.09, +1.55]
deberta-v3-base, weight average of the 3 seeds	184.4M	0.8375	98.53	83.88	78.65	80.60	-2.55 [-3.94, -1.17]
roberta-base, seed 42 (Mac)	124.6M	0.8463	99.41	88.32	68.91	80.17	+1.88 [+0.63, +3.22]
roberta-base, seed 1	124.6M	0.8421	99.41	88.86	72.28	77.17	+2.43 [+1.13, +3.73]
distilroberta-base, seed 42 (Mac)	82.1M	0.8389	97.94	87.56	77.53	75.99	+1.13 [-0.21, +2.43]
ProsusAI/finbert, seed 42 (Mac) *	109.5M	0.8283	99.71	87.23	72.66	75.88	+0.80 [-0.50, +2.05]
roberta-large, seed 42	355.4M	0.8617	98.82	86.77	78.28	80.71	+0.34 [-1.01, +1.84]
deberta-v3-large, seed 42 (this model)	435.1M	0.8751	99.41	90.16	78.28	79.74	+3.73 [+2.47, +5.07]
ens: deberta-v3-base s42 + s1 + v3	402.2M	0.8700	99.41	88.74	77.53	80.39	+2.30 [+1.17, +3.52]
ens: deberta-v3-large + roberta-large + v3	823.8M	0.8772	99.12	90.12	79.03	80.92	+3.69 [+2.51, +4.94]
ens: deberta-v3-large + deberta-v3-base s2 + v3	652.9M	0.8756	99.12	90.62	79.40	80.28	+4.19 [+2.97, +5.40]

* ProsusAI/finbert is CC-BY-NC-SA-4.0 and was itself fine-tuned on Financial PhraseBank, so its PhraseBank score is not a clean held-out number; it was a reference only.

Seeds matter: three deberta-v3-base seeds span 86.6 to 88.6 on tweets. deberta-v3-large was trained with one seed only, so part of its margin may be seed luck.

Size and speed

CPU latency per sentence, batch size 1, 4 threads, median (90th percentile) over 300 tweets after 20 warm-up calls, fp32, measured on a shared cloud CPU (absolute numbers vary by machine; ratios are the useful part):

Model	Parameters	Weights (fp32)	CPU latency	Tweets accuracy
finbert-minilm-sentiment (v3)	33.4M	0.13 GB	13.5 ms (18.5)	86.43
deberta-v3-base fine-tune (seed 42)	184.4M	0.74 GB	83.6 ms (104.9)	88.61
roberta-large fine-tune	355.4M	1.4 GB	127.3 ms (155.6)	86.77
this model	435.1M	1.7 GB	259.3 ms (301.2)	90.16

How far can tweet accuracy go? (label-noise estimate)

This model gets 235 of the 2,388 validation tweets wrong. I read 100 of those errors, drawn at random (seed 0), and judged each one against the text:

Judgement	Count
Gold label looks wrong; the model's label fits the text better	32
Genuinely ambiguous (either label defensible)	29
Clear model error (gold label is right)	39

Examples of the first kind: "Margins suffer at Ituran Location in Q3" (gold neutral, model negative), "iAnthus Defaults On Debt" (gold neutral), "Howard Bancorp Shareholders Have Enjoyed A 54% Share Price Gain" (gold neutral, model positive), "U.S. oil firms vented or flared record amount of gas" (gold bullish).

If a model fixed every clear error and nothing else, it would reach about 94.0% (95% interval 93.1% to 95.0%, from the Wilson interval on 39/100). Going past that means agreeing with labels that look wrong or with coin-flip judgements, which is fitting the annotators, not the sentiment. So 95% or more on this tweet set is not a realistic honest target. This is one reviewer's judgement, made while seeing the model's prediction, so treat it as an estimate.

Training data

Held out and never used for training or model selection: the PhraseBank 15% test split, the full tweet validation file, the headline test split and all of SEntFiN. Before training, any row whose text matches a held-out or validation row was dropped. The match key is lower-cased text with URLs and punctuation removed. 111 rows were dropped.

Note on v3: v3 deduplicated on lower-cased, whitespace-normalised text only. With the stricter key, 80 rows of the tweet training file and 8 rows of the headline training file match held-out text (for example the same headline with a different link). v3 was trained with those rows; this model was not.

Source	Used for training	Used for validation
PhraseBank Sentences_AllAgree	70% train split (1,584), repeated 3x per epoch	15% validation split (340)
PhraseBank Sentences_66Agree	only sentences not in the all-agree file at all (1,951); majority label	-
Tweets, sent_train.csv	90% stratified (8,588)	10% stratified (955)
Headlines (JB), train split	90% stratified (1,360), repeated 2x per epoch	10% stratified (152)

17,900 training rows per epoch after deduplication. Dataset licences apply to the data.

Training details

Setting	Value
Base encoder	microsoft/deberta-v3-large (MIT)
Parameters	435,064,835
Objective	class-weighted cross-entropy (inverse class frequency)
Optimizer	AdamW, lr 1e-5, weight decay 0.01, gradient clip 1.0
Schedule	linear, 10% warmup; batch 16; max 96 tokens; 3 epochs (3,357 steps)
Precision	fp32 weights, bf16 autocast for the forward pass
Checkpoint	best mean validation macro-F1, evaluated twice per epoch (step 3,354)
Hardware	Hugging Face ZeroGPU (half of an RTX Pro 6000), 481 GPU-seconds

The learning rate (1e-5 for the large encoders, 2e-5 for base, 3e-5 for distilroberta) was fixed before training, not tuned.

Reproduce

The temporary private Space (9mark9/trainer-tmp) and checkpoint repos (tmp-train-*) named in the commands below were deleted after training. To reproduce, recreate the Space from training/app.py on a ZeroGPU Space and use your own tmp-train-* names.

training/ holds the code that produced and scored this model: app.py (a Gradio app for a ZeroGPU Space: train_chunk, evaluate, latency, soup), common.py (data loading at pinned revisions, dedupe key, splits), requirements.txt, drive.py (client that calls the Space) and analyze.py (accuracy, macro-F1, per-class precision/recall, paired bootstrap against v3).

1. Create a Gradio Space with ZeroGPU hardware, add app.py, common.py and requirements.txt, and set a write token as the secret HF_TOKEN (used to push checkpoints to private repos).
2. Point SPACE in drive.py at it, then run:

mkdir -p probs
python drive.py train debL_s42 '{"base":"microsoft/deberta-v3-large","lr":1e-5,"pb_extra":"66Agree","jb_rep":2,"seed":42,"gpu_seconds":600,"margin":80}' 2
python drive.py eval debL_s42_fp32 <your-user>/tmp-train-debL_s42 best:fp32 probs/debL_s42_fp32.npz
python drive.py eval v3 9mark9/finbert-minilm-sentiment - probs/v3.npz
python analyze.py results.json v3 debL_s42_fp32

GPU training is not bit-for-bit deterministic, so a rerun lands close to, not exactly on, these numbers.

Usage

Install pip install "transformers>=4.51,<5" sentencepiece torch to reproduce the reported numbers (see the library-version note above).

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model_id = "9mark9/finbert-deberta-large-sentiment"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
model.eval()

texts = [
    "The company reported record quarterly profit, beating analyst expectations.",
    "Net sales decreased by 12% and the firm warned of further losses ahead.",
    "The board will meet next Tuesday to review the quarterly schedule.",
]

enc = tokenizer(texts, padding=True, truncation=True, max_length=96, return_tensors="pt")
with torch.no_grad():
    probs = model(**enc).logits.softmax(-1)

for text, p in zip(texts, probs):
    label = model.config.id2label[int(p.argmax())]
    print(f"[{label:8s} {p.max():.2f}] {text}")

Tested with torch 2.8, transformers 4.x and sentencepiece installed. The three example sentences come out positive, negative and neutral.

Intended use

• Sentiment scoring of short financial text: news sentences, headlines, finance tweets, analyst notes.
• Research and offline or GPU batch scoring where accuracy matters more than latency.

Limitations

• English only. Max sequence length 96 tokens; longer text is truncated.
• Weak on negative press-release headlines: it finds 2 of the 12 negatives in the JB test split, the same as v3. On the larger SEntFiN set (259 negatives) negative recall is 0.792.
• On finance tweets about 17% of the tweets it calls negative are labelled neutral or bullish.
• Tweet labels have a noise ceiling (see above); about 94% is a realistic upper bound on that set.
• PhraseBank AllAgree is the cleanest, easiest part of that benchmark; expect lower accuracy on ambiguous sentences.
• Tweet labels describe bullish / bearish market direction, which is not the same as the tone of a sentence. The model mixes both conventions.
• About 19x slower than the compact model on CPU.
• Not financial advice. Outputs are sentiment labels, not buy/sell signals.

Citation

Dataset:

@article{malo2014good,
  title={Good debt or bad debt: Detecting semantic orientations in economic texts},
  author={Malo, Pekka and Sinha, Ankur and Korhonen, Pekka and Wallenius, Jyrki and Takala, Pyry},
  journal={Journal of the Association for Information Science and Technology},
  volume={65}, number={4}, pages={782--796}, year={2014}
}

Contact

Maintainer: nagizaazs@gmail.com

License

MIT