andreadm/reddit-pulse-bert

Reddit-pulse InflaBERT

Retrained checkpoint. This model was trained again after the paper, on the same gold dataset and with the same seed protocol. The metrics on this card come from that retraining and may differ from the ones reported in the paper.

Reddit-pulse InflaBERT is a three-way directional inflation-expectation classifier for short English texts about the economy, fine-tuned on Reddit submission titles. Given a title (or any sentence-length text), it predicts whether the text conveys that inflation / prices are going up, going down, or carries no directional signal (neutral).

It is not a sentiment model: "inflation falls sharply" is good news but labelled down; "rents are out of control" is bad news but labelled up. The direction is about the price level, not the mood.

The checkpoint is the model behind the Reddit inflation signal in

Del Monaco, A., Longo, L., Marcucci, J. & Tafani, I. (2026). Reddit's 'pulse' on US inflation: forecasting with large language models. Journal of Applied Econometrics, forthcoming. Working-paper version: Banca d'Italia, Questioni di Economia e Finanza (Occasional Papers) No. 1028, June 2026, doi:10.32057/0.QEF.2026.1028.

The fine-tuning and full-corpus inference code lives at andrea-dm/reddit-pulse.

Model lineage

Stage	Model	Notes
Pre-training	distilroberta-base	6-layer distilled RoBERTa, 82.1 M parameters
Domain adaptation	mrm8488/distilroberta-finetuned-financial-news-sentiment-analysis	Financial-news sentiment
Task adaptation	MAPAi/InflaBERT	Inflation-news sentiment (negative / neutral / positive)
This checkpoint	andreadm/reddit-pulse-bert	Full fine-tuning on Reddit titles, labels re-mapped to down / neutral / up

Architecture: RobertaForSequenceClassification, 6 hidden layers, hidden size 768, 12 attention heads, 50 265-token byte-level BPE vocabulary, 512-token context. Every parameter was updated (no adapters, no quantization); the classification head was re-initialised for the three directional classes.

Labels

id	label	meaning	encoding used in the paper's corpus files
0	down	inflation / prices heading lower, disinflation, falling expectations	-1
1	neutral	no directional signal (questions, definitions, unrelated, mixed)	0
2	up	inflation / prices heading higher, rising expectations	+1

How to use

from transformers import pipeline

clf = pipeline("text-classification", model="andreadm/reddit-pulse-bert")
clf("Fed officials warn prices will keep climbing as CPI hits 40-year high")
# [{'label': 'up', 'score': 0.999}]

Batched, with the paper's -1 / 0 / +1 encoding:

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

name = "andreadm/reddit-pulse-bert"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name).eval()
encoding = {"down": -1, "neutral": 0, "up": 1}

texts = [
    "Inflation expectations drop to lowest level since 2021, NY Fed survey shows",
    "What is the difference between CPI and PCE?",
]
with torch.no_grad():
    batch = tokenizer(texts, padding=True, truncation=True, max_length=512, return_tensors="pt")
    ids = model(**batch).logits.argmax(dim=-1).tolist()

print([encoding[model.config.id2label[i]] for i in ids])   # [-1, 0]

Tested with transformers 5.16 and PyTorch 2.13; the checkpoint uses the standard safetensors + fast-tokenizer layout and loads on any recent 4.x release as well. Inference is cheap: the full paper corpus (≈ 243 k texts) labels in a few minutes on one A100 at batch size 64.

Intended use and limitations

Intended use. Labelling large volumes of short, informal, English, economy-related texts (Reddit titles and comments, headlines, social media posts) with a directional inflation signal that is then aggregated over time — the paper's use case. The model is a building block for a high-frequency indicator, not a stand-alone oracle.

Limitations.

• Single predictions are noisy. Held-out accuracy is ≈ 74 %; the value of the model comes from averaging thousands of predictions per period, where idiosyncratic errors wash out. Do not rely on any single label.
• Domain and register. Trained on r/economy, r/Economics and r/wallstreetbets titles about US inflation from 2008 to 2022. Other countries, other registers (formal reports, central-bank prose) and post-2022 vocabulary are out of distribution.
• Class imbalance. down is the minority class (16 % of the gold set) and the hardest one (test F1 0.67 vs 0.79 for neutral). The loss was class-balanced during training, but down recall remains lower.
• Short texts. Fine-tuned on titles (median 11 words, max 52). Long comments are truncated at 512 tokens and were not seen during training.
• Direction, not stance or sentiment. The model does not say whether the author wants inflation to move, nor whether the news is good or bad; only which way prices are said to be going.
• No factuality. A confidently worded false claim is labelled by its direction, not its truth.

Training data

The gold set is a hand-labelled sample of 1 383 Reddit submission titles from r/economy, r/Economics and r/wallstreetbets, dated February 2008 to December 2022. Labels were produced by a human-in-the-loop protocol (manual annotation assisted by zero-shot LLaMA-70B, a fine-tuned LLaMA-8B classifier and ChatGPT-assisted adjudication of disagreements) described in the paper. The gold set itself is not redistributed here.

label	titles	share
neutral	623	45.0 %
up	537	38.8 %
down	223	16.1 %

Titles are lower-cased as found on Reddit; median length 11 words (IQR 8–16, max 52).

Training procedure

Seed protocol and model selection

The paper's protocol asks how sensitive the classifier is to which titles it is trained on, so it separates two sources of randomness:

• 19 split seeds (training.seeds) each draw a different stratified 71 / 19 / 10 % train / validation / test partition (981 / 263 / 139 titles) of the same gold set.
• One fixed seed (42) governs everything else: classification-head initialisation, batch shuffling and dropout. Every split therefore trains the same model the same way on different data.

Each split is fine-tuned independently; the checkpoint with the median test weighted-F1 across the 19 runs (the upper median, rank 10 of 19) is kept and the others discarded. The result is a typical run, not the best one — the reported metrics are an honest estimate of what a rerun yields, not a lucky draw. This checkpoint is split seed 2786505123.

Hyperparameters

Objective	Cross-entropy with balanced class weights (sklearn.utils.class_weight)
Optimizer	AdamW (fused), β = (0.9, 0.999), ε = 1e-8, weight decay 0.01
Learning rate	5e-5, linear decay, no warm-up
Batch size	64 (train and eval), dynamic padding to multiples of 8
Epochs	up to 15, early stopping with patience 5 on validation weighted-F1
Checkpoint	best epoch by validation weighted-F1 (load_best_model_at_end)
Precision	fp16 mixed precision, fp32 master weights
Max sequence length	512 tokens (truncation only; titles are far shorter)
Gradient clipping	1.0
Hardware	1 × NVIDIA A100 80 GB; ≈ 2 min per seed, 38 min for all 19

The exact TrainingArguments are in training_args.json and the governing configuration extract in training_config.yml.

Evaluation

Selected checkpoint (split seed 2786505123)

split	n	accuracy	F1 weighted	F1 macro	precision macro	recall macro	ROC-AUC
validation	263	0.757	0.756	0.739	0.742	0.738	0.864
test	139	0.741	0.739	0.722	0.725	0.724	0.872

Per class, on the test split:

label	precision	recall	F1	support
down	0.652	0.682	0.667	22
neutral	0.746	0.841	0.791	63
up	0.778	0.648	0.707	54

Confusion matrix (rows = true, columns = predicted):

	down	neutral	up
down	15	3	4
neutral	4	53	6
up	4	15	35

Most errors are up titles absorbed by neutral (15 of 54): hedged or question-shaped titles whose direction a human infers from context. Confusions between up and down — the ones that would bias an aggregate signal — are rare (8 of 139).

Split sensitivity across the 19 seeds

Test metrics of every run, sorted by weighted F1. The selected checkpoint is the median.

split seed	val F1w	test acc	test F1w	test F1m	test AUC
709964709	0.792	0.676	0.676	0.640	0.813
477284336	0.726	0.691	0.689	0.688	0.907
107935903	0.793	0.698	0.702	0.680	0.878
1245093080	0.740	0.705	0.707	0.693	0.866
4280088979	0.803	0.712	0.714	0.704	0.891
2928142788	0.744	0.712	0.716	0.694	0.863
3144693271	0.810	0.719	0.719	0.700	0.862
239080115	0.759	0.734	0.732	0.716	0.876
1361883482	0.786	0.734	0.733	0.709	0.901
2786505123	0.756	0.741	0.739	0.722	0.872	selected
228277762	0.707	0.748	0.749	0.714	0.895
3266123502	0.794	0.748	0.752	0.731	0.916
1329496050	0.792	0.748	0.752	0.740	0.923
4054871397	0.768	0.755	0.755	0.736	0.874
1389303030	0.750	0.755	0.757	0.745	0.898
3154447144	0.783	0.784	0.785	0.767	0.886
2565555162	0.775	0.791	0.794	0.766	0.921
2078237541	0.783	0.799	0.797	0.779	0.911
4203596092	0.765	0.813	0.813	0.804	0.923

test metric	mean	std	min	median	max
accuracy	0.740	0.038	0.676	0.741	0.813
F1 weighted	0.741	0.038	0.676	0.739	0.813
F1 macro	0.722	0.039	0.640	0.716	0.804
ROC-AUC	0.888	0.028	0.813	0.891	0.923

The spread is what a 139-title test set implies (one title ≈ 0.7 accuracy points) and is the reason the paper reports the median run rather than a single split. Full per-seed numbers: evaluation/.

Corpus labelling in the paper

Applied to the paper's filtered corpus, this checkpoint labels 33 460 submissions and 209 995 comments (r/economy, r/Economics, r/wallstreetbets, 2008–2022):

	up	neutral	down
submissions	43.6 %	42.0 %	14.4 %
comments	40.3 %	51.8 %	8.0 %

Files

file	content
model.safetensors, config.json	weights (fp32, 328 MB) and architecture / label map
tokenizer.json, tokenizer_config.json	byte-level BPE tokenizer inherited from RoBERTa
training_args.json	the transformers.TrainingArguments of the selected run
training_config.yml	dataset split, hyperparameters, seed list and label map from the project config
evaluation/seeds_test_metrics.csv	test metrics of all 19 split seeds
evaluation/seeds_validation_metrics.csv	validation metrics of all 19 split seeds
evaluation/selected_seed_report.json	per-class report and confusion matrices of this checkpoint
LICENSE.md	MIT license, scope statement and upstream Apache-2.0 / MIT notices

Reproducing

git clone https://github.com/andrea-dm/reddit-pulse && cd reddit-pulse
uv venv && uv pip install -e .
reddit run --model inflabert --gpu 0     # 19 seeds, median selection, corpus labelling

The gold set (data/labelled.xlsx) and the subreddit corpus are not part of the repository; see the paper for the data-construction stages.

Citation

@article{delmonaco2026reddit,
  title   = {Reddit's `pulse' on {US} inflation: forecasting with large language models},
  author  = {Del Monaco, Andrea and Longo, Luigi and Marcucci, Juri and Tafani, Irene},
  journal = {Journal of Applied Econometrics},
  year    = {2026},
  note    = {forthcoming},
}

@techreport{delmonaco2026reddit_qef,
  title       = {Reddit's `pulse' on {US} inflation: forecasting with large language models},
  author      = {Del Monaco, Andrea and Longo, Luigi and Marcucci, Juri and Tafani, Irene},
  institution = {Banca d'Italia},
  series      = {Questioni di Economia e Finanza (Occasional Papers)},
  number      = {1028},
  year        = {2026},
  month       = jun,
  doi         = {10.32057/0.QEF.2026.1028},
}

License

Apache License, Version 2.0 (see LICENSE.md). The direct parent MAPAi/InflaBERT is MIT-licensed; the deeper lineage (distilroberta-base, mrm8488/distilroberta-finetuned-financial-news-sentiment-analysis) is Apache-2.0, which this checkpoint now matches. The attribution notices these upstream licenses require are included in LICENSE.md.

The views expressed in the paper are those of the authors and do not necessarily reflect those of the Bank of Italy, the Eurosystem, or the European Commission.