cap-policy-subtopic

LoRA adapter on answerdotai/ModernBERT-base that labels a policy text with one of 124 Comparative Agendas Project (CAP) subtopics.

Whereas a CAP major topic classifier tells you a text is about Health, this adapter tells you it is about Health — Insurance rather than Health — Prescription Drug Coverage.

Intended Use

Classifying short policy texts. Labels are formatted {Major Topic} — {Subtopic}, e.g. Environment — Air Pollution, Government Operations — Branch Relations, Civil Rights & Civil Liberties — Voting Rights. The full list is in label_classes.json.

Training Data

CAP's US Legislative CRS report dataset (US-Legislative-crs_reports_24.1_1), in which CAP coders assigned each Congressional Research Service report a single major topic and subtopic. Input text is the report's title and summary concatenated.

Step	Count
Raw rows	16,746
Rows with a summary and a valid subtopic	10,027
Subtopics present	206
Subtopics dropped (fewer than 20 examples)	82
Subtopics kept	124
Training documents	9,293

Split 70/15/15 into 6,505 train / 1,394 validation / 1,394 test.

Evaluation

Held-out test set, n = 1,394 across 124 classes.

Metric	Score
Macro-F1	0.724
Accuracy	0.772
ROC-AUC (OvR)	0.993

For reference, the same recipe trained on CAP's 20 major topics reaches macro-F1 0.846 / accuracy 0.868. The 124 fine-grained classes are harder because several of them are very similar.

How wrong are the errors?

Of the 318 errors on the test set (22.8% of 1,394 documents), 134 — 42.1% — name the correct major topic and miss only the subtopic within it.

	Count	Share of errors
Correct subtopic	1,076	—
Error, correct major topic	134	42.1%
Error, wrong major topic	184	57.9%

If you treat any subtopic within the right major topic as a hit, 86.8% of documents land correctly, against 77.2% that get the exact subtopic.

Scoring the same predictions as a 20-way major-topic problem — collapsing both the label and the prediction to the major topic, and summing subtopic probabilities within each major for ROC-AUC — gives a model statistically indistinguishable from the one trained on major topics directly:

Metric	This model, collapsed	Trained on major topics
Accuracy	0.868 [95% CI 0.849, 0.885]	0.868
Macro-F1	0.850	0.846
ROC-AUC (OvR)	0.990	0.992

So in short, the 124-way head costs nothing at the major-topic level: you can just read this model as a major-topic classifier if you want to be more sure the labels are correct at the cost of granularity.

Training Procedure

Random search over learning rate, epochs, LoRA rank, and dropout (20 trials), selecting on validation macro-F1.

Parameter	Value
Base model	answerdotai/ModernBERT-base
LoRA rank (r)	32
LoRA alpha	64
LoRA dropout	0.0
Target modules	Wqkv
Learning rate	2e-4
Epochs	8 (best epoch 8)
Batch size	32
Max length	1024
Seed	416

Best validation macro-F1 was 0.707; test macro-F1 was 0.724.

Usage

import json

import torch
from huggingface_hub import hf_hub_download
from peft import PeftModel
from transformers import AutoModelForSequenceClassification, AutoTokenizer

BASE_MODEL = "answerdotai/ModernBERT-base"
ADAPTER = "JoshuaAshkinaze/cap-policy-subtopic"

labels = json.load(open(hf_hub_download(ADAPTER, "label_classes.json")))

base = AutoModelForSequenceClassification.from_pretrained(BASE_MODEL, num_labels=len(labels))
model = PeftModel.from_pretrained(base, ADAPTER)
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)

device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device).eval()


def classify(texts, max_length=1024, top_k=3):
    """Return the top_k (subtopic, probability) pairs for each text."""
    inputs = tokenizer(
        texts,
        truncation=True,
        padding="max_length",
        max_length=max_length,
        return_tensors="pt",
    ).to(device)

    with torch.no_grad():
        probs = model(**inputs).logits.softmax(-1)

    out = []
    for row in probs:
        scores, idx = row.topk(top_k)
        out.append([(labels[i], float(s)) for s, i in zip(scores, idx)])
    return out


for ranked in classify([
    "A bill to require voters to present photographic identification at the polls.",
    "A bill to limit the local tax rate on groceries to 2 percent of base price.",
]):
    print([(label, round(p, 3)) for label, p in ranked])

Citation

@misc{ashkinaze2026capsubtopic,
  author    = {Ashkinaze, Joshua},
  title     = {cap-policy-subtopic: ModernBERT LoRA adapter for Comparative Agendas Project subtopics},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/JoshuaAshkinaze/cap-policy-subtopic}
}

The underlying labels are CAP's:

@misc{cap_us_legislative,
  title        = {Comparative Agendas Project: US Legislative CRS Reports},
  howpublished = {\url{https://www.comparativeagendas.net/us}},
  note         = {Dataset US-Legislative-crs_reports_24.1_1}
}