YesLab-KSU/folk-relevance-classifier

Folk Relevance Classifier (v7)

A seed-conditioned cross-encoder that classifies whether a passage is a folk description of a CHC (Cattell-Horn-Carroll) narrow cognitive ability — i.e., whether an ordinary person is naturally describing that ability in everyday language, without knowing the psychological term.

---

Model description

Property	Value
Base model	microsoft/deberta-v3-base
Architecture	Cross-encoder (sequence classification)
Labels	off_topic (0) · incidental_mention (1) · folk_description (2)
Input format	[CLS] ability anchor [SEP] reddit passage [SEP]
Max length	256 tokens

---

Training details

Hyperparameter	Value
Training examples	7,519
Epochs	8
Batch size	16
Learning rate	2e-5
Weight decay	0.01
Warmup ratio	0.1
Random seed	42
Hardware	NVIDIA RTX 3090
Framework	PyTorch 2.4.1+cu121 · Transformers 4.42.3

---

Evaluation

Metric	Value
Precision	0.575
Recall	0.450
F1	0.505
Threshold	0.30

---

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_id = "Jiho-YesNLP/folk-relevance-classifier"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
model.eval()

def score(ability_name: str, seeds: list[str], passage: str,
          threshold: float = 0.30) -> dict:
    sep = tokenizer.sep_token or "[SEP]"
    anchor = f" {sep} ".join([ability_name, *seeds])
    inputs = tokenizer(anchor, passage,
                       return_tensors="pt", truncation=True, max_length=256)
    with torch.no_grad():
        probs = torch.softmax(model(**inputs).logits, dim=-1)[0]
    labels = model.config.id2label
    prob_folk = probs[max(labels, key=int)].item()
    return {
        "prob_folk": round(prob_folk, 4),
        "is_folk": prob_folk >= threshold,
        "probs": {labels[i]: round(probs[i].item(), 4) for i in range(len(probs))},
    }

# Example
result = score(
    ability_name="Induction",
    seeds=["finding patterns", "figuring out the rule", "spotting regularities"],
    passage="I'm really good at spotting patterns in data that others miss.",
)
print(result)
# {'prob_folk': 0.812, 'is_folk': True, 'probs': {...}}