JoshuaAshkinaze/interesting-informative

ibm_debate_speeches / interestingspeaker

Task: regression
Base model: answerdotai/ModernBERT-base
Description: This model predicts the average annotator score for the interestingspeaker field of this dataset (https://huggingface.co/datasets/ibm-research/debate_speeches). It predicts if an argument is rated as interesting and informative on a 1-5 scale.

Config

{
  "learning_rate": 0.0002,
  "num_train_epochs": 2,
  "per_device_train_batch_size": 32,
  "gradient_accumulation_steps": 1,
  "lora_r": 64,
  "lora_alpha": 128,
  "lora_alpha_ratio": 2,
  "lora_dropout": 0.05,
  "target_modules": "Wqkv"
}

Test metrics

{
  "test_loss": 0.1849627047777176,
  "test_spearman": 0.672744517518574,
  "test_kendall_tau": 0.4856048788096411,
  "test_pearson": 0.674930517660111,
  "test_rmse": 0.4300729063516064,
  "test_r2": 0.4492316628726045,
  "test_runtime": 5.2726,
  "test_samples_per_second": 24.656,
  "test_steps_per_second": 0.948
}

How to use

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from peft import PeftModel


####################
# Load Model
####################

BASE_MODEL = "answerdotai/ModernBERT-base"
ADAPTER = "JoshuaAshkinaze/interesting-informative"

base_model = AutoModelForSequenceClassification.from_pretrained(
    BASE_MODEL,
    num_labels=1,
)
model = PeftModel.from_pretrained(base_model, ADAPTER)
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)

device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)
model.eval()


####################
# Inference
####################

def score_arguments(model, tokenizer, texts, max_length=1024):
    """Score a list of argument texts. Higher = more interesting speaker."""
    device = next(model.parameters()).device

    inputs = tokenizer(
        texts,
        truncation=True,
        padding="max_length",
        max_length=max_length,
        return_tensors="pt",
    ).to(device)

    with torch.no_grad():
        logits = model(**inputs).logits

    return logits.squeeze(-1).tolist()


####################
# Example
####################

args = [
    "Climate change poses existential risks because rising CO2 levels correlate "
    "with extreme weather events, threatening food security and displacing millions.",
    "Climate change is bad and we should do something about it because scientists say so.",
]

scores = score_arguments(model, tokenizer, args)
for arg, score in zip(args, scores):
    print(f"{score:.4f}: {arg[:80]}...")