JoshuaAshkinaze/argument-support

ibm_debate_speeches / mostargumentssupport

Task: regression
Base model: answerdotai/ModernBERT-base
Measures: Predicts the "mostargumentssupport" field of this dataset (https://huggingface.co/datasets/ibm-research/debate_speeches), taking the mean of annotator ratings as the ground truth. It predicts if experts would say a claim is supported by arguments (1-5 scale).

This model was trained using LoRA, performing a random search over hyperparameters and picking the best model by spearnman rho.

Config

{
  "learning_rate": 6e-05,
  "num_train_epochs": 8,
  "per_device_train_batch_size": 32,
  "gradient_accumulation_steps": 1,
  "lora_r": 128,
  "lora_alpha": 256,
  "lora_alpha_ratio": 2,
  "lora_dropout": 0.05,
  "target_modules": "Wqkv"
}

The other hyperparameters used the Transformers Trainer defaults. Training used early stopping with a patience of 2; we report test set performance from the best checkpoint, selected by validation loss at epoch 8. The test Spearman's rho exceeds the average inter-annotator agreement, measured as each annotator's rho with the mean of all other annotators.

Test metrics

{
  "test_loss": 0.3013608753681183,
  "test_spearman": 0.7058525835607535,
  "test_kendall_tau": 0.5102838725418962,
  "test_pearson": 0.6425724592223252,
  "test_rmse": 0.5489634825472827,
  "test_r2": 0.38694407752977733,
  "test_runtime": 5.3284,
  "test_samples_per_second": 24.397,
  "test_steps_per_second": 0.938
}

How to use

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from peft import PeftModel


####################
# Load Model
####################

BASE_MODEL = "answerdotai/ModernBERT-base"
ADAPTER = "JoshuaAshkinaze/argument-support"

base_model = AutoModelForSequenceClassification.from_pretrained(
    BASE_MODEL,
    num_labels=1,
)
model = PeftModel.from_pretrained(base_model, ADAPTER)
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)

device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)
model.eval()


####################
# Inference
####################

def score_arguments(model, tokenizer, texts, max_length=1024):
    """Score a list of argument texts. Higher = the argument supports its claims."""
    device = next(model.parameters()).device

    inputs = tokenizer(
        texts,
        truncation=True,
        padding="max_length",
        max_length=max_length,
        return_tensors="pt",
    ).to(device)

    with torch.no_grad():
        logits = model(**inputs).logits

    return logits.squeeze(-1).tolist()


####################
# Example
####################

args = [
    "This is an argument right here",
    "And this is an argument too"
]

scores = score_arguments(model, tokenizer, args)
for arg, score in zip(args, scores):
    print(f"{score:.4f}: {arg[:80]}...")