HumOmni 2026 Track 1 — EmpathyEval (team tryanderror2)

Pick the most empathetic spoken response to a human utterance. The candidate responses are clean TTS with essentially the same words — they differ only in vocal delivery — so this is a listening problem, not a text problem: the answer is in the prosody, never the transcript.

The method

A single open-weights model, run offline with no API and no dataset-specific handling:

• WavLM-large (MIT, ~1.2 GB) as an open listener — a pairwise good > bad delivery ranker trained on the released training triples. The trained head is inference/ranker.pkl (160 KB); the WavLM backbone downloads from the Hugging Face Hub on first run.
• Layer 9 features (prosody/paralinguistics live in the mid layers), with segment statistics pooling: the global mean + std over time (prosodic variance) plus the mean of 3 equal-time segments (the delivery's temporal trajectory).
• One logistic ranker scores every candidate and picks the argmax — identically for every question (the code never looks at which corpus a clip came from).

Scored against our expert annotations with the official metric: FINAL = 0.832 (tone 0.933, context 0.805), +0.285 over the 0.547 baseline. The method, ablations, environment, and reproduction details are in the technical report (submitted separately, not part of this repository).

Reproduce the submission

The self-contained inference/ package regenerates and scores the submission:

pip install -r inference/requirements.txt
# edit inference/config.yaml: list your test *_release.json file(s) under `releases:`
python inference/run.py          # -> inference/output.jsonl  ({"question_id","answer"} per line)
python inference/evaluate.py     # official metric vs. inference/annotations.jsonl (our labels)

First run downloads WavLM-large (~1.2 GB) from the Hub; afterwards it runs fully offline (GPU ≈10–15 min for 530 questions, or CPU).

Windows / PowerShell: .venv\Scripts\activate can be silently blocked by the default script execution policy, after which pip installs into the wrong Python. Safest is to skip activation and call the venv's interpreter directly — .venv\Scripts\python.exe -m pip install -r inference\requirements.txt, then .venv\Scripts\python.exe inference\run.py. After activating, pip -V must show a path inside .venv.

GPU: install the CUDA torch build before the requirements (see the note in requirements.txt). If a CPU-only torch is already installed, pip will skip the CUDA build unless you force it: pip install torch --index-url https://download.pytorch.org/whl/cu128 --force-reinstall

To retrain the ranker from scratch (the full pipeline that produced inference/ranker.pkl):

pip install -r requirements.txt
PYTHONPATH=. python scripts/produce_submission.py   # source audio -> submissions/<team_id>.jsonl (530 lines)

Layout

inference/                      # ⭐ SELF-CONTAINED SUBMISSION: run.py (infer) + evaluate.py (score)
  ranker.pkl                    #    the pre-trained model (StandardScaler + LogisticRegression)
  annotations.jsonl             #    ground-truth answers ({"question_id","answer"} format)
  config.yaml requirements.txt README.md
scripts/produce_submission.py   # full pipeline: source data -> train ranker -> submission
experiments/                    # one-command reproduction of every table/finding in the report
empathyeval/                    # release parsing + the 16 kHz-mono audio loader + the official metric
configs/phase1.yaml             # data paths, audio, cache

For the competition, only inference/ is needed — it is fully self-contained (the pre-trained model plus a pure-inference script) and, delivered alone, regenerates the exact result. scripts/produce_submission.py is the full pipeline that trained the ranker saved as inference/ranker.pkl.