Dev4IC/MEva-checkpoints

MEva evaluator checkpoints

Trained CNN evaluators for the ISMIR 2026 paper "Do Music Generative Models Understand Musical Qualities? Automatic Music Evaluation with Model-Intrinsic Signals" (Xiaosha Li, Chun Liu, Ziyu Wang).

• Code: https://github.com/YoEv/MEva
• Demo: https://yoev.github.io/MEva

Each checkpoint maps intrinsic signals of a frozen MusicGen (per-token loss, predictive entropy, and SAE latents under teacher forcing, delay pattern) to a human quality rating. 14 variants: 7 signal combinations × MusicGen-{small,large}.

Layout

per_benchmark/{small,large}/{musiceval,songeval,aime,musicpref,music_arena}/
    f01_loss_only_cnn_clean_best.pth
    f02_entropy_only_cnn_clean_best.pth
    f03_sae_only_cnn_clean_best.pth
    f04_loss_entropy_cnn_clean_best.pth
    f05_entropy_sae_cnn_clean_best.pth
    f06_loss_sae_cnn_clean_best.pth
    f07_loss_entropy_sae_cnn_clean_best.pth
pooled/{small,large}/          # trained on the union of the five benchmarks
    (same seven files)

f01–f07 follow the paper's experiment registry (single-signal, pairwise, and full hybrids). Per-benchmark models produce Table 1's per-benchmark columns; pooled/ models produce the All benchmarks column.

Usage

See the training/eval scripts — training/hybrid/train_cnn.py (architecture) and scripts/eval/eval_14_experiments.py (evaluation protocol).

Citation

@inproceedings{li2026meva,
  title     = {Do Music Generative Models Understand Musical Qualities? Automatic Music Evaluation with Model-Intrinsic Signals},
  author    = {Li, Xiaosha and Liu, Chun and Wang, Ziyu},
  booktitle = {Proc. of the 27th Int. Society for Music Information Retrieval Conf. (ISMIR)},
  year      = {2026}
}