MixEval-X: Any-to-Any Evaluations from Real-World Data Mixtures
Authors: name: Jinjie Ni; name: Yifan Song; name: Deepanway Ghosal; name: Bo Li; name: David Junhao Zhang; name: Xiang Yue; name: Fuzhao Xue; name: Zian Zheng; name: Kaichen Zhang; name: Mahir Shah; name: Kabir Jain; name: Yang You; name: Michael Shieh
Published: 2024-10-17 16:52:28

Abstract: Perceiving and generating diverse modalities are crucial for AI models to
effectively learn from and engage with real-world signals, necessitating
reliable evaluations for their development. We identify two major issues in
current evaluations: (1) inconsistent standards, shaped by different
communities with varying protocols and maturity levels; and (2) significant
query, grading, and generalization biases. To address these, we introduce
MixEval-X, the first any-to-any real-world benchmark designed to optimize and
standardize evaluations across input and output modalities. We propose
multi-modal benchmark mixture and adaptation-rectification pipelines to
reconstruct real-world task distributions, ensuring evaluations generalize
effectively to real-world use cases. Extensive meta-evaluations show our
approach effectively aligns benchmark samples with real-world task
distributions and the model rankings correlate strongly with that of
crowd-sourced real-world evaluations (up to 0.98). We provide comprehensive
leaderboards to rerank existing models and organizations and offer insights to
enhance understanding of multi-modal evaluations and inform future research.