Title: Video Language Models are Human-Aligned Evaluators for Text to Motion Generation
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: text-driven human motion generation

Abstract: Recently, text-to-motion (T2M) has become a basic setting for human motion generation. This work studies the evaluation of alignment between text and generated motion, to credit the reliable use of T2M models. We consider solving the T2M evaluation task by making use of a video language model (VLM). Our basic idea is: render the generated human motion into a skinned video, and then use a VLM for evaluation. To address information loss problem when 3D motion is rendered into 2D video, we develop a method, which ensures reliable evaluation score by analyzing VLM entropy. Our evaluation method, named VeMo, frees T2M evaluation from reliance on motion data while seamlessly leveraging the semantic understanding and reasoning capabilities of advanced VLMs trained on Internet-scale data. To systematically compare the empirical usefulness of different evaluation methods, we manually annotate a meta-evaluation benchmark that includes coarse-grained alignment labels and fine-grained reasons. Extensive experiments and case studies demonstrate the effectiveness of VeMo.

TL;DR: The paper considers solving the T2M evaluation task by making use of a video language model, and provides a meta-evaluation dataset.