MIKU-PAL: An Automated and Standardized Multi-Modal Method for Speech
  Paralinguistic and Affect Labeling
Authors: name: Yifan Cheng, name: Ruoyi Zhang, name: Jiatong Shi

Acquiring large-scale emotional speech data with strong consistency remains a
challenge for speech synthesis. This paper presents MIKU-PAL, a fully automated
multimodal pipeline for extracting high-consistency emotional speech from
unlabeled video data. Leveraging face detection and tracking algorithms, we
developed an automatic emotion analysis system using a multimodal large
language model (MLLM). Our results demonstrate that MIKU-PAL can achieve
human-level accuracy (68.5% on MELD) and superior consistency (0.93 Fleiss
kappa score) while being much cheaper and faster than human annotation. With
the high-quality, flexible, and consistent annotation from MIKU-PAL, we can
annotate fine-grained speech emotion categories of up to 26 types, validated by
human annotators with 83% rationality ratings. Based on our proposed system, we
further released a fine-grained emotional speech dataset MIKU-EmoBench(131.2
hours) as a new benchmark for emotional text-to-speech and visual voice
cloning.