InterFeedback: Unveiling Interactive Intelligence of Large Multimodal
  Models via Human Feedback
Authors: name: Henry Hengyuan Zhao; name: Wenqi Pei; name: Yifei Tao; name: Haiyang Mei; name: Mike Zheng Shou
Published: 2025-02-20 20:27:06

Abstract: Existing benchmarks do not test Large Multimodal Models (LMMs) on their
interactive intelligence with human users which is vital for developing
general-purpose AI assistants. We design InterFeedback, an interactive
framework, which can be applied to any LMM and dataset to assess this ability
autonomously. On top of this, we introduce InterFeedback-Bench which evaluates
interactive intelligence using two representative datasets, MMMU-Pro and
MathVerse, to test 10 different open-source LMMs. Additionally, we present
InterFeedback-Human, a newly collected dataset of 120 cases designed for
manually testing interactive performance in leading models such as OpenAI-o1
and Claude-3.5-Sonnet. Our evaluation results show that even state-of-the-art
LMM (like OpenAI-o1) can correct their results through human feedback less than
50%. Our findings point to the need for methods that can enhance the LMMs'
capability to interpret and benefit from feedback.