Auto Cherry-Picker: Learning from High-quality Generative Data Driven by
  Language
Authors: name: Yicheng Chen; name: Xiangtai Li; name: Yining Li; name: Yanhong Zeng; name: Jianzong Wu; name: Xiangyu Zhao; name: Kai Chen
Published: 2024-06-28 17:53:18

Abstract: Diffusion-based models have shown great potential in generating high-quality
images with various layouts, which can benefit downstream perception tasks.
However, a fully automatic layout generation driven only by language and a
suitable metric for measuring multiple generated instances has not been well
explored. In this work, we present Auto Cherry-Picker (ACP), a novel framework
that generates high-quality multi-modal training examples to augment perception
and multi-modal training. Starting with a simple list of natural language
concepts, we prompt large language models (LLMs) to generate a detailed
description and design reasonable layouts. Next, we use an off-the-shelf
text-to-image model to generate multiple images. Then, the generated data are
refined using a comprehensively designed metric to ensure quality. In
particular, we present a new metric, Composite Layout and Image Score (CLIS),
to evaluate the generated images fairly. Our synthetic high-quality examples
boost performance in various scenarios by customizing the initial concept list,
especially in addressing challenges associated with long-tailed distribution
and imbalanced datasets. Experiment results on downstream tasks demonstrate
that Auto Cherry-Picker can significantly improve the performance of existing
models. In addition, we have thoroughly investigated the correlation between
CLIS and performance gains in downstream tasks, and we find that a better CLIS
score results in better performance. This finding shows the potential for
evaluation metrics as the role for various visual perception and MLLM tasks.
Code will be available.