Title: Balancing Fidelity and Diversity: Synthetic data could stand on the shoulder of the real in visual recognition
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Synthetic data selection, Generative model, Image classification

Abstract: As generative models (GMs) advance, producing high-quality, photorealistic synthetic images is now feasible and increasingly common. Beyond use in entertainment, such synthetic data offers a promising solution to data scarcity in AI research. With the growing research interest in using synthetic data, a critical question arises: How can we maximize performance gains for downstream tasks using synthetic data? Fine-tuning or prompt engineering are two common strategies to fully exploit the potential of GMs to produce task-specific synthetic data. However, unlike these tedious optimizations, is it possible to exploit the data potential by selecting an optimal synthetic subset from a given pool? Motivated by this question, we propose an efficient data selection strategy to improve the utility of existing synthetic datasets without adjusting the GMs' output. Experiments on several benchmarks demonstrate that balancing the trade-off between fidelity and diversity in synthetic data benefits model performance and robustness. In summary, this paper presents a practical and scalable approach to harnessing synthetic data, particularly valuable in scenarios where customizing the outputs of generative models is infeasible.

TL;DR: Proposing scoring method, we select synthetic data for model training. Results show that balancing fidelity and diversity is key to unlock the potential of generative data in visual recognition.