Anything in Any Scene: Photorealistic Video Object Insertion
Authors: name: Chen Bai; name: Zeman Shao; name: Guoxiang Zhang; name: Di Liang; name: Jie Yang; name: Zhuorui Zhang; name: Yujian Guo; name: Chengzhang Zhong; name: Yiqiao Qiu; name: Zhendong Wang; name: Yichen Guan; name: Xiaoyin Zheng; name: Tao Wang; name: Cheng Lu
Published: 2024-01-30 23:54:43

Abstract: Realistic video simulation has shown significant potential across diverse
applications, from virtual reality to film production. This is particularly
true for scenarios where capturing videos in real-world settings is either
impractical or expensive. Existing approaches in video simulation often fail to
accurately model the lighting environment, represent the object geometry, or
achieve high levels of photorealism. In this paper, we propose Anything in Any
Scene, a novel and generic framework for realistic video simulation that
seamlessly inserts any object into an existing dynamic video with a strong
emphasis on physical realism. Our proposed general framework encompasses three
key processes: 1) integrating a realistic object into a given scene video with
proper placement to ensure geometric realism; 2) estimating the sky and
environmental lighting distribution and simulating realistic shadows to enhance
the light realism; 3) employing a style transfer network that refines the final
video output to maximize photorealism. We experimentally demonstrate that
Anything in Any Scene framework produces simulated videos of great geometric
realism, lighting realism, and photorealism. By significantly mitigating the
challenges associated with video data generation, our framework offers an
efficient and cost-effective solution for acquiring high-quality videos.
Furthermore, its applications extend well beyond video data augmentation,
showing promising potential in virtual reality, video editing, and various
other video-centric applications. Please check our project website
https://anythinginanyscene.github.io for access to our project code and more
high-resolution video results.