Reflecting Reality: Enabling Diffusion Models to Produce Faithful Mirror
  Reflections
Authors: name: Ankit Dhiman; name: Manan Shah; name: Rishubh Parihar; name: Yash Bhalgat; name: Lokesh R Boregowda; name: R Venkatesh Babu
Published: 2024-09-23 02:59:07

Abstract: We tackle the problem of generating highly realistic and plausible mirror
reflections using diffusion-based generative models. We formulate this problem
as an image inpainting task, allowing for more user control over the placement
of mirrors during the generation process. To enable this, we create SynMirror,
a large-scale dataset of diverse synthetic scenes with objects placed in front
of mirrors. SynMirror contains around 198K samples rendered from 66K unique 3D
objects, along with their associated depth maps, normal maps and instance-wise
segmentation masks, to capture relevant geometric properties of the scene.
Using this dataset, we propose a novel depth-conditioned inpainting method
called MirrorFusion, which generates high-quality geometrically consistent and
photo-realistic mirror reflections given an input image and a mask depicting
the mirror region. MirrorFusion outperforms state-of-the-art methods on
SynMirror, as demonstrated by extensive quantitative and qualitative analysis.
To the best of our knowledge, we are the first to successfully tackle the
challenging problem of generating controlled and faithful mirror reflections of
an object in a scene using diffusion based models. SynMirror and MirrorFusion
open up new avenues for image editing and augmented reality applications for
practitioners and researchers alike.