Mind-the-Glitch: Visual Correspondence for Detecting Inconsistencies in
  Subject-Driven Generation
Authors: name: Abdelrahman Eldesokey, name: Aleksandar Cvejic, name: Bernard Ghanem, name: Peter Wonka

We propose a novel approach for disentangling visual and semantic features
from the backbones of pre-trained diffusion models, enabling visual
correspondence in a manner analogous to the well-established semantic
correspondence. While diffusion model backbones are known to encode
semantically rich features, they must also contain visual features to support
their image synthesis capabilities. However, isolating these visual features is
challenging due to the absence of annotated datasets. To address this, we
introduce an automated pipeline that constructs image pairs with annotated
semantic and visual correspondences based on existing subject-driven image
generation datasets, and design a contrastive architecture to separate the two
feature types. Leveraging the disentangled representations, we propose a new
metric, Visual Semantic Matching (VSM), that quantifies visual inconsistencies
in subject-driven image generation. Empirical results show that our approach
outperforms global feature-based metrics such as CLIP, DINO, and
vision--language models in quantifying visual inconsistencies while also
enabling spatial localization of inconsistent regions. To our knowledge, this
is the first method that supports both quantification and localization of
inconsistencies in subject-driven generation, offering a valuable tool for
advancing this task. Project
Page:https://abdo-eldesokey.github.io/mind-the-glitch/