Refine-by-Align: Reference-Guided Artifacts Refinement through Semantic Alignment
Authors: Yizhi Song
Liu He
Zhifei Zhang
Soo Ye Kim
He Zhang
Wei Xiong
Zhe Lin
Brian L. Price
Scott Cohen
Jianming Zhang
Daniel Aliaga
Venue: ICLR 2025 Poster (2025)
Keywords: diffusion model; inpainting; generative artifacts; image editing; image synthesis; artifacts refinement
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Personalized image generation has emerged from the recent advancements in generative models. However, these generated personalized images often suffer from localized artifacts such as incorrect logos, reducing fidelity and fine-grained identity details of the generated results. Furthermore, there is little prior work tackling this problem. To help improve these identity details in the personalized image generation, we introduce a new task: reference-guided artifacts refinement. We present Refine-by-Align, a first-of-its-kind model that employs a diffusion-based framework to address this challenge. Our model consists of two stages: Alignment Stage and Refinement Stage, which share weights of a unified neural network model. Given a generated image, a masked artifact region, and a reference image, the alignment stage identifies and extracts the corresponding regional features in the reference, which are then  used by the refinement stage to fix the artifacts. Our model-agnostic pipeline requires no test-time tuning or optimization. It automatically enhances image fidelity and reference identity in the generated image, generalizing well to existing models on various tasks including but not limited to customization, generative compositing, view synthesis, and virtual try-on. Extensive experiments and comparisons demonstrate that our pipeline greatly pushes the boundary of fine details in the image synthesis models.

Published as a conference paper at ICLR 2025
BEFINE-B-AALIGH REFERENCE-GUIDED ARTIFACTS REFINEMENT THROUGH SEMANTIC ALIGNMENT
Yizhi Song'
Liu He
Zhifei Zhang Scott Cohen Adobe Research
Soo Ye Kim?
He Zhang?
Wei Xiong?
Zhe Lin
Brian Price*
Jianming Zhang² Daniel Aliaga
Purdue University
a. Refine Text-to-Image Customization
b. Refine Virtual Try-On Result
Refine Object Composition Result
Batter
Ban Appotil
Ba Appita
d. Refine Novel-View Synthesis Result
Image artifacts
Reference
Refined by model
Image artifacts
Reference
Refined by model
Figure Refine-by-Align. Given a generated image (with artifacts), free-form mask indicating the artifacts region in the generated image, and high-quality reference image containing important details such as identity logo or font, our model can automatically refine the artifacts in the gener- ated image by leveraging the corresponding details from the reference. The proposed method could benefit various applications (e.g., DreamBooth (Ruiz et al., 2023a) for text-to-image customization, IDM-VTON (Choi et al., 2024) for virtual try-on, Any (Chen et al., 2023) for object composi- tion, and Zero 1-to-3++ (Shi et al., 2023b) for novel view synthesis.
ABSTRACT
Personalized image generation has emerged from the recent advancements in gen- erative models. However, these generated personalized images often suffer from localized artifacts such incorrect logos, reducing fidelity and fine-graing -raigrainn iden- tity details of the generated results. Furthermore, there is little prior work tackling this problem. To help improve these identity details in the personalized image generation, we introduce a new task: reference-guiee present Refine-by-Align. first-of-its-kind model that employs diffusion-based framework to address this challenge. Our model consists of two stages: Align- ment Stage and Refinement Stage, which share weights unified neural net- work model. Given a generated image, masked artifact region, and a reference image, the alignment stage identifies and extracts the corresponding regional fea- tures in the reference, which are then used by the refinement stage to fix the ar- tifacts. Our model-agnostic pipeline requires no test-time tuning or optimization. It automatically enhances image fidelity and reference identity in the generated image, generalizing well to existing models on various tasks including but not
artifacts refinement. We
Thiis work done while the author intern Adohe Research.

---

Published as a conference paper at ICLR 2025
limited to customization, generative compositing, view synthesis, and virtual try- on. Extensive experiments and comparisons demonstrate that our pipeline greatly pushes the boundary of fine details in the image synthesis models. Project page: hwtp:///oiiiitu.o/oeeeeeeeee -iiiiee----
INTRODUCTION
Generative models (Goodfellow et al., 2014; Karras et al., 2019; He & Aliaga, 2024; 2023) for image synthesis (Ho et al., 2020; Rombach et al., 2022: Peebles & Xie, 2023; Podell et al., 2023; Luo et al., 2023b) have made significant advancement. Moreover, the traditional task of reference- guided image generation (Sheng et al., 2022; 2023; 2024) has been enabled by recent diffusion models (DM), where a text and/or visual prompt is provided and the subject object is generated in specified novel context. This ability has been widely applied applications such as subject customization (Ruiz et al., 2023a), object composition (Chen et al., 2023), novel view synthesis (Shi al., 2023b) and virtual try-on (Choi et al., 2024). While these works seek generation in a single step, in practice undesired blemishes, detail omissions, and blurriness may occur in the generated images. These localized unpleasant anomalies are typically called artifacts as perceived by human eyes (e.g.. "artifacts" row in Figure 1 (Zhang et al., 2023b)). The artifacts reduce image fidelity and the overall prompt-alignment quality of the synthesized images. Thus, a localized refinement tool to remove or reduce artifacts is beneficial.
Recently, a few limited approaches to artifact detection and refinement have been presented. PAL (Zhang et al., 2023b) presents an early work in this area that trains artifact detection model in a supervised end-to-end manner using input images with artifacts and corresponding ground truth images. The detected artifacts be partially removed using pre-trained image inpainting tool. As PAL identifies, the artifacts are typically very small and irregular-shag further complicates the refinement process. PAL ameliorates artifacts but struggles with diversity of artifact refinement. Lack of diversity is also observed in RealisHuman (Wang et al., 2024a) that focuses on artifacts human hands and faces, and the SynArtifact (Cao al., 2024) using vision- language model and reinforcement learning for artifact annotation and removal. In general, none of these methods are able to provide a controllable and predictable artifact refinement output with free-form support automatically to precisely preserve the original identity details.
(Appendix Fig. 8), which
Our main approach leverage the identity detail info in the reference-image to guide the refine the artifacts. This provides contoly-c output (i.e., we specify the desired refinement), works for urbitrarily-shaper artifacts, preserves the identity and background the provided generated im- age, and is applicable to multiple image generation approaches. As shown in Fig. 1, we provide stfrennce-guided artifacts refinement: Given generated image with marked artifact regions, and reference image (containing reference object), our model refines the artifacts by transferring corresponding details from the reference image to the artifact regions in the originally generated image. Our reference- guided approach shares insights with many reference- baasdd image customiza-
(4
Image with Artitacts
DIFT
DHFF
Ours
Imago with Artitacts
DIFT
DNF
Ours
Figure 2: Comparisons of our region-matching method with keypoint matching. We utilize DIFT (Tang et al., 2023) and DHF (Luo et al., 2023a) to perform keypoint matching from the ar- tifacts region (10 points sampled along the artifacts contour) to the reference. DIFT and DHF often fail to find the accurate corresponding region; in addition, they have trouble in distinguish- ing between repeating patterns such (a)(c). In contrast, method is more robust. The results demonstrate that artifacts alignment is a non-trivial process.
2

---

Published as a conference paper at ICLR 2025
tion models (Song et al., 2023; Chen et al., 2023; Yang et al., 2023), but those models overlooked between reference image and artifact region. In Fig. 2. SOTA models
the accurate specialized on finding correspondences may fail on key point matching for arbitrary- regions.
we design a novel alignment algorithm (Algorithm 1) for arbitrarily-happe regional feature matching which utilizes the artifact regions to query corresponding features from the reference image. In particular, the best match is obtained by maximizing the spatial correlation between DM-captured cross-attention maps of both artifact region and the reference image. (2) Refinement Stage: we train a diffusion model to use the best matched features from the alignment stage to refine the artifact region and preserve identity. Our model is trained in self-supervised scheme and we demonstrate its application to artifacts generated by various generative models. Quantitative and qualitative comparisons (i.e., using several well- established metrics and a user study; Sec. 4.2, Sec. 4.3) show that in terms of detail and appearance preservation our model outperforms all six baseline models (Paint-y--Eaampe
Our approach proceeds in two stages (Fig. 3). (1) Alignment Stage:
(Yang et al., 2023), ObjectStitch (Song et al., 2023), AnyDoor (Chen et al., 2023), PAL (Zhang et al., 2023b), Cross- Image Attention (Alaluf et al., 2024) and MimicBrush (Chen et al., 2024)).
Our contributions can be summarized as follows:
A first-of-its-kind generative artifacts refinement framework supporting control via refer- ence image specification, identity preservation, arbitrary artifact shapes, and good fidelity.
A novel artifacts matching algorithm which matches urbitrarily-shape artifacts corre- sponding patterns in the reference image.
An effective reference-guided refinement strategy that ameliorates artifacts in provided generated image.
A comprehensive benchmark, GenArtifactBench, consisting of artifacts generated by sev- eral well-known models, reference images, and dense human annotations which can serve to evaluate future efforts this area.
2
RELATED WORK
2.1 REFERENCE-GUIDER IMAGE EDITING
Reference-guided image editing has been a traditional task that has various applications, including reference-guides super resolution (Zhang et al., 2019; Jiang et al., 2022), and guided inpainting or outpainting (Zhou et al., 2021; Tang et al., 2024a). With the significant advancements in diffusion models (DM) (Ho et al., 2020; Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Rombach et al., 2022; Ho & Salimans, 2022; Peebles & Xie, 2023; He et al., 2023; Huang et al., 2023) in text-to- image (T2I) synthesis, there have been many works subject-driven image editing. The notion of using an additional reference image to guide image editing (e.g.. (Ruiz et al., 2023a; Kawar et al., 2023)) has led series of techniques (Ruiz al., 2023b; Gal et al., 2022: Shi et al., 2023a; Kumari et al., 2023; Liu et al., 2023b) using optimization to learn concepts. In spite of their high-fidelity editing results, they usually require inference time fine-tuning or multiple subject images. Another branch of works (Yang et al., 2023; Song et al., 2023; Chen et al., 2023; Zhang et al., 2024b; Li et al., 2024; Xiong al., 2024) replace the text embedding of T2I models with image embedding based on CLIP or DINOv2 (Radford et al., 2021; Oquab et al., 2023; Song et al., 2024), which are tuning-free. Subsequent works have extended the applications to image compositing (Lu et al., 2023; Wang et al., 2024b; Avrahami et al., 2022; Sarukkai et al., 2024; Meng et al., 2021), novel view synthesis (Liu et al., 2023a; Shi et al., 2023b; Liu et al., 2024b), instruction-guided editing (Pan et al., 2023; Yu et al., 2024; Tang et al., 2022) via Large Language Models (Liu et al., 2024a; Zhang al., 2024a) or Vision-Language Models (He et al., 2024; Hua et al., 2024b;a; Tang et al., 2024c;b), and subject swapping (Gu et al., 2023; 2024). However, they all face critical challenge of controllability and identity preservation of the original object.
2.2 CORRESPONDENCE MATCHING
Traditional methods use purefull-ddesinnee features (Bay et al., 2006; Lowe, 2004; Rublee et al., 2011) 1) basdd approaches (Aberman et al., 2018; Rocco al., 2018 8; Seo et al., 2018;
3

---

Published as a conference paper at ICLR 2025
Training
rotay trage Sq
Dundbed Irago
emtedding S
Oriinel lpst
Antlidds rond 14
Complia rnfersoo
Parial vafronnce
U-- grrmennt vodk
Aigerttre
akommnn
Image n ertlads f,
Arbhic madk Mr
Refernnce
Relerrcc entreddnng
map Al"
relernce
Localond rolerence
1. C
U--
Inference
Inference
(Alignment Stage)
Partal roliy Yrogo to
(Refinement Stage)
Rolieed Irage 2
During training, we train a DM for object completion, guided by reference image Lr. In alignment mode, the reference model learns to locate the relevant region from the reference for object completion, thus maximizing the spatial correlation attention maps. In refinement mode, this region is directly provided as reference. Bottom: During inference, the inputs include generated image Ia with the artifacts marked as Ma. and a reference object 7,- In the alignment stage, we perform cross-attention alignment algorithm (see Alg. and Fig. 4) to find the correspondence map M In the refinement stage, M is used to find the region in I, that corresponds to artifacts, which guides refining to Ia.
Figure 3: Overview of our framework. Top:
complete object, so the
Simonyan et al., 2014) to establish correspondences. Recent work has shifted towards adapting dif- fusion models. Tang et al. (2023); Luo al. (2023a); Zhang et al. (2023a); Hedlin al. (2023) leverage the features maps or embeddings of pretrained DMs to predict correspondences. Appear- ance transfer (Go et al., 2024; Chen al., 2024; Alaluf et al., 2024) is downstream application of such task. However, they still suffer from the loss of fine-grane fine-grained identity details from the reference.
2.3 LOCALIZATION AND REFINEMENT OF ARTIFACTS
challenging for the state-of-the-an models to capture intricate details, such as object textures and human hands. In Zhang et al. (2023b), these artifacts are defined implausible content or display of unpleasant artifacts in specific regions of the image. Although artifacts are commonly observed in the leading generative models, few works have explored the topic of artifacts detection and refinement. Recently, Zhang et al. (2023b) curate a numan--nnotatee dataset for artifact segmentation, train segmentation model for artifact localization and zoom-in inpainting pipeline for refinement. Cao et al. (2024) classify the artifacts and fine-tune T21 DMs to reduce artifacts. Specifically handling artifacts on human body, Wang al. (2024a) propose an approach to refine artifacts in faces and hands. Concurrently, Chen et al. (2024) propose MimicBrush for appearance transfer, which is a potential use for artifact refinement. However, none of these works are designed to perform reference guided artifacts refinement, thus lacks control. For the first time, we provide solution for guided artifacts refinement with fine-grained control.
METHOD
The proposed artifacts refinement framework, Refine-by-Align is shown in Fig. Formally, we define the task of artifacts refinement conditioned on a reference image following: Let T E H x3 be an image with artifacts generated by any reference-guinn guided generation model, M, R HWW be user provided artifacts mask, I, I2 f/ x3 be the segmented reference object image; we train Diffusson Model base refinement model to generate refined image I::
M R(Ia: Ma, I,) e B #xWx3
(1)
Ideally Ig the refined area I @ M,, should be consistent with the background and preserve the identity from the reference object I, the whole image T should appear natural and artifacts-free.
4

---

Published as a conference paper at ICLR 2025
Hadamard product X
Reshape
Reference embedeing K R
M R3
Image @ E
mosk
Mf loxts
Comeepondence map
M2
Visualization of Algorithm
Image artifacts Correspondence maps across all transformer loyers
Reference
1=3
1=5
1=7
1=9
1=11
1=13
artifactt
Reference
1=9
Carrespondence maps across all time steps (at layer 9)
Figure 4: Top: Visualization of our cross-attention alignment algorithm. The artifacts mask is used to extract the spatial correlations between the artifacts and the reference; the output of this algorithm, the correspondence map, indicates the region in the reference that corresponds to the artifacts area. Middle and Bottom: Correspondence maps across different transformer layers and timesteps.
mieU scross Emesseps
miBU across lnyers
aiessoppo
Figure 5: Running the cross-attention alignment algorithm on GenArtifactBench to find the best combination of timestep and transformer layer l. Left: mloU across all timesteps, averaged over all layers and images; Right: mloU across all layers, averaged over all timesteps and images.
However, directly using I, as the visual guidance may cause significant degradation, Hence, an optimal correspondence mask M is necessary for an optimized input reference as I, & M for better guidance. To tackle this task, we created a two stage approach R: the Alignment Stage and the Refinement Stage, which share the weights of a single unified neural network model. In the alignment stage, the artifacts region La @ M, is used as the query to localize the optimal correspon- dence region mask M in L,, In the refinement stage, 1, @ A1* is fed into DINOv2 (Oquab et al., 2023) encoder to extract expressive visual features, guiding the local generation process to eliminate the artifacts while ensuring identity preservation.
Our design motivated by two key observations:
Refine by localization: As summarized (Zhang et al., 2023b), generative artifacts occurs more frequently around tiny object details such as logos, texts and other complex textures. However, these areas have higher fidelity when such details are prominent the image. Based on this observation, assume that the fidelity can be improved by performing local generation guided by a local region from the reference.
Align via cross-attention: As demonstrated in Hertz et al. (2022), the appearance of the generated image depends the interaction between the pixels the text embedding, which occurs in cross attention layers. Similarly, when replacing the text embedding with image embedding, we assume that spatial correspondence exists between the generated image and the image embedding.
Based on the above observations, review cross-attention and prove that the spatial correspon- dence exists in Sec. 3.2; the algorithm to align I, @ M,, to the corresponding local region in I, is explained in Sec. 3.3. Sec. 3.4 describes the two training modes. Sec. 3.5 describes the inference.
5

---

Published as a conference paper at ICLR 2025
Algorithm Optimal Cross-Attention Alignment (refer to Fig. 4 for visualization)
Input:
Target image I, resized artifacts mask Ma E Ig reference image 4, DINOV2 encoder Φ, the refinement model R, the ground truth correspondence mask Mgt on I,-
Parameter: The diffusion time steps T', total number of transformer block L in R, the noisy image resolution d in latent space.
Output: Optimal correspondence mask M (for I,).
I: E MLP()(I,)): E I x66
# Get reference embedding
2: 27 ~ (O0 [); Zero array I E IRTX
3: for T 1,.... ,0 do
4: for 1 0, 0,......................... ..-.....11111
5:
A R(z,,,, E); A R(dxd)x256 M = En, A (i,j.))
Extract cross-attention map at I-th block # 2D correspondence mask on I,-
6:
7: Te m Mgt)
# Calculate mloU
8: end for
9:
end for
10: 1",1" <- arg max;,(()
. Find optimal and /
11: return M +- M
3.1 DIFFUSION MODEL
We leverage the architecture of AnyDoor (Chen et al., 2023) and IMPRINT (Song et al., 2024) as the backbone of our refinement network. It is based on the Stable Diffusion (Rombach et al., 2022) model, which contains three major components: the variational autoencoder (VAE) to code images into latent embeddings 2, the U-Net backbone R parameterized by 0 for sequentially denoising dif- fusion steps on z by Gaussian noise e, and a text encoder for guidance injection. We replace the text encoder by vision encoder ø based on pretrained DINOv2 (Oquab et al., 2023) to enable visual prompt guidance by a reference image I, E IR xxxxxxx of a subject. particular, the conditional embedding o(I,) will be utilized for optimization of loss functions on artifact refinement:
Cartifact -2, [|I=
Rø (zt, t, a (1,))
(2)
where 2 the latent embedding time step t. R trained iteratively denoise žr 2g-
3.2 SPATIAL CORRESPONDENCE IN ENCODER DACSEER-BASED CUSTOMIZATION MODELS
Our vision encoder & based on DINOv2 is employed to extract the image embedding from the reference object, which then fed to the cross attention layers of the U-Net backhone. ViT backbone divides the input image into 16 x 16 square patches, thus the encoded o(4,) is sequence of 256 tokens and can be mapped back to 2D space with original spatial information as followed:
E = (TT); E E 256xd
In each attention layer, (l,,) is projected the key & v/a (E) and the value (.(E) by the linear projections 4/k and th. Meanwhile, the noisy image encoded by VAE to latent embedding is projected to the query q (2x). Specifically, 2 E Raxe where d {64, 32, 16, 8) is the resolution of the extracted image features at different depth of layers, and e is the number of channels, 2 also contains spatial information. Altogether, cross attention map A calculated as:
A = softmar gk" A (ddxd)x256
(4)
Intuitively, A measures the similarity between q and k, and A(r.y, stores the amount of informa- tion flow from the kth reference token to the latent pixel at (x, v) (Xiao et al., 2023). Therefore, through cross attention, A effectively encodes the interaction between the noisy image and the ref- erence embedding connected by spatial correlations.
3.3 THE ALIGNMENT ALGORITHM
Although the cross attention map A encodes the spatial correlations between I, and I,, two chal- lenges remain in identifying M* : (1) given A, accurately locating the region in the reference image

---

Published as a conference paper at ICLR 2025
that corresponds to the free-form artifacts La & Mai (2) finding the optimal correspondence map M from the numerous maps M of all possible combinations of diffusion timesteps and transformer present our cross-attention alignment algorithm,
layers L. Aiming at these two challenges, shown as pseudo code in Algorithm 1, and visualized in the top part of Fig. 4.
Optimal Cross-Attention Alignment
Since 2, is obtained by encoding a partially noisy version of la (see Sec. 3.4), each pixel ji can be mapped to its spatial correspondence in 2z. Given that A encodes the correlation between the noisy image 2, and the reference tokens E (discussed in Sec. 3.2), it can be concluded that any pixel Ia [ can find its match in E via A.
For the correspondence map of I, at diffusion timestep t. and transformer layer I, we accomplish this by aggregating all pixels belonging to the artifacts:
M
E(Ma o i(i
(5)
where the Hadamard product is calculated bétween the artifacts mask and the cross-attention map. Intuitively, the 2D correspondence map M measures the similarity between all reference tokens and the artifacts pixels. To obtain the optimal M some post-processing needs to be applied on M Refer to Sec. A.4 for more details.
A grid search over all possible i, values is performed on our proposed benchmark (see Sec. 1) and evaluated using mloU. The results are shown in Fig. 5, demonstrating that the optimum combination t 24 and = 9. However, during test time, we choose 0, 9 to accelerate the inference. Detailed comparisons can be found in Sec. 4.4.
3.4 TRAINING
For training, we implement two modes sharing weights of the same model, where the only difference lies in the inputs. In both modes, we train the U-Net, and the MLP connecting DINOv2 and U-Net.
Alignment Mode
To maximize the spatial correlations contained in the cross-attention maps, we design a fully self- supervised training scheme. The idea behind the construction of training pairs is: Use a complete reference object to guide object completion, forcing the model to learn to identify the corresponding local region from the reference.
We use Pixabay (Song et al., 2023) with panoptic segmentation labels as the training dataset. Fol- lowing the aforementioned idea, given Pixabay image I, and an object mask M,, we start by applying random mask generator to sample free-form artifacts mask M2 G(M,) within M,. We then design color perturbation S and affine transformations T on To s M, simulate the lighting and view changes in the real world: 4, S(T (Lo & Mo)). In this mode, the artifacts image Ia = to x (1 - Mo) and the perturbed reference I, are used as the inputs, and lo is the target.
Refinement Mode
In this mode, it is assumed that the local region corresponding to the artifacts is already given from the original reference, which is used as I, to guide the inpainting of the incomplete object.
Our training data consists of Pixabay and MVObj. a manually annotated dataset where an object appears in multiple images with different contexts and views. MVObj is added since pertur- bations S, T are not sufficient to simulate non-rigid or 3D pose changes. 1) When dealing with a Pixabay image,
obtain the partial reference using the artifacts mask: I, = S(T(I, Ma)); 2) when processing a pair of MVObj images (Jat, Toz, Moi, Ma2), Ia & (ro1e(Ma1) is used as the reference, and T,2
(erode( Ma2)) the input artifacts image Ia, where the mask is eroded.
We then adopt the zoom-in inpainting strategy from Zhang et al. (2023b), cropping around Ia and perform inpainting on the zoom-in patch.
3.5 INFERENCE
Inference is performed two stages, corresponding to the above two modes. In the alignment stage, given Ta, Ma, Ir, our fine-tuned model R. follows Alg. and produces the correspondence map
7

---

Published as a conference paper at ICLR 2025
Table I: Quantitative Comparison with the baseline models GenArtjfactBench. РьE (Paint- Xy-Example) and OS (ObjectStitch) are finctuned on our training set; the local regions (instead of the complete reference) are provided to PbE, OS and AnyDoor. Our method outperforms the other baselines in fidelity. Since CLIP-T only captures high-level semantics, it cannot accurately measure the detail preservation. We show additional comparisons in Tab. and Fig. 6.
Categories
Methods
CLIP-T
CLIP-I1
DINO-I
Paint-by by Example*
23.5938
80.7500
58.7625
Reference- guided inpainting
ObjectStitch"
24.4844
83,9375
72.6152
AnyDoor
25.0625
83,4375
71.3398
Text-guided inpainting
PAL
25.8906
81,5000
53.8117
Cross- lross-Image Attention
23.2500
80.2500
56.7892
Appearance transfer
MimicBrush
25,0156
85.0625
67.6194
Ours
25.4063
86.6250
75.3135
M Note that the grid search is skipped and R only proceeds one timestep. directly producing the optimal correspondence map M timestep 0 and layer The local region that corresponds the artifacts can be extracted from the reference as I, @ M
In the refinement stage, I, @ M is provided as the reference through DINOv2, guiding the refine- ment of la via inpainting the artifacts region.
4 EXPERIMENT
4.1 EVALUATION BENCHMARK
Dataset. provide insight on the appearance of generative artifacts and an effective evaluation of our artifacts refinement approach, we present GenArffacctBenc the first benchmark for reference- guided artifacts refinement (refer to the Appendix for examples), featuring:
We generate images using four popular models: DreamBooth (Ruiz et al.,, 2023a), 7erol23++ (Shi et al., 2023b), AnyDoor (Chen et al., 2023) and IDM-VTON (Choi et al., 2024), covering real- world applications of T2I customization, view synthesis, compositing and virtual try-on. Diverse artifacts are shown in these images.
We collect 146 generated images with notable artifacts, paired with 146 reference images of more than 40 objects (from DreamBooth and Pixabay) and 27 garments (from Choi et al. (2021)).
Human annotation of the artifacts, and the corresponding regions from the reference images.
To measure the semantics and identity preservation of refined object, we utilize CLIP- Image Score (Radford et al., 2021) and DINO Score (Oquab et al., 2023) to compute the similarity between the refined region of the generated image and its corresponding region from the reference. When calculating Texp-Text Score, BLIP2 (Li et al., 2023) is used to generate captions. Since measurement of the overall quality is absent, we further conduct a user study.
Metrics.
Parameters. / = and | are used in all the comparisons below.
4.2 QUANTITATIVE COMPARISONS
To demonstrate the effectiveness of our model, we compare our model with 6 baseline models (PbE or Paint-by- by-Exmplle (Yang al., 2023), OS ObjectStitch (Song et al., 2023). AnyDoor (Chen et al., 2023), PAL (Zhang et al., 2023b), Cross-Image Attention (Alaluf et al., 2024) and Mim- ieBrush (Chen et al., 2024), which is concurrent work) on GenArtifactBench using the aforemen- tioned metrics. For fair comparison, РЫЕ and OS are fine-tuned our training dataset.
Quantitative comparisons are shown in Tab. 1, where the baselines are categorized into three classes. Note that we provide the accurate reference regions for РbЕ, OS and AnyDoor, greatly reducing the difficulty for these models. In terms of semantics preservation measured by CLIP-T, PAL shows a slight advantage over model; however, as illustrated in Fig.6, 6, it fails gain fine- -eeegrained control over the details in local editing. This is because PAL is built on SDXL (Podell et al., 2023), which is only performing inpainting following high-level text description. CLIP-I and DINO score
8

---

Published as a conference paper at ICLR 2025
ETLOVE
Imsto with artiott
OROW
Pelereroo and the reae Paato-edd Dagion (TARGET
GROW UA
ET
Oura
GROW
MinDruthh
GRONS
GRO KB
Okyeatttch
CMVMO
AnyDoor
STLOVE
PAL
GROW
Pam-syCample
STLOVE
LEROT
BILeVe
Close image Amtrttinn
GIDUA
CIOU
Figure 6: Qualitative comparisons. Zoom in view details. Note that the accurate reference regions corresponding to the artifacts (not the complete reference) are provided to РbE, OS and AnyDoor. the second row of the references, we overlay the correspondence maps on them. Com- pared with the baselines, model not only preserves identity (most similar to the second row), but also generate smooth and natural results where artifacts are significantly reduced.
computed only over the masked region. Our model outperforms all baseline models in details and appearance preservation. Owning to our correspondence matching strategy, irrelevant feature is removed and consequently the identity is significantly improved.
To take human perception into account as well as adding quality metrics, we conduct a user study on Amazon Mechanical Turk. We design two questions to assess identity preservation and generation realism, respectively. In each question, side-by-side comparisons result alongside one selected from a baseline. it important to note that the reference images remain hidden from the user when assessing quality. Each question has 240 comparisons, and 720 votes collected from more than 150 workers. The user preference is reported in Tab. 2. The preference rate demonstrates the superiority of our model over all baselines in both fidelity and realism.
presented to the user: our

---

Published as a conference paper at ICLR 2025
Table 2: User study. Results are user preference win rate (%). Two questions are designed to measure identity preserving and realism; in each question the user is presented side-by-side results of our model and random baseline. For fairness, PbE and OS are fine-tuned on our training dataset.
Identity
Realism
Ours
70.83
29.17
Ours 71.67
28.33
Ours
55,83
ObjectStitch"
44.17
Ours
56.67
ObjectStitch"
43.33
Ours
57.50
AnyDoor
42.50
Ours
60.00
AnyDoor
40.00
Ours
74.17
Cross-Image Attention
25.83
Ours
83.33
Cross-Image Attention
16.67
Ours
61.67
PAL
38.33
Ours
71.67
PAL
28.33
Ours
60.00
MimicBrush
40.00
Ours 59.17
MimicBrush
40.83
4.3
QUALITATIVE RESULTS
Qualitative comparisons with the baseline models are shown in Fig. The correspondence maps obtained by our model are overlaid on the reference image, highlighting the local regions that match the artifacts. When testing on РbE, OS and AnyDoor, these local regions (instead of the complete reference objects) are directly provided as input. In particular, РbE. OS and PAL struggl in preserv- ing the finer details from the reference. This especially the case for complex patterns since they only have high-level semantic control over the generation. AnyDoor can capture identity but fails to generate smooth transition areas. MimicBrush, concurrent work, is designed for reference- guided local editing which shows superiorities over the other baselines; however, our model outperforms MimicBrush in both realism and fidelity.
4.4 ABLATION STUDY
Diffusion timestep and transformer layer. To evaluate the accuracy of the correspondence maps
Mr., we ablate on all timesteps and layers & E {0, (0..,..,, 49}; / E {0, 10,1,.., 15). and compute the mloU over all images from GenArtifactBench, which is shown in Fig. 4. We also show 2D mloU figure on all combinations of t and in the Appendix (Fig. 7), visualizing I E RTxL, To balance between efficiency and accuracy. we choose 0, 9.
Comparisons with keypoint matching. We compare our alignment algorithm to keypoint matching performance by two correspondence matching methods: DIFT (Tang et al., 2023) and DHF (Luo et al., 2023a). Visual results are in Fig. 2. While baselines struggle with repeating or irregular patterns, our alignment algorithm is more robust, with only a single query to locate the target region. Furthermore, the alignment and refinement stages are integrated into unified model.
Ablation on model design. To demonstrate the effectiveness of our architecture design, we compare our full model with two settings (Tab. 3). 1) the model is only trained in the alignment mode, where a complete object is used as the reference. When the irrelevant patterns have been removed from the reference, identity preservation is significantly improved; 2) DINOv2 encoder is replaced by CLIP. The comparison proves that CLIP fails to encode low-level details, thus losing identity.
5 LIMITATIONS, CONCLUSION AND FUTURE WORK
We have shown a novel approach to artifact re- finement via region aligmment and reference- guided generation. Our approach makes use of a high-quality reference image provide predictable and controllable refinement output, that also preserves identity and transfers the de- tails from the reference image to the input im- age containing artifacts. Our method has been compared to several baseline methods and has shown consistently superior performance. As limitations, first, our method does not always work well when there is a large disparity between the objects in the original input image and in the ref- erence. Second, the accuracy of our alignment method is limited by the number of vision tokens, where only 16 16 patches used to represent an image. As future work, we would like to extend our method to automate artifact detection and to incorporate the use of multiple reference images and text-descriptions as obtain blended outputs.
Table 3: Ablation Study on two model designs.
1) using a model which is only trained in the align- ment mode to perform single stage artifacts refine-
ment; 2) DINOv2 is replaced by CLIP encoder.
CLIP-T f
CLIP-I
DINO f
Single-stage CL.IP-encnder
24.4375
84,5000
71.7609
24.3750
84,3750
70.7714
Full
25.4063
86,6250
75.3135
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research partially funded by NSF Grant 2107096, 1835739 and 24 1273.
REFERENCES
Kfir Aberman, Jing Liao, Mingyi Shi, Dani Lischinski, Baoquan Chen, and Daniel Cohen-Or. Neural best-buddies: Sparse cross-domain correspondence. ACM Transactions Graphics (TOG), 37 (4):1-14, 2018.
Yuval Alaluf, Daniel Garibi, Or Patashnik, Hadar Averbuch-Elor, and Daniel Cohen-Or. Cross- image attention for zero-shot appearance transfer. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-12, 2024.
Omri Avrahami, Dani Lischinski, and Ohad Fried, Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18208-18218, 2022.
Herbert Bay, Tinne Tuytelaars, and Luc Van Gool. Surf: Speeded up robust features. In Computer Vision-ECCV 2006: 9th European Conference on Computer Vision Graz, Austria, May 7-13, 2006. Proceedings, Part / 9, pp. 404-417. Springer, 2006.
Bin Cao, Jianhao Yuan, Yexin Liu, Jian Li, Shuyang Sun, Jing Liu, and Bo Zhao. Synartifact: Classifying and alleviating artifacts in synthetic images via vision linion-language model. arXiv preprint arXiv:2402. 18068, 2024.
Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao Anydoor: Zero- shot object-level image customization. arXiv preprint arXiv:2307.0948) 2023.
Xi Chen, Yutong Feng, Mengting Chen, Yiyang Wang, Shilong Zhang, Yu Liu, Yujun Shen, and Hengshuang Zhao. Zero-shot image editing with reference imitation. arXiv preprint arXiv22406.07547, 2024.
Seunghwan Choi, Sunghyun Park, Minsoo Lee, and Jacgul Choo. Viton-hd: High-resolution virtual try-on via misalignment- -wware and pattern recognition (CVPR), 2021.
aware normalization. Proc. of the IEEE conference on computer vision
Yisol Choi, Sangkyung Kwak, Kyungmin Lee, Hyungwon Choi, and Jinwoo Shin. Improving dif- fusion models for virtual tny arXiv preprint 2r3.v:2403.05139, 2024.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309. 16588, 2023.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing eoxt-to-mage generation using textual inversion, 2022. URL https:tpp/.... http://axxvvvrroo
2208 01 618.
Sooyeon Go, Kyungmook Choi, Minjung Shin, and Youngjung Uh. Eye-for-an-eye: Appearance transfer with semantic correspondence in diffusion models. arXiv preprint TrXi:22406.07003 2024.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Jing Gu, Yilin Wang, Nanxuan Zhao, Tsu-Jui Fu, Wei Xiong, Qing Liu, Zhifei Zhang, He Zhang, Jianming Zhang, HyunJoon Jung, et al. Photoswap: Personalized subject swapping in images. arXiv preprint arXiv:2305.1 00223055.88880 18286, 2023.
Jing Gu, Yilin Wang, Nanxuan Zhao, Wei Xiong, Qing Liu, Zhifei Zhang, He Zhang, Jianming Zhang, HyunJoon Jung, and Xin Eric Wang. Swapanything: Enabling arbitrary object swapping in personalized visual editing. arXiv preprint arXiv:2404.05771, 2024.

---

Published as a conference paper at ICLR 2025
Liu He and Daniel Aliaga. Globalmapper: Arbitrary- shaped urban layout generation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 454 464, 2023.
Liu He and Daniel Aliaga. Coho: Sontext-sensitive city-scale hierarchical urban layout generation.
arXiv preprint arXiv:2407.11222 2024.
Liu He, Yijuan Lu, John Corring, Dinei Florencio, and Cha Zhang. Diffusion- based document layout generation. In International Conference on Document Analysis and Recognition, pp. 361-378. Springer, 2023.
Liu He, Yizhi Song, Hejun Huang, Daniel Aliaga, and Xin Zhou. Kubrick: Multimodal agent collaborations for synthetic video generation. arXiv preprint arXiv:2408. 10453, 2024.
Eric Hedlin, Gopal Sharma, Shweta Mahajan, Hossam Isack, Abhishek Kar, Andrea Tagliasacchi, and Kwang Moo Yi. Unsupervised semantic correspondence using stable diffusion, 2023.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-promy image editing with cross attention control. arXiv preprint 1.Xi::2088.01266 2022.
Jonathan Ho and Tim Salimans. Classifier free diffusion guidance, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, $3:6840-6855 2020.
Hang Hua, Jing Shi, Kushal Kafle, Simon Jenni, Dacan Zhang, John Collomosse, Scott Cohen, and Jiebo Luo. Finematch: Aspect-based fine-grained image and text mismatch detection and correction, arXiv preprint arXiv:2404. 11..0044444111 14715, 2024a.
Hang Hua, Yunlong Tang, Chenliang Xu, and Jiebo Luo. V2xum-Ilm: Cross-modal video summa- rization with temporal prompt instruction tuning. arXiv preprint arXiv:2404. /2353, 2024b.
Chao Huang, Susan Liang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Davis: High-quality audio-visual separation with generative diffusion models. arXiv preprint arXiv: 2020000 2023.
Yuming Jiang, Kelvin CK Chan, Xintao Wang, Chen Change Loy. and Ziwei Liu. Reference-bart based image and video super-resolution via 22-matching. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(7):8874-888) 2022.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6007-6017, 2023.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 1931-1941, 2023.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. image pre-training with frozen image encoders and large language models. arXiv preprint 2rXIv:2301.1259/) 2023.
Blip-2: Bootstrapping language-
Nannan Li, Qing Liu, Krishna Kumar Singh, Yilin Wang, Jianming Zhang, Bryan A Plummer, and Zhe Lin. Unihuman: unified model for editing human images in the wild. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2039-2048, 2024.
Jayi Liu, Tinghan Yang, and Jennifer Neville. Cliqueparcel: An approach for batching Ilm prompts that jointly optimizes efficiency and faithfulness. arXiv preprint arXiv:2402. /4833, 2024a.
12

---

Published as a conference paper at ICLR 2025
Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang. Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, and Hao Su. One-2-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10072-10083, 2024b.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: sero-shot one image 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9298-9309, 2023a.
Zhiheng Liu, Ruili Feng, Kai Zhu, Yifei Zhang, Kecheng Zheng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Cones: Concept neurons in diffusion models for customized generation. arXiv preprint urXiv:2303.05123 2023b.
David G Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60:91-110, 2004.
Shilin Lu, Yanzhu Liu, and Adams Wai-Kin Kong. Tf-icon: Diffusion-based training-free cross- domain image composition, In Proceedings of the IEEE/CVF International Conference on Com- puter Vision, pp. 2294-2305, 2023.
Grace Luo, Lisa Dunlap, Dong Huk Park. Aleksander Holynski, and Trevor Darrell. Diffusion hyperfeatures:
Searching through time and space for semantic correspondence. In Advances in Neural Information Processing Systems, 2023a.
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthe- sizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04377 2023b.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint (rXiv:2108.01073 2021.
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo. Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wer Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Ar- mand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
Xichen Pan, Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, and Furu Wei. Kosmos- g: Generating images in context with multimodal large language models. arXiv preprint arXiv:2310.02992, 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of 2023.
the IEEE/CVF International Conference on Computer Vision, pp.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXhv:2307.01955, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pp. 8788-88766 PMLR, 2021.
Ignacio Rocco, Mircea Cimpoi, Relja Arandjelović, Akihiko Torii, Tomas Pajdla, and Josef Sivic. Neighbourhood consensus networks. Advances in neural information processing systems, 31, 2018.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. Orb: An efficient alternative to sift or surf. In 2 International conference on computer vision, pp. 2564-2571. leee, 2011.
13

---

Published as a conference paper at ICLR 2025
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuming text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22500- 22510, 2023a.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yacl Pritch, Neal Wadhwa, Michael Rubinstein, and Kfrr Aberman. Hyperdreambooth: Hypernetorrs for fast personaliza- tion of text-to-image models. arXiv preprint arXiv:2307.06949 2023b.
Vishnu Sarukkai, Linden Li, Arden Ma, Christopher Ré, and Kayvon Fatahalian. Collage diffusion. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp- 4208 4217, 2024.
Paul Hongsuck Seo, Jongmin Lee, Deunsol Jung, Bohyung Han, and Minsu Cho. Attentive semantic alignment with offset-aware correlation kernels. In Proceedings of the European Conference Computer Vision (ECCV), pp. 349-364, 2018.
Yichen Sheng, Yifan Liu, Jianming Zhang, Wei Yin, Cengiz Oztireli, He Zhang, Zhe Lin, Eli Shechtman, and Bedrich Benes. Controllable shadow generation using pixel height maps. In European Conference on Computer Vision, pp. 240-256. Springer, 2022.
Yichen Sheng. Jianming Zhang. Julien Philip, Yannick Hold Good-Geoffroy, Xin Sun, He Zhang, Lu Ling, and Bedrich Benes. Pixht-lab: Pixel height based light effect generation for image compositing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 16643-16653, 2023.
Yichen Sheng. Zixun Yu, Lu Ling, Zhiwen Cao, Xuaner Zhang, Xin Lu, Ke Xian, Haiting Lin, and Bedrich Benes. Dr. bokeh: Differentiable ocelusion-awan bokeh rendering- Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4515-4525, 2024.
Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image generation without test-time finetuning. arXiv preprint Sttl:2304.034/111 2023a.
Ruoxi Shi, Hansheng Chen. Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero123++: a single image to consistent multi-view diffusion base model. arXiv preprint arXiv:2310.1511 2023b.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Learning local feature descriptors using convex optimisation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(8): 1573-1585, 2014.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing, pp. 2266-2265. PMLR, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Yizhi Song, Zhifei Zhang, Zhe Lin, Scott Cohen, Brian Price, Jianming Zhang, Soo Ye Kim, and Daniel Aliaga. Objectstitch: Object compositing with diffusion model. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 183 18310-18319, 2023.
Yizhi Song, Zhifei Zhang, Zhe Lin, Scott Cohen, Brian Price, Jianming Zhang, Soo Ye Kim, He Zhang, Wei Xiong, and Daniel Aliaga. Imprint: Generative object compositing by learning Sreettty-preservve representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8048-8058, 2024.
Luming Tang, Menglin Jia, Qianqian Wang. Cheng Perng Phoo, and Bharath Hariharan. Emer- gent correspondence from image diffusion. In Thirth- Conference on Neural Information Processing Systems, 2023. URL https: httpps://ppenreeee http//ppereveeeeeeeeee net forr/m d=ypoiijjffmn
14

---

Published as a conference paper at ICLR 2025
Luming Tang, Nataniel Ruiz, Qinghao Chu, Yuanzhen Li, Aleksander Holynski, David E Jacobs, Bharath Hariharan, Yael Pritch, Neal Wadhwa, Kfir Aberman, et al. Realfill: Reference-driven generation for authentic image completion. ACM Transactions Graphics (TOG), 43(4):1-12, 2024a.
Yunlong Tang. Siting Xu, Teng Wang, Qin Lin, Qinglin Lu, and Feng Zheng. Multi segment assemblage network for ad video editing with aeportance-coherene reward. In Proceedings of the Asian Conference on Computer Vision (ACCV). pp. 3519-3535, December 2022.
Yunlong Tang, Daiki Shimada, Jing Bi, Minggian Feng. Hang Hua, and Chenliang Xu. Empower- ing Iims with pseudo-untrimmed videos for audio-visual temporal understanding, 2024b. URL https: / arxiv. org/ abs 2203 16276.
Yunlong Tang, Gen Zhan, Li Yang. Yiting Liao, and Chenliang Xu. Cardiff: Video salient ob- ject ranking chain of thought reasoning for saliency prediction with diffusion. arXiv preprint arXiv:2408.12000 12009, 2024c.
Benzhi Wang, Jingkai Zhou, Jingqi Bai, Yang Yang, Weihua Chen, Fan Wang, and Zhen Lei. Real- ishuman: two-stage approach for refining malformed human parts in generated images. arXiv preprint arXiv:2409.036444 2024a.
Yibin Wang, Weizhong Zhang, Jianwei Zheng, and Cheng Jin. Primecomposer. Faster pro- gressively combined diffusion for image composition with attention steering. arXiv preprint arXiv:2403.05053, 2024b.
Guangxuan Xiao, Tianwei Yin, William T Freeman, Frédo Durand, and Song Han. Fastcom- poser: Tuning-free multi-subject image generation with localized attention. arXiv preprint arXiv:2305. 10431, 2023.
Zhexiao Xiong, Wei Xiong, Jing Shi, He Zhang, Yizhi Song, and Nathan Jacobs. Groundinghooth: Grounding text-to-image customization. arXiv preprint arXiv:2409.08522 2024.
Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar-based image editing with diffusion models. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18381- 18391, 2023.
Yongsheng Yu, Ziyun Zeng, Hang Hua, Jianlong Fu, and Jiebo Luo. Promptfix: You prompt and we fix the photo. arXiv preprint S.tI::2005.667855 2024.
Jiaxing Zhang, Jiayi Liu, Dongsheng Luo, Jennifer Neville, and Hua Wei. Llmexplainer: Large language model based bayesian inference for graph explanation generation. arXiv preprint 153551 2024a.
Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. 2023a.
Lingzhi Zhang, Zhengjie Xu, Connelly Barnes, Yuqian Zhou, Qing Liu, He Zhang, Sohrab Amirgh- odsi, Zhe Lin, Eli Shechtman, and Jianbo Shi. Perceptual artifacts localization for image synthesis tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7579- 7590, 2023b.
Yuxuan Zhang, Yiren Song, Jiaming Liu, Rui Wang. Jinpeng Yu, Hao Tang. Huaxia Li, Xu Tang, Yao Hu, Han Pan, et al. Ssr-encoder: Encoding selective subject representation for subject- driven generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8069-8075 8078, 2024b.
Zhifei Zhang, Zhaowen Wang. Zhe Lin, and Hairong Qi. Image super-resolution by neural texture transfer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7982-7991, 2019.
Yuqian Zhou Connelly Barnes, Eli Shechtman, and Sohrab Amirghodsi. Transfill: Reference- guided image inpainting by merging multiple color and spatial transformations. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 2266-2276. 2021.
15


Reviews:
[{"review_id": "GoHpFqVCK7", "reviewer": "Reviewer_FAkR", "summary": "The paper proposes an artifacts refinement framework to refine the artifacts (e.g., logos, details) in the generated image by leveraging the corresponding details from a reference image. The method is evaluated qualitatively and quantitatively on a new benchmark, GenArtifactBench, consisting of artifacts generated by several well-known models, reference images, and dense human annotations.", "questions": "I think the paper adds values to better reference-based inpainting literature, but the significance of the scope and technical differences may require further explanation.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.The visual results are interesting and show the effectiveness of the method for fixing different types of details.\n\n2.The proposed benchmark could be a good complementary dataset to facilitate studies on the reference-based image inpainting.", "weaknesses": "1.I feel like the overall pipeline is still quite similar to previous reference-based image inpainting methods (Paint-by-Example, ObjectStitch). Using cross attention to localize matches is also already prevalent in the community. So the technical novelty is a bit limited. Could the authors please explain more why this is more effective?  What is the component that distinguishes the method from previous ones? Is it mainly because of the zoom-in of the reference image?\n\n2.The method focuses on the details of the reference-based inpainting results, I wonder if there are any advantages of the method given an object level inpainting? If so, why? If not, then I feel the scope of the method may not be not general enough.\n\n3.For evaluation, I wonder if it is possible to also evaluate the CLIP score on the masked region only, which potentially may be a better metric for the task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dojuMBgIyx", "reviewer": "Reviewer_bn5t", "summary": "This paper addresses an intriguing problem where a guidance image is used to restore artifacts in local regions of generated images. To achieve this, a two-stage approach is proposed: (i) an alignment stage that extracts the target region from the reference by computing correlation with the input ROI content; and (ii) a refinement stage that utilizes features from the reference patch to address the artifacts in the input ROI. Experiments demonstrate the superiority of the proposed method over several baselines, and ablation studies validate the importance of the design choices.", "questions": "- The training scheme should be clarified more clearly. In its current form, it is unclear which parts of the model will be trained. For example, if the model of Anydoor is adopted, will the full trainable parameters of Anydoor and the newly added modules by the proposed method be trained jointly? If so, how do you avoid the issue of overfitting or damaging the capability of the pre-trained Anydoor when the U-Net is trained on the collected dataset?\n\n- For the proposed task, one key consideration should be the generalization ability to diverse artifacts induced by different task-specific models. Therefore, it is crucial to design a comprehensive image degradation simulation pipeline. Can you elaborate on the data perturbation used in preparing the training dataset?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed task is well-motivated and interesting, potentially benefiting practical scenarios in reference-guided image generation applications.\n\n- The idea of the proposed two-stage pipeline is simple yet effective, contributing valuable insights to the community.\n\n- The experiments are comprehensive, effectively justifying the major claims and the significance of the proposed innovation.", "weaknesses": "- As claimed in the paper, the proposed method is model-agnostic and applicable to diverse tasks. Therefore, it is crucial to study the robustness of the input data. Although stated as a limitation, a comprehensive study is still needed to show the milestones the proposed method has achieved and what remains for future work.\n\n\t- For the alignment stage, will the performance degrade if the input artifact images contain different contents (though still sharing the common object) compared to the reference image?\n\t- For the refinement stage, will the performance degrade if the ROI of the input artifact images is not very similar to that in the reference image (possibly due to structural distortion from image-guided generation algorithms)?\n\n- As the major contribution, the importance of the two-stage method is validated in a less convincing manner. In Table 3, the metrics may not effectively capture the local artifacts. It would be stronger if a user study were provided, i.e., a comparison between the full method and the model trained only with the alignment mode using the full reference image directly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ftw0lLQTkZ", "reviewer": "Reviewer_Hbs1", "summary": "This paper proposed a reference-guided artifacts refinement method, which follows the Refine-by-Align. Specifically, the proposed method can be divided into two stages: the alignment stage and the refinement stage. The alignment stage leverages the cross-attention maps to locate the artifact region in DINOv2 features (target view). Then the target view would be filtered by the mask and then refines the final results through the reference-guided inpainting. The authors present a new GenArtifactBench dataset to verify the effectiveness of the proposed method. The proposed method outperforms other competitors in artifact refinement. Moreover, this paper proposed a manually annotated dataset called MVObj for training.", "questions": "1. Will the benchmark and MVObj be open-released?\n2. The authors should provide more novel view synthesis examples and clarify it. Otherwise, the claim to address novel view synthesis would conflict with the limitation of \"large disparity\".\n3. It is interesting to explore whether larger target image with more DINOv2 tokens would benefit the final performance.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The newly proposed task, reference-guided artifacts refinement is interesting, which could facilitate many tasks, such as reference-guided editing and novel view synthesis.\n2. The solution of Refine-by-Align is reasonable.\n3. The manually collected MVObj is interesting, including multi-view scenes with non-rigid objects, such as animals shown in Fig.13. Unfortunately, no more details are discussed to clarify how to collect this dataset in this paper.", "weaknesses": "1. The overall typesetting of this paper is confusing. For example, Fig.3, Fig.4, and Fig.5 are displayed in completely unrelated paragraphs. Moreover, the order of Fig3,4,5 is also confusing, while Fig.5 should be mentioned before Fig.4.\n2. More details about MVObj should be discussed in the paper.\n3. As discussed in the limitation, the proposed method struggles to repair artifacts with large viewpoint changes, which largely restricts its application. Most cases shown in the paper could be addressed as a simple homography warping. More examples of novel view synthesis should be considered as the paper claims.\n4. Actually, the experiments in this paper are not solid and fair enough. Artifact removal is a new task proposed in this paper, while all other competitors are not specially trained for this. As shown in Fig.6, all other methods are completely failed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "THgsELVK1C", "reviewer": "Reviewer_azdH", "summary": "This paper introduces Refine-by-Align, a novel model for reference-guided artifact refinement in personalized image generation. It employs a two-stage diffusion-based framework to enhance image fidelity and identity without requiring test-time tuning, significantly improving fine details across various applications.", "questions": "The reviewer raised concerns about the task workflow design. Since it is necessary to annotate the artifact mask regions in the image with artifacts during inference, why not simultaneously annotate the corresponding mask regions in the reference image, instead of relying on feature matching?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.This paper is clearly written and easy to understand.\n\n2.The authors conducted extensive experiments to demonstrate the effectiveness of the method.\n\n3.It is meaningful for the practical applications of customized generation.", "weaknesses": "The visual results presented in the paper mostly showcase examples where the target region in the reference image and the corresponding region in the given image with artifacts have little difference in angle and position. The reviewer would like to see the effectiveness across a broader range of tests, which is important for practical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)