Refine-by-Align: Reference-Guided Artifacts Refinement through Semantic Alignment

Authors: Yizhi Song, Liu He, Zhifei Zhang, Soo Ye Kim, He Zhang, Wei Xiong, Zhe Lin, Brian L. Price, Scott Cohen, Jianming Zhang, Daniel Aliaga
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: diffusion model; inpainting; generative artifacts; image editing; image synthesis; artifacts refinement

Abstract:
Personalized image generation has emerged from the recent advancements in generative models. However, these generated personalized images often suffer from localized artifacts such as incorrect logos, reducing fidelity and fine-grained identity details of the generated results. Furthermore, there is little prior work tackling this problem. To help improve these identity details in the personalized image generation, we introduce a new task: reference-guided artifacts refinement. We present Refine-by-Align, a first-of-its-kind model that employs a diffusion-based framework to address this challenge. Our model consists of two stages: Alignment Stage and Refinement Stage, which share weights of a unified neural network model. Given a generated image, a masked artifact region, and a reference image, the alignment stage identifies and extracts the corresponding regional features in the reference, which are then  used by the refinement stage to fix the artifacts. Our model-agnostic pipeline requires no test-time tuning or optimization. It automatically enhances image fidelity and reference identity in the generated image, generalizing well to existing models on various tasks including but not limited to customization, generative compositing, view synthesis, and virtual try-on. Extensive experiments and comparisons demonstrate that our pipeline greatly pushes the boundary of fine details in the image synthesis models.

Decision: accept (poster)
Reviewer scores: 6, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1729867313584

thread_type:
official_review

<thread_object>
creator: reviewer_azdh

<content>
summary:
This paper introduces Refine-by-Align, a novel model for reference-guided artifact refinement in personalized image generation. It employs a two-stage diffusion-based framework to enhance image fidelity and identity without requiring test-time tuning, significantly improving fine details across various applications.

strengths:
1.This paper is clearly written and easy to understand.

2.The authors conducted extensive experiments to demonstrate the effectiveness of the method.

3.It is meaningful for the practical applications of customized generation.

weaknesses:
The visual results presented in the paper mostly showcase examples where the target region in the reference image and the corresponding region in the given image with artifacts have little difference in angle and position. The reviewer would like to see the effectiveness across a broader range of tests, which is important for practical applications.

questions:
The reviewer raised concerns about the task workflow design. Since it is necessary to annotate the artifact mask regions in the image with artifacts during inference, why not simultaneously annotate the corresponding mask regions in the reference image, instead of relying on feature matching?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730391042559

thread_type:
official_review

<thread_object>
creator: reviewer_hbs1

<content>
summary:
This paper proposed a reference-guided artifacts refinement method, which follows the Refine-by-Align. Specifically, the proposed method can be divided into two stages: the alignment stage and the refinement stage. The alignment stage leverages the cross-attention maps to locate the artifact region in DINOv2 features (target view). Then the target view would be filtered by the mask and then refines the final results through the reference-guided inpainting. The authors present a new GenArtifactBench dataset to verify the effectiveness of the proposed method. The proposed method outperforms other competitors in artifact refinement. Moreover, this paper proposed a manually annotated dataset called MVObj for training.

strengths:
1. The newly proposed task, reference-guided artifacts refinement is interesting, which could facilitate many tasks, such as reference-guided editing and novel view synthesis.
2. The solution of Refine-by-Align is reasonable.
3. The manually collected MVObj is interesting, including multi-view scenes with non-rigid objects, such as animals shown in Fig.13. Unfortunately, no more details are discussed to clarify how to collect this dataset in this paper.

weaknesses:
1. The overall typesetting of this paper is confusing. For example, Fig.3, Fig.4, and Fig.5 are displayed in completely unrelated paragraphs. Moreover, the order of Fig3,4,5 is also confusing, while Fig.5 should be mentioned before Fig.4.
2. More details about MVObj should be discussed in the paper.
3. As discussed in the limitation, the proposed method struggles to repair artifacts with large viewpoint changes, which largely restricts its application. Most cases shown in the paper could be addressed as a simple homography warping. More examples of novel view synthesis should be considered as the paper claims.
4. Actually, the experiments in this paper are not solid and fair enough. Artifact removal is a new task proposed in this paper, while all other competitors are not specially trained for this. As shown in Fig.6, all other methods are completely failed.

questions:
1. Will the benchmark and MVObj be open-released?
2. The authors should provide more novel view synthesis examples and clarify it. Otherwise, the claim to address novel view synthesis would conflict with the limitation of "large disparity".
3. It is interesting to explore whether larger target image with more DINOv2 tokens would benefit the final performance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730613158845

thread_type:
official_review

<thread_object>
creator: reviewer_bn5t

<content>
summary:
This paper addresses an intriguing problem where a guidance image is used to restore artifacts in local regions of generated images. To achieve this, a two-stage approach is proposed: (i) an alignment stage that extracts the target region from the reference by computing correlation with the input ROI content; and (ii) a refinement stage that utilizes features from the reference patch to address the artifacts in the input ROI. Experiments demonstrate the superiority of the proposed method over several baselines, and ablation studies validate the importance of the design choices.

strengths:
- The proposed task is well-motivated and interesting, potentially benefiting practical scenarios in reference-guided image generation applications.

- The idea of the proposed two-stage pipeline is simple yet effective, contributing valuable insights to the community.

- The experiments are comprehensive, effectively justifying the major claims and the significance of the proposed innovation.

weaknesses:
- As claimed in the paper, the proposed method is model-agnostic and applicable to diverse tasks. Therefore, it is crucial to study the robustness of the input data. Although stated as a limitation, a comprehensive study is still needed to show the milestones the proposed method has achieved and what remains for future work.

	- For the alignment stage, will the performance degrade if the input artifact images contain different contents (though still sharing the common object) compared to the reference image?
	- For the refinement stage, will the performance degrade if the ROI of the input artifact images is not very similar to that in the reference image (possibly due to structural distortion from image-guided generation algorithms)?

- As the major contribution, the importance of the two-stage method is validated in a less convincing manner. In Table 3, the metrics may not effectively capture the local artifacts. It would be stronger if a user study were provided, i.e., a comparison between the full method and the model trained only with the alignment mode using the full reference image directly.

questions:
- The training scheme should be clarified more clearly. In its current form, it is unclear which parts of the model will be trained. For example, if the model of Anydoor is adopted, will the full trainable parameters of Anydoor and the newly added modules by the proposed method be trained jointly? If so, how do you avoid the issue of overfitting or damaging the capability of the pre-trained Anydoor when the U-Net is trained on the collected dataset?

- For the proposed task, one key consideration should be the generalization ability to diverse artifacts induced by different task-specific models. Therefore, it is crucial to design a comprehensive image degradation simulation pipeline. Can you elaborate on the data perturbation used in preparing the training dataset?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730928815691

thread_type:
official_review

<thread_object>
creator: reviewer_fakr

<content>
summary:
The paper proposes an artifacts refinement framework to refine the artifacts (e.g., logos, details) in the generated image by leveraging the corresponding details from a reference image. The method is evaluated qualitatively and quantitatively on a new benchmark, GenArtifactBench, consisting of artifacts generated by several well-known models, reference images, and dense human annotations.

strengths:
1.The visual results are interesting and show the effectiveness of the method for fixing different types of details.

2.The proposed benchmark could be a good complementary dataset to facilitate studies on the reference-based image inpainting.

weaknesses:
1.I feel like the overall pipeline is still quite similar to previous reference-based image inpainting methods (Paint-by-Example, ObjectStitch). Using cross attention to localize matches is also already prevalent in the community. So the technical novelty is a bit limited. Could the authors please explain more why this is more effective?  What is the component that distinguishes the method from previous ones? Is it mainly because of the zoom-in of the reference image?

2.The method focuses on the details of the reference-based inpainting results, I wonder if there are any advantages of the method given an object level inpainting? If so, why? If not, then I feel the scope of the method may not be not general enough.

3.For evaluation, I wonder if it is possible to also evaluate the CLIP score on the masked region only, which potentially may be a better metric for the task.

questions:
I think the paper adds values to better reference-based inpainting literature, but the significance of the scope and technical differences may require further explanation.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper receives mixed but mostly positive reviews by the reviewers after the rebuttal. Reviewers appreciate the interesting task, the simple yet effective design, and the extensive experiments. Reviewer Hbs1 raised several concerns regarding the details of MVObj, missing novel view examples, and unfair comparisons. However, most of them are addressed in the authors' rebuttal. AC did not find any major remaining concerns to reject the paper and agrees with other reviewers.