Preserving the Unique Heritage of Chinese Ancient Architecture in Diffusion Models with Text and Image Integration

Authors: Biao Li, Yunxi Yan, Wenxiu Ma, Jinyuan Feng, Hemin Li, Gang Kou
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Diffusion, RAG, Representation Learning, Chinese Ancient Architecture

Abstract:
Leveraging the impressive generative capabilities of diffusion models, we can create diverse images from imaginative prompts with careful design. To be noticed, the key components, such as CLIP, are essential for aligning prompts with image representations. However, these models often underperform in specialized areas, like the Chinese ancient architecture. One of the important reasons is that historical buildings include not only architectural information, but also historical and cultural content. The preservation and integration of these unique characteristics has become a significant challenge in model expansion. In this paper, we propose an Image-Annotation-Augmented Diffusion pipeline combining human feedback to explore the specific-area paradigm for image generation in the context of small amounts of data and professional concepts. We first leverage Segment Anything 2 (SAM2) to obtain a refined content image to enable an in-depth analysis of the relationship between unique characteristics and multimodal image generation models, and reselected representative images and regrouped them according to their distinctive objective and the existing dataset. Then, we introduce the effective RAG and GraphRAG module to identify the complex structure of relationships among different entities in the training and inference stages respectively. Based on the initial text by BLIP3, the RAG instructs GPT4 to facilitate more accurate, content-aware annotations during training, and augment a high-quality object prompt using the GraphRAG during inference. Benefit from these outstanding models and architectures, we train fine-tuning models to showcase the enhanced performance of our proposed pipeline compared to other existing models. Experiments demonstrate that our pipeline effectively preserves and integrates the unique characteristics of ancient Chinese architecture.

Decision: reject
Reviewer scores: 3, 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1729687404627

thread_type:
official_review

<thread_object>
creator: reviewer_uzyh

<content>
summary:
In this paper, the authors present an Image-Annotation-Augmented Diffusion pipeline integrated with human feedback. This pipeline aims to explore the specific-area paradigm for image generation considering small data amounts and professional concepts. Initially, the Segment Anything 2 (SAM2) model is utilized to acquire refined content images, enabling an in-depth examination of the relationship between unique characteristics and multimodal image generation models. Subsequently, the Retrieval-Augmented Generation (RAG) and BLIP3 are employed to obtain high-quality annotations. Through these steps, the fine-tuned model demonstrates the enhanced performance of the proposed pipeline compared to existing models. Experiments confirm the pipeline's effectiveness in preserving and integrating the unique features of ancient Chinese architecture.

strengths:
1. The writing is clear and understandable, facilitating comprehension of the research content and methods.

2. The experimental results appear promising, indicating the potential of the proposed pipeline in handling the specific task of generating images related to ancient Chinese architecture.

weaknesses:
The research domain is relatively narrow, focusing specifically on ancient Chinese architecture. This limited scope might pose a challenge for broader acceptance at a conference like ICLR, despite the seemingly good performance within the specific domain.

questions:
1. Why not attempt to apply the proposed pipeline to other domains to further verify its effectiveness?
2. Are there any potential limitations or challenges that might arise when applying this pipeline to other types of cultural or architectural heritages?
3. How might the performance of the pipeline be affected by different levels of data availability and quality in other domains?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730440372098

thread_type:
official_review

<thread_object>
creator: reviewer_tkwj

<content>
summary:
This paper introduces an Image-Annotation-Augmented Diffusion pipeline that preserves the unique heritage of Chinese ancient architecture in diffusion models. The process begins by using SAM2 to accurately crop architectural objects from images, followed by generating text descriptions using the image captioning model BLIP3. Next, RAG, GraphRAG, and LLM models process professional texts sourced from the Cultural Relics Bureau to identify relationships between architectural elements. Finally, human feedback is applied to refine the text, producing the final text prompt for the diffusion model.

strengths:
1. This paper focuses on fine-tuning a diffusion model for synthesizing images of Chinese ancient architecture, which is beneficial for the digital preservation of historical relics and the advancement of related industries.

2. The pipeline is well-structured and practical. It considers both the subject and style of the architecture during model training. Additionally, combining RAG methods and LLM with human feedback improves the accuracy of the text descriptions.

weaknesses:
1. This paper resembles an engineering project that applies existing methods and pipelines for generating images of Chinese ancient architecture. Regarding methodological novelty, the combination of these existing methods is not particularly innovative. However, the proposed pipeline effectively addresses the unique features of Chinese ancient architecture and successfully achieves accurate image generation.

2. The experimental testing could be strengthened. For instance, attributes like dynasty, cultural style, and location are valuable for characterizing architectural features. During inference, the authors could vary one or more of these attributes to observe differences in the generated architecture. Such testing would help verify whether the learned diffusion model accurately understands key architectural information.

questions:
1. In Figure 5, the generated image of Ours (LoRA) seems to depict an incorrect view for the prompt“A top-down view of palace.”

2. In Figure 1, unique feathers --> unique features?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730527809980

thread_type:
official_review

<thread_object>
creator: reviewer_3fwp

<content>
summary:
This paper aims to use diffusion models to generate Chinese ancient architecture with specific features or styles. The author uses LoRA to fine-tune a pretrained diffusion model, and proposes "Image-Annotation-Augmented Diffusion" pipeline that combines image segmentation, annotations enhanced by human feedback, and domain-specific cultural data. The approach involves using SAM2 for precise image segmentation, BLIP3 for text annotations, and GraphRAG modules to retrieve and augment cultural data. The experimental results show effectiveness of fine-tuning methods on diffusion models.

strengths:
1)	The topic is interesting and meaningful. The author tries to fine-tune diffusion models to work on specialized domain, such as Chinese ancient architecture. 
2)	The author introduces several models, such as SAM2 and BLIP3, to help improve the quality of image-caption pairs, so as to fine-tune the diffusion model to generate accurate outputs. 
3)	The experimental results show effectiveness of the method that can reduce the FID of Chinese ancient architecture images generated by diffusion models.

weaknesses:
1)	The representation is poor. There are many spelling mistakes and grammatical mistakes in the paper, such as “mothods”, “catogory”, “... to proof the advantage” and “The generated results exhibits ...”. Therefore, I question the completeness of the paper. 
2)	The effectiveness of some experimental results is not obvious. For example, Fig. 5 shows the comparison of the method with other models. However, there is little difference between the outputs of fine-tuned SDXL and the paper, and the output image of paper’s method with LoRA in line 5 is not reasonable. Also, the style of reference pictures is too monotonous, which is not enough to prove the effectiveness of the method in this paper to achieve the goal of generating buildings with various specific styles. 
3)	The lack of novelty. I appreciate that the author uses SAM and other models for processing images and annotation to obtain better data quality. However, there is not enough contribution to innovation methods.

questions:
What if I want to generate Chinese ancient architecture with other specific styles accurately?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730649123444

thread_type:
official_review

<thread_object>
creator: reviewer_uzar

<content>
summary:
The paper introduces a pipeline for generating images of Chinese ancient architecture by preserving cultural and structural details in generated images. By combining many existing techniques, the proposed method can capture architectural elements in fine detail. The authors compare the proposed method with pre-trained text-to-image models, showing the effectiveness of the proposed method in image-text alignment and preservation of unique architectural traits.

strengths:
- The research has potentials in architectural heritage preservation.
- Use of RAG and GraphRAG for enhancing text prompts with cultural background improves alignment between image content and historical context.

weaknesses:
- The novelty is limited, which is not enough for ICLR.
- Contributions are unclear. The proposed method is just a combination of existing techniques, which is not sufficient for ICLR.
- The paper writing is not good and need to be revised.
- The comparison is unfair. The proposed method is finetuned while compared methods are based on pre-trained models.

questions:
- The paper should be revised to correct English writing mistakes and typos.
- What is the complexity of the proposed method when the authors combine many techniques?
</content>
</thread_object>

</thread>


Metareview:
metareview:
In this work, most reviewers vote for rejection, considering the limited applications for ancient Chinese architecture, the limited novelty of combining several existing modules, and bad writing. Thus, it is hard for AC to accept this paper.

### Summary
The paper introduces a diffusion pipeline for preserving ancient Chinese architecture's cultural and structural uniqueness.

### Pros
1. **Domain-Specific Contribution**: Focuses on the underexplored area of ancient Chinese architecture.
2. **Methodical Framework**: Combines existing tools effectively for enhanced data quality and alignment.
3. **Human Feedback Integration**: Improves annotation accuracy and cultural relevance.

### Cons
1. **Limited Novelty**: Combines existing techniques with minimal innovation.
2. **Narrow Focus**: Restricts applicability to Chinese architecture only.
3. **Poor Presentation**: Contains numerous grammatical errors and unclear contributions.
4. **Experimental Gaps**: Results lack diversity and fail to validate broader applicability.
5. **Unfair Comparisons**: Evaluate fine-tuned models against baselines without similar adaptation.

**Suggestions**: Broaden the scope to other architectural styles, improve writing quality, clarify contributions, and strengthen experiments with diverse and unbiased comparisons.