Knowledge-Aware Artifact Image Synthesis with LLM-Enhanced Prompting and Multi-Source Supervision

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 13:57:33

Abstract: Ancient artifacts are an important medium for cultural preservation and restoration. However, many physical copies of artifacts are either damaged or lost, leaving a blank space in archaeological and historical studies that calls for artifact image generation techniques. Despite the significant advancements in open-domain text-to-image synthesis, existing approaches fail to capture the important domain knowledge presented in the textual description, resulting in errors in recreated images such as incorrect shapes and patterns. In this paper, we propose a novel knowledge-aware artifact image synthesis approach that brings lost historical objects accurately into their visual forms. We use a pretrained diffusion model as backbone and introduce three key techniques to enhance the text-to-image generation framework: 1) we construct prompt with explicit archeological knowledge elicited from large language models (LLMs); 2) we incorporate additional textual guidance to correlated historical expertise in a contrastive manner; 3) we introduce further visual-semantic constraints on edge and perceptual features that enable our model to learn more intricate visual details of the artifacts. Compared to existing approaches, our proposed model produces higher-quality artifact images that align better with the implicit details and historical knowledge contained within written literature.

Main paper: 
 introduction: 
 Ancient artifacts are crucial for cultural preservation, as they represent tangible evidence of the past, oering insights into history. In recent years, innovative artifact-related projects have emerged, including the restoration of degraded character images [33] , the generation of captions for ancient artwork [32] , and the deciphering of oracle bone inscriptions [5] . These works have opened up new avenues for researchers to study artifacts and gain insights into the past. Despite these advancements, there are still many areas to be explored in artifact-related tasks, one of which is to recreate visual images of artifacts from text descriptions, as many physical copies of artifacts are often damaged or lost, leaving only textual records behind. This task could prove immensely invaluable to historical studies and cultural preservation because it provides historians with new visual angles to study the past and enables people to connect with their cultural heritage.One line of techniques that has shown potential to aid in the recreation of visual images of ancient artifacts is text-to-image synthesis. This task has been a popular area of research, especially in recent years with the introduction of diusion models [14, 22, 27, 36, 39, 49] that have demonstrated signicant capabilities in generating photo-realistic images based on a given text prompt in open-domain problems [10, 23, 26, 27, 30] . However, in the specialized area of archaeological studies, where data is often limited and domain knowledge is required, vanilla diusion models struggle to produce promising results even with netuning, as shown in Figure 1 . The generated images often display errors in shape, patterns, and details that fail to match the implicit knowledge in the textual information and the underlying historical context of the target artifact.We have identied a key cause for this problem to be the lack of knowledge supervision during the generating process, which can be attributed to two main aspects. 1). Current text prompts may not be infused with domain-specic knowledge from the archaeological and historical elds, leading to noisiness and the lack of wellpresented knowledge information in the text prompt. 2). The text and visual modules in the vanilla diusion models [14, 27, 35, 37, 38] may be unable to capture domain-specic knowledge under the standard training pipeline, resulting in the absence of detailed textual and visual signals of ancient artifacts in the generation process.To address these challenges, we propose our knowledge-aware artifact image synthesis approach with a pretrained Chinese Stable Diusion model [27, 51] as our backbone. Our method can generate visualizations of lost artifacts that well align with the underlying domain knowledge presented in their textual records. Specically:1). To address the issue of noisiness and lack of well-presented knowledge information in the text prompt, we propose to use Large Language Models (LLMs) to enhance our text prompts in two ways: for one, we use LLMs to extract the core and meaningful information in the given text prompt and reorganize them in a more structured way to explicitly present the current knowledge information; for another, we use LLMs as an external knowledge base to retrieve relevant archaeological knowledge information and augment them in the restructured text prompt.2). To address the lack of both textual and visual knowledge supervision in the generation process, we introduce additional supervision in both modalities. Firstly, we introduce a contrastive training paradigm that enables the text encoder to make the textual representation of the artifact more in line with their archaeological knowledge. Secondly, we apply stricter visual constraints using edge loss [31] and perceptual loss [15] to make the nal visual output align with the visual domain knowledge of ancient artifacts.Both quantitative experiments and a user study involving human experts demonstrate that our knowledge-aware artifact image synthesis approach signicantly outperforms existing text-to-image models and greatly improves the generation quality of historical artifacts.Overall, our main contributions can be summarized as follows:• To our best knowledge, we are the rst to explore the textto-image synthesis task in archaeology as an attempt to visually recreate lost historical artifacts, thus aiding archaeologists in gaining deeper insights into our past cultural treasury. • We propose to use LLMs as both an information extractor and external knowledge base to elicit archaeological knowledge that explicitly aids better prompt construction in the specialized domain requiring high historical precision. • We introduce additional multimodal supervisions to enable our model to learn textual representations and visual features that better align with archaeological knowledge and historical context, thus improving the current netuning paradigm of diusion models.Figure 2 : Raw artifact descriptions fail to depict the artifact with sucient archaeological information, such as their artifact-"type" which determines important aspects of their visual appearance. As shown in this gure, the artifact on the left side is classied as a "Round Ding" rather than a "Square Ding", which strictly connes its shape to a round body having only three legs as opposed to four legs. 
background 2.1 problem statement: 
 As mentioned in Section 1, for many artifacts, text documents are the only available source of information. Hence our task is to recreate a visual image 0 8 given artifact text information ) 8 . The synthesized image 0 8 needs not only to align with the textual meanings conveyed in ) 8 but also to be in line with the implicit historical knowledge about the artifact. Only then will the generated image be historically correct and thus valuable to archaeological studies. Correspondingly, the training dataset ⇡ comes in the form of pairs as ⇡ = {() 8 , 8 )} = 8 , where ) 8 2 ) is the available text information and 8 2 is the corresponding artifacts image. The raw text information available for an artifact -as often cataloged in museums -contains roughly four parts: the name or title of the artifact; the time period of origin; a raw description of the artifact (often presented in a messy way); the physical size of the artifact. Formulated from accessible resources of such kind, the task of our work is then to generate accurate artifact images based on these textual descriptions of historical objects.To solve the task dened in the above section 2.1, we propose to build our model upon the text-conditioned Stable Diusion pipeline [27] . Before diving deeper into our approach, we present a detailed mathematical introduction of diusion models and Stable Diusion in Appendix F1 . Here, we only briey summarize the standard training objective of a Stable Diusion (SD) model as follows:! (⇡ (\ ) := E C,E (G 0 ),n ||n n \ (I C , C, F)|| 2 (1)where n is the applied random noise. I C 2 Z is the representation of a noised image in the latent space at time step C and F is the encoded text representation. n \ aims to denoise the latent space. G 0 is the real image at timestep 0 and E is the latent space encoder. 
our method: 
 Our proposed approach for knowledge-aware artifact image synthesis is built upon a pretrained Stable Diusion model, which retains its powerful generative capability of common domains and is further netuned to align with the specic characteristics of ancient artifacts. The generic Stable Diusion model, even with netuning, however, struggles to generate visually and historically accurate artifact images and shows multifaceted errors, as is demonstrated in Figure 1 .To address these issues, we propose specic modications at three steps in the Stable Diusion system: 1). Given source text information ) 8 of an artifact, we pass it into an LLM (in our case, GPT-3.5-TURBO) with carefully designed querying message and in-context examples to obtain a clean and augmented prompt input of our diusion model ) 0 8 . (Section 3.1). 2). During training, when ) 0 8 is passed into the text encoder, we apply an additional contrastive learning module on the text encoder to align the description of an artifact with its name, which is essentially an expert summary of its description. (Section 3.2).3). After the added noise is predicted in the training phase, we reconstruct the model-predicted image 0 8 and apply additional visual-semantic supervision with edge loss [31] and perceptual loss [15] to steer the generation of our model closer towards the ground-truth appearance of artifacts. (Section 3.3).The overall framework for our approach is illustrated in Figure 3 and explained in detail in the following subsections.We have noticed that the raw description of an artifact accessible in museum resources (as mentioned in Section 2.1) is far from ideal for prompting a text-to-image model. It is often incomplete and lled with noisy messages and fails to suciently depict a historical object. Other than the messiness problem, these o-the-shelf descriptions may well lack specic information about an artifact that is essential to its visual form, such as its fundamental classication (or: artifact-"type"). An example2 of this is given in Figure 2 . As per archaeological terminology, the fact that the artifact on the left side is classied as a "Round Ding" rather than a "Square Ding" connes its shape to a round body having only three legs as opposed to four legs. However, key implicit archaeological information of this kind is often missing in the raw description of the artifact, prohibiting a text-to-image model from suciently understanding the association between the visual appearance and the textual prompt.To alleviate the problems of noisiness and knowledge deciency in the original text information, we propose to utilize an LLM as both an information extractor to retrieve the most useful information, and as an external knowledge base to complete any missing important attributes of the artifact.Based on archaeological expertise, we have compiled a list of key attributes that are vital for eectively describing artifacts and dening their physical forms, see Table 1 . Examples of these attributes are given in Table 5 in Appendix A. While the "name", "time period" and "size" of an artifact are usually available in museum resources, the specic "material", "shape" and "paern" need to be extracted or derived from the raw description of the object. Further, as explained above, the classied "type" of an artifact determines certain fundamental aspects of its looks, which are specied by the generic denition of this artifact-type (i.e., "type denition"). It requires a general knowledge of archaeology to be able to categorize an ancient object into a certain artifact-type and to dene the basic appearance of this type. An LLM is well-suited for fullling these two tasks with its ability to obtain a certain extent of world knowledge from the massive pretraining corpus [20] and to learn to perform specialized downstream tasks using the in-context learning paradigm [3] . Specically, we use GPT-3.5-TURBO as our knowledge-base LLM, and the prompt for querying GPT-3.5 is designed with a similar format following self-instruct [45] . Our prompt template consists of three parts: 1). A task statement that describes to GPT-3.5 the task to be done; 2). Two in-context examples of high quality sampled from our labeled pool of 54 artifacts written by archaeology experts; 3). The target artifacts whose "material", "shape", "pattern", "type" and "type denition" are left blank and need to be answered by GPT-3.5. The former 3 attributes can be retrieved from the given "description" and the latter 2 artifact-type related features need to be fullled via the world knowledge of GPT-3.5 and its in-context learning from the given human-labeled examples. An example of our prompt for querying artifact information is illustrated in Figure 7 in Appendix B .By leveraging the power of LLM as both an information extractor and external knowledge provider, we are able to collect all the key attributes of a given artifact, which are then rearranged into the prompt to our diusion model with a [(⇢%] (implemented as a Chinese comma in our work) splitting each key feature, as shown by the example in Table 5 synthesizing a more realistic result that better corresponds to the ground-truth artifacts.Another issue we identify is that the text encoder might not encode the text into a representation that reects the underlying archaeological knowledge and thus needs further netuning. We observe that the "names" of ancient artifacts are often accurate and concise summarizations of the artifact's key attributes, while the "descriptions" provide an extended version of the artifact's features. Given that both the names and descriptions are provided by domain experts -as written in museum sources, they reect a high level of expertise in the eld. Thus, we believe that closely aligning the names and descriptions is essential to reect this domain knowledge.To achieve this goal, we propose the use of contrastive learning that aims to minimize the distance between positive pairs consisting of matching ([34B2A8?C8>=] 8 , [=0<4] 8 ) pairs extracted from ) 0 8 , and to maximize the distance between negative pairs with mismatching descriptions and names.However, we have also observed that artifacts with similar attributes (i.e., similar "description" contents) and historical origins often share similar names, making it unintuitive to netune the text encoder to dierentiate between these similar pairs. We believe that such pairs should be close to each other in the semantic space. Therefore, we readjust our sampling strategy for negative pairs. From the perspective of historical studies, "time period" is one of the most determining factors in the style and appearance of an artifact, where dierent artifacts from dierent eras can be vastly dierent. Therefore, aiming to separate hard negatives rather than slightly dierent ones, we sample our negative samples from artifact names in dierent eras.In our approach, we use InfoNCE [42] to penalize the misalignment in the representation encoded by the text encoder. The formula for text contrastive learning can be written as:EQUATIONwhereG 8 = E([34B2A8?C8>=] 8 ) • E([=0<4] 8) denotes the similarity between a pair positive sample in the text encoder's embedding space. Andis the set of # similarities between sampled pairs from the entire dataset, containing one positive sample G 8 and # 1 negative samples G 9 2where 8 < 9 and %4A8>3$ 5 () 8 ) < %4A8>3$ 5 () 9 ).Artifact images generated by the vanilla Stable Diusion model suffer from blurry edges and false color and patterns under the current setting (see Figure 1 ), implying that stricter visual constraints need to be enforced to address these issues. Therefore, we propose to use edge loss [31] and perceptual loss [15] that apply additional visual-semantic supervision on images generated by our Stable Diusion model.Edge Loss. Building upon the insights from [31] , we penalize the dierences in contours between two images by aiming to minimize the ! 2 distance between their edge maps, as shown in part c) of Figure 3 . Since the vanilla Stable Diusion model often produces images that suer from the problem of incorrect and blurry shape compared to the ground-truth artifact, it is necessary to penalize such errors as dened here in the edge loss:EQUATIONwhere ⇢⇡⌧⇢ (•) is an edge extracting function. In our approach, we use the Canny Edge Detector [4] as our edge extractor to extract edge maps (see examples in Figure 4 ), then compare the dierence between two contours in ! 2 distance. Perceptual Loss. Similar to [15] , we also penalize the problem of mismatching high-level details between the generated image and the real one. As we have also observed on the images generated by the vanilla Stable Diusion model, the high-level details (such as colors and patterns) are often misaligned with the original ones. Therefore, we incorporate perceptual loss into our training process to tackle such an issue, as perceptual loss works by mapping the images into a semantic space using a pretrained network, and then minimizing the dierence between the high-level features of the generated image and the original image. The formula for perceptual loss is dened as:EQUATIONwhere q denotes a pretrained image encoder to extract the highlevel features of an image. This is applied to impose stricter supervision on color, texture, and other high-level features. In our method, we use a CLIP-ViT-L/14 [25] image encoder to act as our pretrained image encoder for perceptual loss.Combining all the extra multi-source multi-modal supervisions above, the overall training objective of our system is:EQUATIONwhere _ 1 , _ 2 and _ 3 are hyperparameters controlling the weight of each supervision loss; G 8 is the similarity between a positive sample pair yielded from ) 0 8 andis a set of similarities of sampled negative pairs; 8 and 0 8 are the ground-truth and the restored image from our model's prediction. 
experiments 4.1 experimental setup: 
 Dataset. Due to the sparsity of paired text-image data in the ancient artifact domain, we build our own text-to-image dataset by collecting artifact information from National Palace Museum Open Data Platform [41] . After careful cleansing of available entries, we are left with 16,092 unique artifact samples with their descriptions and ground-truth images. We split the data by 80%/10%/10% for training, validation, and testing. 3Implementation Details. For our backbone model, we use a pretrained Chinese Stable Diusion Taiyi-Stable-Diusion-1B-Chinese-v0.1 [51] (dubbed Taiyi-SD) which was trained on 20M ltered Chinese image-text pairs. Taiyi-SD inherits the same VAE and U-Net from stable-diusion-v1-4 [28] and trains a Chinese text encoder from Taiyi-CLIP-RoBERTa-102M-ViT-L-Chinese [52] to align Chinese prompts with the images. Further training details are left in Appendix C. Evaluation Metrics. To comprehensively evaluate our method for text-to-image synthesis quantitatively, we employ three commonly used metrics that measure image generation quality: CLIP Visual Similarity, Structural Similarity Index (SSIM) [46] and Learned Perceptual Image Patch Similarity (LPIPS) [53] . Each of them highlights dierent aspects of the generated image. Together, they provide a thorough judgment of a synthesized artifact image in terms of its overall resemblance to the ground truth, the accuracy of its shape and pattern, and its perceptual anity to the target image. We leave an extensive explanation of these metrics in Appendix D.In Table 2 , we compare the quantitative results of our approach with the baselines on our test set. The rst column denotes the models we experimented with.For the baselines, we use Taiyi-SD via two versions:• Taiyi-SD-netuned-description: the netuned Taiyi-SD with the raw description (directly available from museum archives) as input prompt; • Taiyi-SD-netuned-attributes: the netuned Taiyi-SD using LLM-enhanced prompt (a sequence of artifact attributes) as designed in Section 3.1. For our approach, we apply the LLM-enhanced prompt by default and also explore three dierent versions of extra supervisions in addition to training the Taiyi-SD backbone: Overall, our proposed artifact image synthesis approach signicantly outperforms the netuned Taiyi-SD-baselines across all metrics. The improvement on SSIM indicates that images generated by our model better preserve the shapes and boundaries of the original artifacts. An increase in CLIP Visual Similarity also indicates that our approach produces images that are more closely aligned to the ground truths.Additional visual-semantic constraints in the form of edge loss and perceptual loss contribute greatly to boosting the SSIM and LPIPS scores. This can be attributed to the fact that edge loss and perceptual loss put a stricter condition on both structural details like edge and contour (captured by SSIM) and perceptual-level image features like color and texture (captured by LPIPS). These visual details are exactly much desired in our case of artifact image synthesis, as the shape, pattern, and texture of artifacts are of vital importance for determining their historical position and status.By further incorporating the text contrastive loss into the overall training objective, we observe a slight increase in CLIP Visual Similarity, yet a decrease in SSIM and LPIPS scores. We believe there are two reasons behind this phenomenon. For one, by aligning the text knowledge (descriptions with names) (see Section 3.2), the textual guidance for generating the image is better represented and closer to the general visual content of the artifact, thus leading to a higher CLIP Visual Similarity. For another, the relative weight of edge and perceptual loss is reduced with the additional text contrastive loss, which might compromise the strict supervision on structural coherence and perceptual similarity and limit the model's performance on SSIM and LPIPS.As is also evidently shown by just comparing the baselines netuned with dierent prompt formats in Table 2 , using LLM to enhance the prompt construction as a sequence of important artifact attributes eectively improves the performance of the netuned baseline model across all three metrics. More about the eects of our LLM-enhanced prompting method will be discussed in the following subsection 4.3.To investigate the contribution of components proposed in our approach and for further studies, we conduct extensive ablation studies on two key designs of our model.As is shown in Table 2, the netuned model benets from LLM-enhanced prompting, achieving better scores on all three quantitative metrics. To further illustrate the eectiveness of our proposed prompting method, we explore the zero-shot setting, where the baseline Taiyi-SD is directly prompted to generate artifact images without any training on our artifact dataset. We use either the raw description from the Figure 5 : Comparison between the netuned Taiyi-SD baseline model and OUR approach trained with additional edge loss and perceptual loss against the ground truth. Clearly, objects generated by OUR model display more accurate shapes, colors, and patterns when compared to the ground truth, whereas these delicate visual details are easily neglected by the baseline. museum archives or the sequence of artifact attributes enhanced by LLM as prompt. The results, as shown in Table 3 , again demonstrate the superiority of LLM-enhanced prompting, which excels across all metrics. This can be credited to the organized information format in the attribute sequence and the additional knowledge provided by LLM (see Section 3.1).Eectiveness of Edge Loss and Perceptual Loss. In Figure 5 , we compare the artifact images generated from our model that uses edge loss and perceptual loss against the netuned Taiyi-SD baseline that does not involve these visual semantic constraints. Evidently, the shapes, colors, and patterns of the artifacts are more accurate and close to the ground truth if the model is additionally supervised by edge and perceptual loss. On the other hand, the vanilla netuning paradigm may easily lead to output objects either lacking proper shapes and forms or manifesting incorrect motifs and patterns. For example, in the second column of Figure 5 , the "tall-footed" aspect of the target bowl is clearly neglected without edge and perceptual constraints. Also, the intricate cracking lines on the "begonia-blossom shaped vase" shown in the third column are better simulated with our model.In addition to quantitative evaluation, we conducted a user study involving archaeology experts to evaluate the generated images. This study is designed to assess various aspects of the generated artifacts, as outlined in our prompt design (see Section 3.1):• Material: How accurately does the generated artifact resemble the actual manufacturing material? • Shape: How closely does the generated artifact match the described shape? • Paern/Color: How faithful is the representation of patterns and colors on the generated artifact? • Size/Ratio: How accurately does the generated artifact maintain the ratio of height and width? • Dynasty (Time Period): How well does the generated artifact reect the characteristics of its era?Each aspect is rated on a scale of 0 to 5, with higher ratings indicating better quality. We randomly select 30 samples from the test set and provide the model-generated images to 20 graduate students of archaeology major from top institutions for assessment. The average ratings of images generated by our proposed method (OURS) are compared with those generated by the baseline Chinese SD model also netuned on our data. The results are presented in Table 4 .Clearly, according to human experts, the artifact images generated by our method are much better in quality across all ve important rating aspects, especially in terms of shape, pattern, and color. These results of our user study resonate with the ndings from the automatic evaluation metrics and further highlight the superior performance of our model in generating artifact images that accurately align with history.To oer a richer qualitative demonstration of our model's capabilities, we present a diverse collection of artifact images generated by our model, showcasing its remarkable delity across a broad spectrum of historical artifacts. Refer to Figure 6 in Appendix E 4 for a comprehensive visual display. 
related work: 
 Multimodal AI for Cultural Preservation. In the domain of fostering the preservation of art and culture, multimodal learning technologies have had an increasing presence in recent years. Revolving around ancient artworks (especially paintings with rich cultural backgrounds), datasets have been released that associate artworks with textual descriptions or question-answer pairs [1, 8, 40] , which in turn inspires algorithmic explorations for the visual question answering (VQA) task on these artworks [2, 54] . In terms of historical artifacts, most recent works focus on restoring ancient character images. These include degraded images of characters printed in old documents [33, 34] as well as inscriptions on oracle bones [5] . [32] also adapts image captioning techniques to generate descriptions of ancient artifacts sourced from museums.With a similar objective to aid cultural preservation, our workas pointed out in Section 1 -represents the rst novel approach to exploring text-to-image reconstruction of physically lost ancient artifacts that only exist in textual documentation. Our proposed system now enables archaeologists to easily obtain a highly accurate visual simulation of any artifact of interest mentioned in historical texts, thus opening up new possibilities for research into our cultural heritage and detailed artifact analyses.Text-to-Image Synthesis. Text-to-image synthesis tasks have long been a vital task at the intersection between computer vision and natural language processing, of which models are given a plain text description to generate the corresponding image. One major architecture in this area is GAN [9] , whose variations [16, 48, 50] have resulted in the state-of-art performance of text-to-image synthesis tasks. Recently, diusion models [14, 27, 35, 37, 38] also have 4 All Appendices are submitted as supplementary material.demonstrated their ability to achieve new state-of-the-art results [6] . Diusion models make use of two Markov chains: forward and reverse. The forward chain gradually adds noise to the data with Gaussian prior. The reverse chain aims to denoise the data gradually. The transition probability at each timestep is learned by a deep neural network, which in the case of text-to-image synthesis is usually a U-Net [29] model. Large Language Models. Language models are a family of probabilistic models that predict the probability of the next word, given a sequence of previous words within a context. The introduction of GPT-3 [3] , which contains 175B parameters, has led to the emergence of Large Language Models (LLMs), referring to language models with a large number of parameters. These LLMs have demonstrated never-seen-before abilities, expanding the frontiers of what is possible with language models. One emerging ability of LLMs is in-context learning [3] , where LLMs are able to perform downstream tasks after being prompted with just a few examples without further parameter updates. Thus, by providing carefully designed examples, we can make use of LLMs as an information extractor given a noisy and unstructured text. LLMs have also shown their ability to acquire world knowledge from the massive training corpus [21, 24, 44 ]. An ecient way to extract the implicit knowledge from LLMs is to ask questions with proper prompt engineering as LLMs are highly sensitive to the prompt input [20] . 
conclusion: 
 In this paper, we present a novel approach to tackle the challenge of artifact image synthesis. Our method features three key techniques: 1) Leveraging an LLM to infuse textual prompts with archaeological knowledge, 2) Aligning textual representations with domain expertise via contrasting learning, and 3) Employing stricter visualsemantic constraints (edge and perceptual) to generate images with higher delity to visual details of historical artifacts. Quantitative experiments and the human evaluation from our user study with archaeology experts conrm the superior performance of our approach compared to existing models, signicantly advancing the quality of generated artifact images.Beyond technological contributions, our work introduces a more profound societal impact. As the rst attempt to restore lost artifacts from the remaining textual descriptions, our work empowers archaeologists and historians with a tool to resurrect lost artifacts visually, oering new perspectives on cultural heritage and enriching our understanding of history. We also hope that this work will open new avenues for further exploration, fostering deeper insights into our past and cultural legacy with the help of technical advances. 
ethics statement: 
 In this project, we only used training data sourced from National Palace Museum [41] . This ensures that the data we worked with has already been scrutinized by authorities and is open to the public. However, we recognize possible inaccuracies in our model's generation despite our extensive eorts to improve its delity. Therefore, anyone using our model and system should be warned of possible mistakes in the generated artifact images and we strongly advise all users to verify important content.  
 
 Appendix: As per the submission guidelines, all technical Appendices mentioned in the main text are submitted separately as supplementary material.To maintain a consistent language usage throughout the paper, we translate all Chinese text (e.g., the textual descriptions of artifacts) into English via ChatGPT.To facilitate further research, we will make the dataset as well as our ChatGPTenhanced prompts publicly available. 


Decision: This paper has received Weak Accept, two Borderline Accept, and one Borderline Reject. This paper has studied a very interesting research direction of the recreation of visual images of ancient artifacts. The paper is clearly written and easy to follow.The technical solution utilizes LLM to explore domain knowledge to enhance the performance. The authors are suggested to revise the description of the edge loss, as raised by one reviewer. FID is also suggested to be added for completeness. Thus, the AC recommends the acceptance of the paper.

Reviews: review: paper_summary: paper_summary: The paper introduces a novel approach for re-visualizing ancient artifacts using text-to-image synthesis. By incorporating explicit archaeological knowledge, additional textual guidance related to historical expertise, and visual-semantic constraints, the proposed model generates higher-quality artifact images that align better with historical details. This research contributes to preserving cultural heritage and commemorating our past through reconstructed visual forms based on textual descriptions.
strength_weakness: strength_weakness: the paper found good point, namely the generation and restoration of artifacts. The two issues raised in the paper are indeed challenges faced by current large language models and vision-language multimodal models. The author enhances semantic understanding by utilizing enriched prompts, optimizes training strategies, and strengthens vision-language modality comprehension by increasing the loss function.
limitations: limitations: The description of the paper is quite clear. However, I have a concern.

In Table 2, the last two rows show that after adding text information, the SSIM and LPIPS metrics worsened. Can this be interpreted as a degradation in visual capability? Why does this phenomenon occur? The paper does not provide an analysis of this issue.
paper_summary: paper_summary: This paper introduces a knowledge-aware approach for synthesizing images of ancient artifacts, effectively capturing their visual forms by integrating archaeological knowledge, historical expertise, and visual-semantic constraints. Compared to existing methods, the proposed approach generates higher-quality images that better align with the implicit details and historical knowledge from textual descriptions, showcasing significant improvements over prior arts.
strength_weakness: strength_weakness: 1. Recreation of visual images of ancient artifacts from text descriptions is a novel and intriguing research direction, and this manuscript makes solid explorations in this area.
2. The authors propose a novel method for prompt construction that leverages domain knowledge, which is highly meaningful for the research community facing data scarcity.
3. The manuscript is written clearly and is easy to follow.
limitations: limitations: 1. The generated images of artifacts lack detail. For instance, in the first column of Fig. 5, where the model recreates the "Cloisonne Chi-Dragon Ear Furnace with Lotus Pattern," while the proposed method captures the artifact's overall shape, the "ear" part notably deviates from the ground truth. Given the historical and cultural significance of these details, further research is recommended to enhance the generated artifact images' level of detail.
2. There is a lack of a quantified ablation study. Despite providing visual ablations in Section 4.3, the manuscript lacks a quantitative evaluation of each component's effects. Incorporating a quantified ablation study would enhance the manuscript's quality.
3. The manuscript employs a base model based on Stable Diffusion v1.4, as mentioned in Section 4.1. If a more advanced base model were utilized, such as SD-XL [1], would it enhance the proposed method's performance? Further discussion on this point would be beneficial.

[1] Podell, Dustin, et al. "Sdxl: Improving latent diffusion models for high-resolution image synthesis." arXiv preprint arXiv:2307.01952 (2023).
paper_summary: paper_summary: The paper addresses the challenge of recreating visual images of ancient artifacts from textual descriptions, given the limited availability of physical copies. It highlights the significance of this task for historical studies and cultural preservation. Despite recent advancements in text-to-image synthesis using diffusion models, the paper identifies shortcomings in applying these models to archaeological studies due to the lack of domain-specific knowledge supervision during the generation process. To overcome these challenges, the paper proposes a knowledge-aware artifact image synthesis approach, utilizing Large Language Models (LLMs) to enhance text prompts and introducing additional supervision mechanisms for both textual and visual modalities. Experimental results and a user study demonstrate the effectiveness of the proposed approach in improving the generation quality of historical artifacts, showcasing its potential to aid archaeologists in gaining deeper insights into cultural heritage.
strength_weakness: strength_weakness: 1. The paper introduces a novel approach to address the specific challenge of recreating visual images of ancient artifacts from textual descriptions, contributing to the intersection of artificial intelligence and archaeology.

2. By integrating Large Language Models and introducing multimodal supervision mechanisms, the paper proposes a comprehensive solution tailored to the specialized domain of archaeological studies, addressing the limitations of existing text-to-image synthesis methods.

3. The paper provides rigorous experimental evaluation and user studies, demonstrating the superior performance of the proposed approach compared to existing models, thereby validating its effectiveness and potential utility for researchers and practitioners in the field.
limitations: limitations: 1. FID serves as a vital metric for assessing the quality and diversity of generated models. The authors should incorporate FID as a quantitative measure and compare it with baseline models.
2. The use of Large Language Models (LLMs) to assist in generating models in the visual domain has been explored extensively. The authors should discuss the distinctions from existing methods and highlight the unique aspects of employing LLMs in this paper.
3. The rationale behind utilizing Chinese Stable Diffusion as the baseline model needs clarification. Official Stable Diffusion models are expected to possess superior generation capabilities. The authors should discuss the advantages of using Taiyi-Stable-Diffusion.
paper_summary: paper_summary: This paper presents a novel approach to tackle the challenge of artifact image synthesis. It adopts a large language model to enrich the textual description with archaeological knowledge. A contrastive learning strategy is used to align the object names and their descriptions. Then the edge loss and perceptual loss are adopted to improve image fidelity. Quantitative experiments and the user study with
archaeology experts verify the strength of this method compared to the simple finetuning method.
strength_weakness: strength_weakness: This paper studies a novel problem of artifact image synthesis, which may benefit archaeological study.

The knowledge of LLM is imported to enhance the synthesis of images in a specific domain.

Some visual results and the user study exhibit the superiority of the proposed method.
limitations: limitations: The training process is not clear, and there could be technical mistakes:

-- Is the Canny detector differentiable?

-- How to obtain I_i^' during training? Did the author update the VAE parameters?

-- Was the denoising loss backpropagated to the text encoder?

-- How to use the validation set?

The evaluation:

-- How many images were sampled for each textual description? For this task, do the authors expect the diversity of synthesized images?

-- The qualitative comparison is insufficient.

-- Did the author try to synthesize some non-existent objects? Since it is one of the possible applications of this model.

-- Incorporating the text contrastive loss improves the CLIP similarity but decreases the SSIM value. It would be better to provide more visual results, and a user study could evaluate these two versions.

Other suggestions:

-- The textual descriptions are not provided in Figure 1.

-- Figure 4 seems unnecessary.