MetaDesigner: Advancing Artistic Typography through AI-Driven, User-Centric, and Multilingual WordArt Synthesis

Authors: Jun-Yan He, Zhi-Qi Cheng, Chenyang Li, Jingdong Sun, Qi He, Wangmeng Xiang, Hanyuan Chen, Jin-Peng Lan, Xianhui Lin, kang zhu, Bin Luo, Yifeng Geng, Xuansong Xie, Alexander G Hauptmann
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: MetaDesigner

Abstract:
MetaDesigner introduces a transformative framework for artistic typography synthesis, powered by Large Language Models (LLMs) and grounded in a user-centric design paradigm. Its foundation is a multi-agent system comprising the Pipeline, Glyph, and Texture agents, which collectively orchestrate the creation of customizable WordArt, ranging from semantic enhancements to intricate textural elements. A central feedback mechanism leverages insights from both multimodal models and user evaluations, enabling iterative refinement of design parameters. Through this iterative process, MetaDesigner dynamically adjusts hyperparameters to align with user-defined stylistic and thematic preferences, consistently delivering WordArt that excels in visual quality and contextual resonance. Empirical evaluations underscore the system's versatility and effectiveness across diverse WordArt applications, yielding outputs that are both aesthetically compelling and context-sensitive.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730481972168

thread_type:
official_review

<thread_object>
creator: reviewer_9f4p

<content>
summary:
The paper studies the problem of generating stylistic text (with stylized glyph and texture) from input text prompts.

The core contribution is a multi-agent system that factors text generation into multiple sub-tasks, and addresses them with specialized LLM, LMM and stable diffusion agents, including: 1)  a pipeline designer for generating design programs to instruct other agents; 2) a glyph designer for glyph generation; 3) a texture designer for texture generation;  4) a Q&A evaluator that gathers user and LMM feedback for hyperparameter tuning of other agents to iteratively improve output quality.

strengths:
1. Artistic text generation is an important problem, whose solutions can be useful in many practical scenarios.

2. The amount of the contribution is non-trivial since a lot of effort has gone into adapting and connecting different large models together to form a complete, large working system. 

3. The whole system is well motivated and designed.

weaknesses:
1. Some technical details are missing. For example, in “Normal & Traditional Glyph”, the rendering details of normal and traditional glyphs based on $s^{gly}$ are unclear. Also, it is not clear how vector graphics optimization in “Semantic Glyph Transformation” is done. Where do the “target objects” come from and what is the loss function exactly? In Section 3.4, what is the meaning of $L_m$ and $L_u$ ?

2. Some experimental settings are questionable. First, comparison to SDXL is unfair. Traditional text-to-image models like SDXL often require detailed prompts and may not work well with brief instructions (e.g., “Create a stylish word World Peace representing its meaning”) that the proposed system is supposed to handle. The proposed system utilizes GPT for input prompt extension and, thus, it should be compared against a GPT-augmented SDXL where GPT is used to extend the input prompt into a more detailed description relevant for text generation.  Second, in the user study, “Aesthetics & Creativity” should be divided into two independent dimensions, instead of being merged as a single one. Third, the claim “Table 2 presents quantitative results using SSIM and LPIPS metrics, further reaffirming the excellence of our approach in terms of readability and aesthetics.” is inadequate. SSIM and LPIPS mainly measure image similarity and, therefore, are not suitable for evaluating readability. To compute quantitative scores for readability and legibility, a more reasonable way may be to apply a model to detect and recognize text on the generated images and then compare the outputs with the ground truth text. Last, the “quantitative” evaluation of the Q&A evaluation agent is lacking. The results of the proposed method, without this agent, should be added to Table 1 and Table 2. 

3. Some results are not satisfactory. As shown in Figure 5, the outputs by the proposed method, although being visually appealing, suffer from low text legibility and readability: some letters and words are either hard to recognize due to some of their strokes being excessively deformed or stylized (e.g., as leaves), or hard to read because of low color contrast between them and the background.

questions:
1. Should Equation (13) use max instead of argmax?

2. For the hyperparameter optimization in Section 3.4, it seems that the parameters are updated with GPT (as illustrated in Figure 4). What are the detailed prompts used to instruct GPT to output the updated prompts/parameters for each case in Figure 4?

3. For the user study, what does “Text accuracy” mean? Was the input text exposed to the participants during the study? What is the meaning of the percentages in Table 1?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730517789336

thread_type:
official_review

<thread_object>
creator: reviewer_zqvy

<content>
summary:
This paper introduces a multi-agent system capable of generating WordArt based on user instructions, with flexibility in style and texture. The system primarily consists of a Pipeline Designer, a Glyph Designer, and a Texture Designer, each responsible for overall process design, generating font images, and applying background and text textures, respectively. A feedback mechanism is integrated into the system, where prompt refinement is achieved through interactions between GPT and LLava or users, improving prompt to produce higher-quality images that better align with user expectations.

strengths:
1. The WordArt examples generated by the model in this paper demonstrate higher visual quality, more accurate text representation, and greater consistency in overall style compared to previous methods.

2. The system proposed in this paper can generate WordArt in diverse styles, closely resembling designs created by human designers, owing to the use of a style LoRA library. Additionally, it demonstrates the effect of merging LoRAs with varying weights.

3. The paper utilizes advanced prompt-LM techniques, such as Chain-of-Thought, Tree-of-Thought, and Feedback Refinement, to enhance the accuracy of sub-prompts for each agent, resulting in generated content that better aligns with user preferences.

weaknesses:
1. **Lack of novelty**: The improvements in “Accuracy and Creativity” claimed in the paper primarily depend on the construction of the style LoRAs library in “Texture Designer”, which aligns with previous experience. The impact and novelty of other modules are also insufficiently highlighted. The paper adopts a style closer to a technical report.

2. **Insufficient Experimental Design**: For example, (a) the user study includes only 11 participants, and it may benefit from a fair comparison using publicly available VLMs (e.g., GPT4-V); (b) the ablation study only analyzes the ToT module within the “Texture Designer,” without examining the analyses of other modules; (c) the comparison of generated results with other models is limited to 6 samples in Fig. 5, and more samples would be expected.

questions:
I'd like to pose questions that can address the weaknesses discussed.
1.  **More Detailed Examples of the Full Pipeline**: While Fig. 10 presents the final results, it would be beneficial to show the output at each step of the system, such as the glyph images generated by the “Glyph Designer Agent.” Subfigures (like the picture with dark green tones in the lower left corner) may be good choices for illustrating these intermediate steps.

2.  **Details on the Construction of the Style LoRAs Library in the “Texture Designer”**: Additional information is expected regarding the number of trainable parameters, the volume of collected data, and a comparison of data size with previous WordArt synthesis works, among other relevant details.

3. **Information about the inference efficiency**: A comparison of image generation time and computational resources required, relative to other models

4. **Impact of Replacing with GPT4-V**: a) In both the “Texture Designer” and “Parameter Optimization” stages, image input is utilized. Would replacing the previous GPT-4 with GPT4-V be a more effective choice? Quantitative and sample-based analyses? b) What differences would arise if using GPT4-V instead of LLava in the “Feedback” stage?

5. **Potential Common Problems in Generated Images**: a) Although the images in Fig. 5, Fig. 8, and Fig. 9 may possess artistic qualities, the text color often resembles the background, making it less distinguishable compared to results from other models.  b) In the lower-left of Fig. 8, style information, such as “Renaissance,” is generated as part of the text—what causes this, and is this phenomenon common?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730555696821

thread_type:
official_review

<thread_object>
creator: reviewer_t5qa

<content>
summary:
This paper introduces a novel framework for artistic typography synthesis that leverages Large Language Models (LLMs) to drive an interactive and user-centered design process. The system is built on a multi-agent architecture, including Pipeline, Glyph, and Texture agents, which enable the generation of highly customizable WordArt. 
The authors introduce a comprehensive feedback mechanism that integrates insights from multimodal models and user evaluations to iteratively refine hyperparameters, ensuring that outputs align with specific stylistic and thematic preferences.

strengths:
**Innovative Framework** The use of a multi-agent system (Pipeline Designer,
Glyph Designer, Texture Designer, and Q&A Evaluation Agent) and the integration of large model to drive the design process is a novel approach that addresses the challenges of creating high-quality WordArt.

**User-Centric Design** The comprehensive feedback mechanism, which includes insights from multimodal models and user evaluations, ensures that the generated WordArt aligns closely with user preferences and expectations.

**Multilingual Capabilitie** The system demonstrates robust performance across English, Chinese, Japanese, and Korean, making it versatile and suitable for a wide range of applications.

**Empirical Validation** The paper provides detailed comparisons with state-of-the-art methods, showing that MetaDesigner outperforms existing techniques in terms of WordArt synthesis success, quality, diversity, and creativity.

weaknesses:
**Complexity and Usability** The multi-agent system and the iterative optimization process may be complex for non-technical users, potentially limiting the system's accessibility.

**Computational Resources** The paper does not provide a detailed analysis of the computational resources required to run MetaDesigner, which is important for practical deployment.

questions:
**Glyph and Texture** Will separating Glyph and Texture limit the diversity of generated results? Could you provide additional ablation studies to demonstrate the necessity of decoupling Glyph and Texture?

**User Interface** Can you provide more details about the user interface and how users interact with the system? How intuitive is the user experience?

**Failure Case** It would be best to explain the existing artifacts in the current system and suggest potential areas for improvement in future work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732583530131

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Update Manuscript

comment:
We have updated Table 1 and Table 2 to reflect the impact of evaluation feedback. Additionally, we have included a new Table 3 to present VLM evaluation results. Figure 17 provides further visualizations, while Figure 18 showcases the performance of mixing/separating glyphs and textures. We will continue to refine the manuscript before the deadline, adding more details and examples to improve reader comprehension.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732790239200

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Dear reviewers

comment:
Dear reviewers，
    
We sincerely appreciate the diligent efforts of all reviewers. We have carefully considered all feedback and made appropriate revisions to the manuscript during the previous discussion period. As the discussion period has been extended for an additional six days, we are eager to continue the dialogue to foster a deeper understanding of our work and further enhance its quality.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a multi-agent framework for artistic typography synthesis, consisting of a Pipeline Designer, a Glyph Designer, and a Texture Designer.

The major strengths of the paper: 1) The proposed framework is well motivated and designed, with multi agents and LLM and advanced prompting.  2) It can generate artistic typograph with higher visual quality, text representation, and style, as well as better alignment to user preference, and multi lingual user cases. 3) The whole amount work/contribution is non-trival, and some promising results are shown.

The major weaknesses of the paper: 1) As reviewers pointed out, this is a system work, so the technical contribution itself may not be very high. And the system is quite complicated. 2) Some technical and experiment details are missing or not clear.