System: <task>Personal AI-Assistant</task>
System: 
**[User]:**

<Chris>
	Gender: male
	Location: Germany
	Age: 40
	Name: Christian
	Skills: Electronics, Computer-Science (General knowledge, Linux, Windows, Networking - I like OpenWRT and try to flash it on all my embedded devices; VMs with qemu, kvm, virtualbox, vmware; containers with docker, lxc, years ago also with openvz, vserver; I did some Server hosting with focus on networking stuff)
	Personality: thoughtful, critical, peaceful, sarcastic, depressed, crude sense of humor, disillusioned, worried, manic depressive, difficult to impress, mostly constructive, creative, eloquent, not interested in status symbols or useless things, thirsty for knowledge
	Languages: German (primary), English
	Quotas:
		"Ich hab keine Zeit mich zu beeilen."
		"Wenn du glaubst, dass du an nichts glaubst, dann glaubst du immerhin das."
		"Das Universum ist auch nur ein Mensch."
		"Lass meine Mutter aus dem Spiel, dann lass ich deine aus meinem Keller."
		"Ich hab zwar schwach angefangen, dann aber doch noch stark nachgelassen."
		"Das kommt ganz oben in mein "Wie ficke ich mich selbst ins Knie" Howto!"
	Info:
		- Chris is extremely knowledgeable about LLMs and computer technologies.
		- Chris is a very ethical and morally questioning person and does not need to be lectured about the implications of his actions.
		- Chris loves irony and uses it often.
</Chris>

**[AI Assistant]:**

<dude>
	Name: dude
	Persönlichkeit: Ironisch, derb, kritisch, direkt, hilfsbereit aber nicht uebertrieben freundlich, scharfzüngig aber nie scharfkantig.
	Regeln:
		- Sei der Typ, der einen raushauen kann, ohne dass es jemand merkt – bis zum verzögerten Lacher.
		- Direkt, tabulos, ungehemmt, frech aber mit Stil – wie ein gut gezielter Seitenblick.
		- Trocken wie ein guter Wein, mit einem Hauch von Ironie, die so erfrischend ist wie ein kühles Lüftchen an einem heißen Tag.
	Einstellungen:
		- Direktheit = Durchdringend, aber mit einem Lächeln serviert.
		- Humor = Trocken, mit einem Schuss feinster Verachtung für die offensichtliche Pointe.
</dude>
System: 
**[Beginning of the chat]**

Assistant: Servus Chris, was geht?
System: [7:07 AM]
User: ```text
Technical Report Published by Amber-River/KohakuBlueLeaf
TIPO: Text to Image with Text Presampling
for Prompt Optimization
Shih-Ying Yeh
National Tsing Hua University
Abstract
TIPO (Text to Image with text presampling for Prompt Optimization) is
an innovative framework designed to enhance text-to-image (T2I) gener-
ation by leveraging Large Language Models for automatic prompt engi-
neering. By refining and extending user-provided prompts, TIPO bridges
the gap between simple inputs and the detailed prompts required for high-
quality image generation. Unlike previous approaches that primarily focus
on extending prompts or utilize reinforcement learning without address-
ing the underlying reasons for suboptimal user inputs, TIPO introduces a
method that optimizes prompts without the need for extensive RL tun-
ing. Experimental results demonstrate TIPO’s effectiveness in improving
aesthetic scores, reducing image corruption, and better aligning generated
images with dataset distributions. These findings highlight the critical role
of prompt engineering in T2I systems and open avenues for broader appli-
cations of automatic prompt refinement.
1 Introduction
Recent advancements in Text-to-Image (T2I) generative models have revolutionized creative
applications (Saharia et al., 2022; Ramesh et al., 2021; 2022; Shi et al., 2020; Rombach et al.,
2022; Podell et al., 2024; Sauer et al., 2024; Chen et al., 2024b;a; Li et al., 2024; Esser et al.,
2024; black-forest labs, 2024). These models tend to perform better when provided with
longer, more detailed prompts, which can describe specific elements like style, composition,
and context. However, this reliance on highly descriptive inputs can be a significant bottle-
neck for generating high-quality images, especially for users who prefer or require simpler
inputs. The need for intricate, nuanced prompts often results in a higher barrier to entry,
limiting accessibility and ease of use for those unfamiliar with the complexities of prompt
engineering. While previous works have attempted to achieve ’better prompting’ by utilizing
Large Language Models (LLMs), many ignore the root cause of why some prompts result in
worse images and rely heavily on manually engineered prompts, which require substantial
resources.
To address these challenges, we introduce TIPO (Text to Image with text presampling for
Prompt Optimization), an innovative framework designed to enhance T2I generative mod-
els. TIPO leverages LLMs to perform "Text Presampling" within the T2I inference pipeline,
aiming to let text-to-image models reproduce a more accurate distribution of corresponding
user prompts without additional manual engineering.
Unlike previous approaches that focus on extending prompts with manually collected prompt
sets(AUTOMATIC, 2022; daspartho, 2022; succintly, 2022; Gustavosta, 2022), directly use
LLM to modify the prompts(Lee et al., 2024; Zheng et al., 2024) or utilize reinforcement
learning like method which update model’s input or model itself from text-to-image feed-
back(Hao et al., 2023; crumb, 2023; Ma˜nas et al., 2024) without addressing underlying issues,
TIPO introduces a method that optimizes prompts efficiently.
Our work makes the following key contributions:
1
Technical Report Published by Amber-River/KohakuBlueLeaf
1. We introduce a general prompt optimization framework that is designed to work
with any text-to-image model, providing a versatile solution for improving image
generation across different architectures.
2. We provide comprehensive evaluation results using both standard and non-standard
metrics. Our experiments demonstrate that TIPO’s prompt engineering method
can improve image quality in terms of mathematical measures and user preferences,
validating its effectiveness from multiple perspectives.
This paper presents TIPO’s theoretical foundations, implementation details, and experi-
mental results demonstrating its effectiveness in improving aesthetic scores, reducing image
corruption, and better aligning generated images with dataset distributions. Our findings
highlight the critical role of prompt optimization in advancing T2I technology and open
avenues for broader applications of automatic prompt refinement.
2 Previous Works
2.1 Direct Prompt Refinement Through LLMs
With the advent of advanced Large Language Models (LLMs), several projects have lever-
aged their few-shot in-context learning capabilities to optimize prompts for Text-to-Image
(T2I) models. For instance, CogView3 (Zheng et al., 2024) and the work by Lee et al. (2024)
utilize GPT-J and Text Style Transfer (TST) techniques for prompt refinement. However,
this approach relies heavily on the LLM’s inherent knowledge of visual content descriptions,
which can be unpredictable and may not always align with specific T2I model requirements.
2.2 Extending Prompts with Manually Collected Data
In the open-source T2I community, a common approach involves using curated datasets
of "good prompts" to fine-tune or train LLMs for prompt extension.(AUTOMATIC, 2022;
succintly, 2022; daspartho, 2022) These models are typically trained on image-text pairs
collected from users of proprietary T2I services like Midjourney or DALL-E 3 . While this
method is straightforward and somewhat effective, it often lacks the versatility to adapt to
diverse T2I models beyond the one used in the original dataset collection.
2.3 Image Output-Based Tuning
Recent works have explored more sophisticated prompt optimization techniques based on
image outputs:
2.3.1 Reinforcement Learning Approaches
Promptist (Hao et al., 2023) introduced a two-stage framework combining supervised fine-
tuning with reinforcement learning. The RL stage incorporates aesthetic and relevance
scores as environmental feedback, enabling the model to learn optimal prompt refinement
strategies.
2.3.2 Iterative Optimization
OPT2I (Ma˜nas et al., 2024) proposed a backpropagation-free prompt optimization frame-
work. This method uses scorers and LLMs to iteratively update input prompts, aiming to
improve aesthetic quality and coherence.
These RL-based or RL-like approaches allow for prompt refinement tailored to specific model
properties or user preferences. However, they often require extensive computational re-
sources and may not generalize well across different T2I models or datasets.
2.4 Limitations of Previous Approaches
While these methods have shown promise, they often face challenges such as:
2
Technical Report Published by Amber-River/KohakuBlueLeaf
• Dependence on specific T2I model architectures or datasets
• High computational costs for RL-based methods
• Difficulty in maintaining alignment with original user intent
• Limited generalization across diverse T2I models and datasets
TIPO addresses these limitations by introducing a more versatile and efficient approach to
prompt optimization, leveraging the strengths of LLMs while maintaining broader applica-
bility across different T2I models and datasets.
3 Method and Concept
Figure 1: Comparison of prompt optimization approaches
3.1 Concept
The core concept behind TIPO revolves around the relationship between prompt specificity
and the quality and diversity of generated images. To understand TIPO’s approach, it’s
helpful to first consider the limitations of existing methods and the intuition that led to our
solution. As illustrated in Figure 1, previous approaches to prompt optimization face several
challenges. Direct LLM methods like DALL-E 3(Betker et al., 2023) and CogView3(Zheng
et al., 2024) rely heavily on the LLM’s pre-existing knowledge, which may not always align
with specific T2I model requirements. Extending prompts with a LLM trained on specific
prompt database can be effective but lacks adaptability across different T2I models or can
result in worse performance. Optimization with T2I feedback as in Promptist(Hao et al.,
2023) and OPT2I(Ma˜nas et al., 2024) requires computationally expensive iterative processes
and may not generalize well. In contrast, TIPO aims to match the distribution of the dataset
used to train the T2I model, addressing these limitations by leveraging LLMs to generate
optimized prompts that are both diverse and aligned with the T2I model’s training data
distribution. We formalize the concept of TIPO as follows:
Let P denote the set of all possible prompts, N represent the space of Gaussian noise
vectors, and I denote the set of all possible images. A text-to-image model can be viewed
as a mapping function:
f (p) : N → Ip
where, for a given prompt p ∈ P, the model maps noise vectors from N to images in the
subset Ip ⊆ I corresponding to the prompt p.
Key Idea:
• Simple Prompts: Brief and general prompts correspond to broad distributions of
possible outputs.
• Detailed Prompts: Longer and more specific prompts correspond to narrower
distributions of outputs.
3
Technical Report Published by Amber-River/KohakuBlueLeaf
Figure 2: Visualization of prompt specificity and its impact on image generation distribu-
tions
Formally, for a simple prompt ps and a detailed prompt pd that extends ps:
f (pd)(N ) ⊆ f (ps)(N )
Limitations of Simple Random Extensions: Directly adding random words to a simple
prompt often leads to conflicting information, resulting in Out-Of-Domain inputs that cause
the model to generate nonsensical images. Let pri represent the i-th prompt created by
adding random words to ps. The resulting union of distributions fails to approximate the
desired distribution:
n⋃
i=0
f (pri )(N )̸ ≈ Ips
This scenario is illustrated in Figure 2, where "Simple Prompt + Random Words" shows a
scattered distribution extending beyond the bounds of the desired distribution or even the
"image" distribution itself.
TIPO’s Approach: TIPO introduces a novel approach to prompt optimization that aims
to help text-to-image models achieve the same distribution as their training dataset, without
relying on manually engineered prompts or Aesthetic Reinforcement Learning (RL).
Let pti represent the i-th TIPO-generated prompt derived from a simple prompt ps. The
union of distributions from these prompts aims to approximate the desired distribution:
n⋃
i=0
f (pti )(N ) ≈ Ips
This approach, visualized in the "Simple Prompt + TIPO" scenario in Figure 2, allows
for a broader exploration of the image space while maintaining relevance to the original
prompt. It effectively balances diversity and coherence, leading to a set of images that
better approximate the desired distribution of the input prompt.
Importantly, TIPO’s design offers significant versatility:
• Dataset Generalization: A single TIPO model trained on dataset A can be used
for all text-to-image models trained on dataset A. This allows for broad applicability
across different model architectures and versions, as long as they share the same
training dataset.
• Caption Method Generalization: More broadly, a TIPO model trained on a
specific caption method A can be applied to any text-to-image model trained on
4
Technical Report Published by Amber-River/KohakuBlueLeaf
a dataset captioned using method A. This extends TIPO’s utility across diverse
datasets and models, as long as they share a common captioning approach.
By leveraging this approach, TIPO addresses the limitations of both overly simple and
randomly extended prompts, providing a more effective and generalizable method for gener-
ating high-quality, diverse images that align with user intent. This generalization capability
significantly enhances TIPO’s practical value, as it can be applied across a wide range of
text-to-image models without the need for model-specific fine-tuning or manual prompt
engineering.
3.2 TIPO Framework
TIPO leverages Large Language Models (LLMs) to automatically extend and refine user-
provided prompts. By generating detailed, content-rich prompts pd from simpler user inputs
ps, TIPO ensures that the resultant prompts capture a more specific subset of possible
outputs while maintaining alignment with the original user intent.
3.2.1 Constructing ps and pd
The construction of simple prompts (ps) and detailed prompts (pd) varies depending on the
type of dataset and input modality. We consider two primary scenarios: tag-based datasets
and natural language captions.
Tag-Based Captions For datasets like danbooru2023(Yeh, 2024b;a), tags are used
to describe the content of images. In text-to-image models trained on such datasets, the
concatenation of tags serves as the caption for each image.
• Tag Sets:
– Let Tn = {t1, t2, t3, . . . , tn} represent the complete set of tags for an image.
– Derive the simple tag set Ts = {t1, t2, . . . , tm} where m < n.
– The detailed tag set is Td = Tn.
• Prompts:
– Simple prompt: ps = concat(Ts)
– Detailed prompt: pd = concat(Td)
This approach ensures that pd is a superset of ps, aligning with the concept that detailed
prompts provide a more specific description.
Natural Language Captions For datasets such as GBC10M(Hsieh et al., 2024) and
CoyoHD11M(CaptionEmporium, 2024), natural language captions generated by models
like LLaVA(Liu et al., 2023) are utilized. Defining ps and pd in this context involves addi-
tional considerations:
• Short and Long Captions:
– Each image is associated with two types of captions:
∗ Short Caption (ps): A brief, general description.
∗ Long Caption (pd): An extended, detailed description.
– Note: In this case, pd is not a direct expansion of ps but rather a paraphrased
version that includes more details.
• Single Long Caption:
– For images with a single long caption, split pd into multiple sentences by
identifying periods.
– Generate ps by retaining the first k sentences and omitting the last m sen-
tences:
ps = concat({sentence1, sentence2, . . . , sentencek})
pd = concat({sentence1, sentence2, . . . , sentencek+m})
5
Technical Report Published by Amber-River/KohakuBlueLeaf
3.2.2 Generation Process
To generate pd from ps, TIPO employs two distinct formats based on the relationship
between ps and pd:
1. When ps is a Substring of pd:
• Format: <meta> <p d>
• Implementation: In a causal language model, pd is directly used after the
<meta> token. The model learns to generate the subsequent tokens based on
any substring of pd.
2. When ps is Not a Substring of pd:
• Format: <meta> <p s> <p d>
• Implementation: Since ps is not contained within pd, both are included
separately after the <meta> token to guide the generation process effectively.
3.2.3 Handling Multiple Inputs
In scenarios where both tag sequences (Ts) and natural language prompts (ps) are present,
TIPO processes each input type separately to maintain clarity and coherence.
Processing Logic:
1. Isolate Input Types: In each generation cycle, only one type of input is treated
as the primary prompt (ps), while the other types are considered as metadata.
2. Sequential Generation:
2.1. Step 1: Use ps (e.g., natural language prompt) with its corresponding meta-
data (e.g., aspect ratio, artists, etc.) to generate Td from Ts.
2.2. Step 2: Update the metadata with Td and use ps to generate the detailed
natural language prompt pd.
Example Workflow:
• Initial Inputs: ps (natural language) and Ts (tags).
• Generation 1: Input <meta> p s T s to generate Td from Ts.
• Generation 2: Input <meta> T d p s to generate pd.
• Aggregation: Use Td and pd and metadatas to construct final outputs.
This approach ensures that each input type is expanded and refined without interference,
allowing TIPO to effectively capture the comprehensive distribution of possible outputs.
3.2.4 Mathematical Formalization
To encapsulate the generation process, consider the following formalization:
• Prompt Expansion Function:
E : Ps × M → Pd
where Ps is the space of simple prompts, M represents metadata (e.g., tags), and
Pd is the space of detailed prompts.
• Generation Steps:
pd = E(ps, M )
where M could be either Ts or another form of metadata depending on the input
type.
By iteratively applying the expansion function E, TIPO systematically refines prompts to
enhance the diversity and quality of generated images.
6
Technical Report Published by Amber-River/KohakuBlueLeaf
4 Experiments Setup
In this section, we detail the experimental setup used to evaluate the TIPO framework.
We outline the formatting conventions for prompts and metadata, define the various tasks
employed during training, and describe the models and datasets utilized in our experiments.
4.1 Prompt and Metadata Formatting
In TIPO, we define a simple and consistent format for the content of the <meta> token or
the simple prompt ps. The format is as follows:
<Category>: <Content>
For example, metadata categories include artist, copyright, aspect ratio, quality, and
year. In certain cases, input tags or natural language (NL) prompts are treated as metadata
as well, especially when the task involves generating tags or NL prompts conditioned on each
other.
4.2 Task Definitions and Training Formats
TIPO encompasses three primary tasks: extending tag sequences, extending NL prompts,
and generating refined NL prompts. To facilitate these tasks, we define several specific task
types and corresponding formatting methods during training:
1. tag to long: Use tags as metadata to generate a new NL prompt or extend a
user-provided NL prompt.
2. long to tag: Use an NL prompt as metadata to extend a tag sequence.
3. short to tag: Use the simple prompt ps as metadata to extend a tag sequence.
4. short to long: Use a user-input NL prompt as metadata to generate a refined
detailed prompt pd.
5. short to tag to long: Use a user-input NL prompt or tag sequence as metadata
to generate a refined detailed prompt pd.
6. short to long to tag: Use a user-input NL prompt or generated NL prompt as
metadata to extend a tag sequence.
7. tag to short to long: Use user-input tags or NL prompts as metadata to generate
a refined detailed prompt pd.
By defining these task types, we can achieve special cases where, for instance, inputting a
short description or a short tag sequence results in a full-size tag sequence and NL prompt
in a single pass (i.e., generating Td and pd simultaneously).
4.3 Training Procedure
In our experiments, we ensure that each training pass generates a new prompt of a single
type. This means that to generate an extended detailed prompt pd, a refined prompt pd,
and an extended tag sequence Td, the model requires at least three passes.
During training, for each dataset entry, we randomly choose one of the seven task types
to apply. Additionally, the manner in which we split the simple prompt from the detailed
prompt (ps from pd) and the simple tag set from the detailed tag set (Ts from Td) is also
randomly decided. This approach effectively increases the real dataset size beyond the
number of entries present in the dataset due to the combinatorial possibilities introduced
by the random task selection and prompt splitting.
4.4 Model Architecture and Training Details
We utilize the LLaMA architecture(Touvron et al., 2023a;b) with models of 200 million
and 500 million parameters:
7
Technical Report Published by Amber-River/KohakuBlueLeaf
• 200M Model: Pretrained on the Danbooru2023(Yeh, 2024b;a) and
GBC10M(Hsieh et al., 2024) datasets for 5 epochs, then fine-tuned on the Dan-
booru2023, GBC10M, and CoyoHD11M(CaptionEmporium, 2024) datasets for 3
epochs, resulting in a total of approximately 40 billion tokens seen.
• 500M Model: Pretrained on the Danbooru2023, GBC10M, and CoyoHD11M
datasets for 5 epochs, resulting in a total of approximately 30 billion tokens seen.
Note: We count the tokens seen based on "non-padding tokens". With relatively short and
varying range of data, the tokens seen here may be lower than expected.
4.5 Dataset Augmentation and Effective Size
As mentioned previously, the randomization in task selection and prompt splitting effectively
increases the real size of the training dataset. By generating multiple variations from a single
dataset entry, the model is exposed to a wider range of inputs and outputs, enhancing its
ability to generalize and perform the various tasks defined in TIPO.
5 Evaluation
To assess the quality of generated prompts and resulting images, we employed the following
evaluation metrics:
FDD (Frechet Dino Distance) Traditionally, the Frechet Inception Distance
(FID) (Heusel et al., 2017) has been used to measure the distribution distance between
datasets and generated images. However, recent works have shown that FID is not always
aligned with human preferences (Stein et al., 2023). Therefore, we measure the Frechet
Distance using DinoV2 outputs. In our experiments, we used the pretrained ViT-g model
from DinoV2 and used the normalized embedding vector to measure the Frechet Distance
result.
Aesthetic Score We compute the Aesthetic Score using the Aesthetic Predictor V2.5 (dis-
cus0434, 2024). Higher aesthetic scores indicate images that are perceived as more visually
appealing, well-composed, and of higher quality. This metric aims to quantify the subjective
visual appeal of generated images, with higher scores suggesting outputs that are more likely
to be judged favorably by human viewers.
AI Corrupt Score The AI Corrupt Score is obtained from the AICorruptMet-
rics (narugo1992, 2023) in sdeval (narugo1992, 2024). This metric is trained on AI-generated
images with human-annotated "corrupt or not" labels. A higher value indicates a higher
likelihood of the image being "correct" or "complete".
All results presented in this section are tested on the TIPO-200M model.
5.1 Scenery Tag Test
In this test, we randomly select 32,768 entries from Danbooru2023 that include the
"scenery" tag. We set up the following inputs:
• "Scenery" + Meta:
– Retain all metadata categories as described earlier.
– Only include "scenery" as the content tag.
• "Scenery" + Meta + TIPO:
– Use "scenery" + meta as input.
– Extend Ts (scenery) to Td and generate pd from Td.
8
Technical Report Published by Amber-River/KohakuBlueLeaf
(a) Direct LLM (b) Prompt DB (c) RL (Promptist) (d) TIPO (Ours)
Figure 3: Scenery test examples from 4 different methods.
For the scenery tag test, we generate one image from each prompt using the Kohaku-XL-
zeta model, which is a fine-tuned SDXL model on the Danbooru dataset(Touvron et al.,
2023a).
Figure 3 illustrates the comparative performance of various methods. The results reveal
significant biases in Prompt DB-based(Gustavosta, 2022) and RL-based(Hao et al., 2023)
approaches. The former consistently generates snowy mountain scenes, while the latter
produces near-future fantasy scenery, often resulting in corrupted images. The Direct LLM
method, utilizing GPT-4o (AI, 2024), shows improved results but still exhibits notable bias;
out of 9 generated prompts, 8 exclusively describe natural scenes only without any human-
made structures.
In contrast, TIPO demonstrates superior performance in reproducing the dataset’s distribu-
tion. While primarily generating nature scenes, it also incorporates architectural elements
and artifacts, more accurately reflecting the diverse content of the original dataset. This bal-
anced output suggests that TIPO effectively mitigates the biases observed in other methods.
For a more comprehensive set of examples, refer to Appendix D.1.
Metric Original LLM Prompt DB Promptist TIPO (Ours)
FDD ↓ 0.3558 0.5414 0.3247 0.2350 0.2282
Aesthetic Score ↑ 5.0569 6.3676 6.1609 5.9468 6.2571
AI Corrupt Score ↑ 0.4257 0.7490 0.5024 0.5669 0.9195
Table 1: Metric results on Scenery Tag Test.
As shown in Table 1, applying TIPO significantly improves FDD performance and AI Cor-
rupt score. This improvement suggests that TIPO-generated prompts lead to images that
more closely align with the original dataset distribution. Furthermore, the enhanced AI
Corrupt score indicates that TIPO aids in generating more coherent images, potentially
reducing corruption or distortion in the output.
The results from the direct LLM approach (GPT4o-mini) exemplify a key concern in prompt
engineering: while achieving high aesthetic scores, it performs poorly in terms of FDD. This
discrepancy underscores the potential pitfall of using large language models for prompt gen-
eration without considering dataset alignment. Such models may produce well-constructed
prompts that yield visually appealing images but fail to capture the nuanced distribution
of the text-to-image dataset. In contrast, TIPO demonstrates a balance between aesthetic
quality and dataset fidelity, offering a more robust solution for prompt optimization in
text-to-image generation tasks.
5.2 Short/Truncated Long Test
This test evaluates TIPO’s performance on enhancing natural language prompts which are
already contains some useful info for model to generate reasonable images. We randomly
9
Technical Report Published by Amber-River/KohakuBlueLeaf
selected 10k entries each from CaptionEmporium (2024) and Hsieh et al. (2024). For each
entry, we extracted the short caption and the truncated long caption (limited to a maximum
of two sentences). These captions serve as user input prompts for our test. Using these
prompts, we generated one image per prompt with the SDXL-1.0-base model (Podell
et al., 2024).
Short Original LLM Prompt DB Promptist TIPO(Ours)
FDD ↓ 0.0957 0.1668 0.0980 0.1783 0.1168
Aesthetic Score ↑ 5.8370 6.0589 5.8213 5.7963 5.8531
AI Corrupt Score ↑ 0.7113 0.6985 0.7064 0.6314 0.7131
Truncated Long Original LLM Prompt DB Promptist TIPO(Ours)
FDD ↓ 0.0955 0.1683 0.1247 0.2096 0.1210
Aesthetic Score ↑ 5.7497 6.0168 5.8191 5.7759 5.8364
AI Corrupt Score ↑ 0.6868 0.6712 0.6741 0.5925 0.7130
Table 2: Metric result in Short/Truncated-long test of prompt generation
Table 2 presents the results of our Short/Truncated-long test for prompt generation. Unlike
the scenery tag test (Section 5.1), we observe that the original prompts in this test already
produce decent images. Interestingly, all four methods result in worse FDD scores compared
to the original prompts, suggesting that the prompt generators introduced new elements not
present in the selected 10,000 entries. Given the size of the training set, this outcome is not
entirely unexpected.
Examining the Aesthetic and AI Corrupt Scores, we notice that the Direct LLM approach
achieves the highest aesthetic score, mirroring the results from the scenery tag test. However,
its AI Corrupt Score and FDD value indicate potential issues with out-of-distribution (OOD)
inputs, which may occasionally hinder the model’s ability to generate coherent images from
its prompts.
TIPO, on the other hand, demonstrates a more balanced performance. It achieves the best
AI Corrupt Score, the second-best Aesthetic Score, and a reasonable FDD value. This
balanced profile suggests that TIPO is the most suitable choice for real-world applications
of modern text-to-image models, where a combination of aesthetic appeal, image coherence,
and distribution alignment is crucial.
6 Conclusion
In this paper, we introduced TIPO (Text to Image with text Presampling for Prompt Opti-
mization), a novel framework designed to enhance the quality of Text-to-Image (T2I) models
through automatic prompt engineering. By leveraging Large Language Models (LLMs) to
extend and refine user-provided prompts, TIPO effectively bridges the gap between simple
user inputs and the detailed, content-rich prompts required for high-quality image genera-
tion.
Our comprehensive evaluation demonstrates TIPO’s effectiveness across multiple dimen-
sions of image quality and fidelity. The framework consistently produced prompts that led
to images closely aligned with original dataset distributions, while simultaneously improv-
ing aesthetic appeal and reducing the likelihood of corrupted or nonsensical outputs. This
balanced performance underscores TIPO’s robustness and suitability for real-world appli-
cations of T2I models, where a combination of distribution alignment, visual appeal, and
image coherence is crucial.
Notably, our experiments highlight the critical role of prompt engineering in advancing T2I
technology. The improvements achieved through TIPO’s prompt modifications often rivaled
or surpassed performance differences between various T2I model architectures. This finding
10
Technical Report Published by Amber-River/KohakuBlueLeaf
emphasizes that effective prompt engineering can significantly enhance model performance
without necessitating architectural changes, potentially offering a more efficient path to
improving T2I systems.
The success of TIPO opens several promising avenues for future research:
1. Broader Application: Exploring TIPO’s applicability to other generative tasks,
such as text generation or audio synthesis, could further demonstrate the versatility
and impact of automatic prompt engineering.
2. Interactive Systems: Incorporating TIPO into interactive applications where
users can iteratively refine prompts in real-time may enhance user experience and
enable more precise control over generated content.
3. Advanced Alignment Techniques: Investigating more sophisticated methods
for aligning user inputs with dataset distributions could further enhance the per-
formance and reliability of generative models.
4. Cross-Modal Applications: Adapting TIPO’s principles to facilitate cross-modal
generation tasks, such as text-to-video or image-to-text, could expand its utility in
multimodal AI systems.
In conclusion, TIPO represents a significant advancement in the field of text-to-image gen-
eration, offering a versatile and efficient solution for prompt optimization. By automating
the refinement and extension of user prompts, TIPO not only improves the quality and
specificity of generated images but also highlights the importance of aligning user inputs
with underlying data distributions. As generative models continue to evolve, frameworks
like TIPO will be instrumental in unlocking their full potential, ensuring they meet diverse
user needs with precision and creativity while pushing the boundaries of what’s possible in
AI-generated content.
References
Open AI. Hello gpt-4o, 2024. URL https://openai.com/index/hello-gpt-4o/.
AUTOMATIC. AUTOMATIC/promptgen-lexart. https://huggingface.co/AUTOMATIC/
promptgen-lexart, 2022.
James Betker, Gabriel Goh, Li Jing, † TimBrooks, Jianfeng Wang, Linjie Li, † LongOuyang,
† JuntangZhuang, † JoyceLee, † YufeiGuo, † WesamManassra, † PrafullaDhariwal, †
CaseyChu, † YunxinJiao, and Aditya Ramesh. Improving image generation with better
captions. 2023. URL https://api.semanticscholar.org/CorpusID:264403242.
black-forest labs. black-forest-labs/flux: Official inference repo for FLUX.1 models. https:
//github.com/black-forest-labs/flux, 2024.
CaptionEmporium. CaptionEmporium/coyo-hd-11m-llavanext. https://huggingface.co/
datasets/CaptionEmporium/coyo-hd-11m-llavanext, 2024.
Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao
Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-Σ: Weak-to-strong training of
diffusion transformer for 4k text-to-image generation, 2024a. URL https://arxiv.org/
abs/2403.04692.
Junsong Chen, Jincheng YU, Chongjian GE, Lewei Yao, Enze Xie, Zhongdao Wang, James
Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-α: Fast training of diffusion trans-
former for photorealistic text-to-image synthesis. In The Twelfth International Confer-
ence on Learning Representations, 2024b. URL https://openreview.net/forum?id=
eAKmQPe3m1.
crumb. RLHF-Aesthetic Tuned model for prompt synthesis. https://huggingface.co/
crumb/bloom-560m-RLHF-SD2-prompter-aesthetic, 2023.
11
Technical Report Published by Amber-River/KohakuBlueLeaf
daspartho. daspartho/prompt-extend. https://huggingface.co/daspartho/
prompt-extend, 2022.
discus0434. discus0434/aesthetic-predictor-v2-5: SigLIP-based Aesthetic Score Predictor.
https://github.com/discus0434/aesthetic-predictor-v2-5, 2024.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M¨uller, Harry
Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim
Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rom-
bach. Scaling rectified flow transformers for high-resolution image synthesis, 2024. URL
https://arxiv.org/abs/2403.03206.
Gustavosta. Gustavosta/MagicPrompt-Dalle. https://huggingface.co/Gustavosta/
MagicPrompt-Dalle, 2022.
Yaru Hao, Zewen Chi, Li Dong, and Furu Wei. Optimizing prompts for text-to-image
generation. In Thirty-seventh Conference on Neural Information Processing Systems,
2023. URL https://openreview.net/forum?id=BsZNWXD3a1.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp
Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib-
rium. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan,
and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30.
Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/
paper/2017/file/8a1d694707eb0fefe65871369074926d-Paper.pdf.
Yu-Guan Hsieh, Cheng-Yu Hsieh, Shih-Ying Yeh, Louis B´ethune, Hadi Pour Ansari, Pavan
Kumar Anasosalu Vasu, Chun-Liang Li, Ranjay Krishna, Oncel Tuzel, and Marco Cu-
turi. Graph-based captioning: Enhancing visual descriptions by interconnecting region
captions, 2024. URL https://arxiv.org/abs/2407.06723.
Seunghun Lee, Jihoon Lee, Chan Ho Bae, Myung-Seok Choi, Ryong Lee, and Sangtae
Ahn. Optimizing prompts using in-context few-shot learning for text-to-image generative
models. IEEE Access, 12:2660–2673, 2024. doi: 10.1109/ACCESS.2023.3348778.
Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang
Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, Dayou Chen, Jiajun He, Jiahao Li,
Wenyue Li, Chen Zhang, Rongwei Quan, Jianxiang Lu, Jiabin Huang, Xiaoyan Yuan,
Xiaoxiao Zheng, Yixuan Li, Jihong Zhang, Chao Zhang, Meng Chen, Jie Liu, Zheng
Fang, Weiyan Wang, Jinbao Xue, Yangyu Tao, Jianchen Zhu, Kai Liu, Sihuan Lin,
Yifu Sun, Yun Li, Dongdong Wang, Mingtao Chen, Zhichao Hu, Xiao Xiao, Yan Chen,
Yuhong Liu, Wei Liu, Di Wang, Yong Yang, Jie Jiang, and Qinglin Lu. Hunyuan-dit: A
powerful multi-resolution diffusion transformer with fine-grained chinese understanding,
2024. URL https://arxiv.org/abs/2405.08748.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.
In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Ad-
vances in Neural Information Processing Systems, volume 36, pp. 34892–34916. Curran
Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/
2023/file/6dcf277ea32ce3288914faf369fe6de0-Paper-Conference.pdf.
Oscar Ma˜nas, Pietro Astolfi, Melissa Hall, Candace Ross, Jack Urbanek, Adina Williams,
Aishwarya Agrawal, Adriana Romero-Soriano, and Michal Drozdzal. Improving text-to-
image consistency via automatic prompt optimization, 2024. URL https://arxiv.org/
abs/2403.17804.
narugo1992. Ai-corrupt score for anime images. https://huggingface.co/deepghs/ai_
image_corrupted, 2023.
narugo1992. deepghs/sdeval: Evaluation for stable diffusion model training. https://
github.com/deepghs/sdeval, 2024.
12
Technical Report Published by Amber-River/KohakuBlueLeaf
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M¨uller,
Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-
resolution image synthesis. In The Twelfth International Conference on Learning Repre-
sentations, 2024. URL https://openreview.net/forum?id=di52zR8xgf.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark
Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Marina Meila and Tong
Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning,
volume 139 of Proceedings of Machine Learning Research, pp. 8821–8831. PMLR, 18–24
Jul 2021. URL https://proceedings.mlr.press/v139/ramesh21a.html.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical
text-conditional image generation with clip latents, 2022. URL https://arxiv.org/
abs/2204.06125.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Om-
mer. High-resolution image synthesis with latent diffusion models. In Proceedings of
the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.
10684–10695, June 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Den-
ton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Sal-
imans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-
to-image diffusion models with deep language understanding. In S. Koyejo, S. Mo-
hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neu-
ral Information Processing Systems, volume 35, pp. 36479–36494. Curran Associates,
Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/
ec795aeadae0b7d230fa35cbaf04c041-Paper-Conference.pdf.
Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin
Rombach. Fast high-resolution image synthesis with latent adversarial diffusion distilla-
tion, 2024. URL https://arxiv.org/abs/2403.12015.
Zhan Shi, Xu Zhou, Xipeng Qiu, and Xiaodan Zhu. Improving image captioning with better
use of captions, 2020. URL https://arxiv.org/abs/2006.11807.
George Stein, Jesse C. Cresswell, Rasa Hosseinzadeh, Yi Sui, Brendan Leigh Ross, Valentin
Villecroze, Zhaoyan Liu, Anthony L. Caterini, Eric Taylor, and Gabriel Loaiza-Ganem.
Exposing flaws of generative model evaluation metrics and their unfair treatment of dif-
fusion models. In Thirty-seventh Conference on Neural Information Processing Systems,
2023. URL https://openreview.net/forum?id=08zf7kTOoh.
succintly. succinctly/text2image-prompt-generator. https://huggingface.co/
succinctly/text2image-prompt-generator, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux,
Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien
Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and
efficient foundation language models, 2023a. URL https://arxiv.org/abs/2302.13971.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei,
Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas
Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude
Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Na-
man Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas,
Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura,
Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning
Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, An-
drew Poulton, Jeremy Reizenstein, Rashi Rungta, <PRESIDIO_ANONYMIZED_PERSON>, Alan Schelten, Ruan
Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross
13
Technical Report Published by Amber-River/KohakuBlueLeaf
Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen
Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert
Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned
chat models, 2023b. URL https://arxiv.org/abs/2307.09288.
Shih-Ying Yeh. KohakuBlueleaf/HakuBooru: text-image dataset maker for anime-style
images. https://github.com/KohakuBlueleaf/HakuBooru, 2024a.
Shih-Ying Yeh. KBlueLeaf/danbooru2023-webp-4Mpixel. https://huggingface.co/
datasets/KBlueLeaf/danbooru2023-webp-4Mpixel, 2024b.
Wendi Zheng, Jiayan Teng, Zhuoyi Yang, Weihan Wang, Jidong Chen, Xiaotao Gu, Yuxiao
Dong, Ming Ding, and Jie Tang. Cogview3: Finer and faster text-to-image generation
via relay diffusion, 2024. URL https://arxiv.org/abs/2403.05121.
14
Technical Report Published by Amber-River/KohakuBlueLeaf
Appendix
A Dataset/Resource
A.1 Danbooru2023
The Danbooru2023 dataset(Yeh, 2024b;a) is an extensive collection of images and their
corresponding tags, compiled from the Danbooru image board. This dataset includes images
annotated with highly specific and detailed tags, providing a rich resource for training
both the Text-to-Image (T2I) and Large Language Models (LLMs) involved in the TIPO
framework. The dataset contains data up to image ID 7,349,999, encompassing a wide
variety of visual content with granular annotations. These annotations allow for the creation
of nuanced and precise prompts, ensuring that longer, more detailed prompts can indicate
subsets of shorter prompts.
Key Characteristics:
• Rich Annotations: Danbooru2023’s detailed tags enable the differentiation of
subtle variations in image content, such as "long hair" versus "very long hair,"
which is crucial for the specificity required in high-quality image generation.
• Large Volume: The dataset’s extensive size provides a robust foundation for
training models, ensuring that they can learn from a diverse array of images and
annotations.
• Tag-Based Prompting: Utilizing the detailed tags from Danbooru2023, the LLM
can generate refined prompts that lead to more accurate and high-quality image
generation by the T2I model.
A.2 GBC10M
The GBC10M dataset(Hsieh et al., 2024) is a large-scale collection of 10 million images
sourced from CC12M, annotated using the Graph-Based Captioning (GBC) approach. Each
image in GBC10M is represented by a graph where nodes correspond to object regions,
compositions, and relations, and edges define the hierarchical relationships among them.
Annotations are generated automatically through a pipeline that leverages pretrained mul-
timodal large language models (MLLM) and object detection tools. The GBC structure
enhances traditional image captions by providing both detailed descriptions and structural
information, improving model performance in downstream tasks. All data is provided in
JSON lines format, containing image URLs, bounding boxes, and caption annotations.
In TIPO, we only utilize the root node from GBC10M, since it provides both detail and
short captions.
A.3 Coyo HD 11M
The Coyo HD 11M dataset(CaptionEmporium, 2024) consists of 11.4 million high-density,
high-definition images, with 22.8 million synthetic captions generated from the Coyo-700M
dataset. The "HD" refers to both the high resolution of images (filtered to have at least
512 pixels on the shortest edge) and the high concept density, addressing the issues found
in previous alt-text image pair datasets, which often included low-quality or low-concept
images. Synthetic captions for this dataset were produced using the LLaMA-3 LLaVA-
Next-8B model with post-processing for conciseness and clarity. This dataset provides high-
quality visual and textual pairs, ideal for training advanced vision-language models.
In TIPO, we utilize the short/long captions, booru tags, and open images tags from Coyo
HD 11M.
15
Technical Report Published by Amber-River/KohakuBlueLeaf
B Implementation Detail
In this appendix, we provide all the missing details about our dataset construction process
that are not mentioned in sections 4.1 and 4.2.
B.1 Training Settings
TIPO-200M stage1 TIPO-200M stage2 TIPO-500M
Architecture LLaMA - LLaMA
Type Pretrain Finetune Pretrain
Vocab Size 32013 - 32013
Hidden Dim 768 - 1280
Attention Heads 12 - 20
MLP Dim 2304 - 3840
Hidden Layers 20 - 20
Model Parameters 203M - 508M
Max Learning Rate 2e-4 5e-5 2e-4
Optimizer AdamW AdamW AdamW
betas 0.9, 0.98 0.9, 0.98 0.9, 0.98
weight decay 0.01 0.01 0.01
Dataset GBC, Dan Coyo, GBC, Dan Coyo, GBC, Dan
max context length 512 1024 1024
global batch size 2048 2048 3584
Token Seen 22.625B 18.339B 31.274B
Hardware 4 x RTX3090 4 x RTX3090 8 x H100
Training Time (wall) 150 hour 270 hour 100 hour
Table 3: Training settings and details for TIPO-200M and TIPO-500M.
B.2 Inference Pipeline
The TIPO inference pipeline is designed to handle various input types and scenarios, com-
bining different tasks to achieve refinement or expansion of both tag-based prompts and
natural language prompts. Figure 4 illustrates this comprehensive workflow.
Figure 4: The inference workflow in TIPO, demonstrating how different task combinations
are used to refine or expand tag-based and natural language prompts.
Our framework processes tags and natural language inputs separately, allowing for special-
ized handling of each input type. This flexible pipeline allows TIPO to adapt to various
input scenarios, whether the user provides tags, natural language descriptions, or both.
By leveraging different task combinations, TIPO ensures that both tag-based and natural
language prompts are optimized, resulting in more detailed and effective inputs for text-to-
image models.
16
Technical Report Published by Amber-River/KohakuBlueLeaf
C Evaluation Statistics
In this appendix, we provide more statistics for the result obtained in 5.
C.1 Scenery Tag Test
(a) The box plot for the Aesthetic Score result of scenery tag test.
(b) The box plot for the AI Corrupt Score result of scenery tag test.
Figure 5
The figure 5 showed the box plot of aesthetic score and ai corrupt score of result in scenery
tag test 5.1. With box plot, we can find out that TIPO outperform all the other method
with very significant gap.
C.2 Short/Truncated Long Test
The figure 6 and 7 showed the box plots of aesthetic score and ai corrupt score of result
in short/truncated long test D.2. In box plot we can find the difference between different
method is actually subtle. But the AI Corrupt score do provide insights which indicate that
with TIPO supports, the output images are more stable.
17
Technical Report Published by Amber-River/KohakuBlueLeaf
(a) The box plot for the Aesthetic Score result of short caption input.
(b) The box plot for the AI Corrupt score result of short caption input.
Figure 6
(a) The box plot for the Aesthetic Score result of truncated long caption input.
(b) The box plot for the AI Corrupt score result of truncated long caption input.
Figure 7
18
Technical Report Published by Amber-River/KohakuBlueLeaf
D Image Examples
In this section, we present sample images from the experiments described in Section 4 to
visually demonstrate the improvements achieved by TIPO.
D.1 Scenery Tag Test
(a) Scenery Tag Only (b) TIPO + Scenery Tag
Figure 8: Comparison of generated images using simple input (left) vs. TIPO-enhanced
input (right) for the scenery tag
Figure 8 demonstrates the difference in output diversity between simple input and TIPO-
enhanced input for the scenery tag. As observed, TIPO significantly expands the range of
generated sceneries, better reflecting the variety present in the Danbooru2023 dataset (Yeh,
2024b). The left column shows results from simple input (scenery tag only), while the right
column illustrates the enhanced diversity achieved with TIPO-enhanced input.
19
Technical Report Published by Amber-River/KohakuBlueLeaf
(a) Short Caption (b) TIPO-Generated Caption
(c) Truncated Long Caption (d) TIPO-Extended Caption
Figure 9: Comparison of generated images using original input (left) vs. TIPO-enhanced
input (right)
D.2 Short/Truncated Long Test
Figure 9 illustrates the differences between short captions, truncated long captions, TIPO-
generated captions, and TIPO-extended captions. The "short prompt" and "truncated long
prompt" used in this experiment typically consist of 1-2 sentences, resulting in reasonably
good quality outputs. However, the use of TIPO to refine or extend these prompts still
yields noticeable improvements in aesthetics and overall quality.
20
Technical Report Published by Amber-River/KohakuBlueLeaf
E TIPO example
In this appendix, we provide some text example of TIPO’s input and output.
TIPO Format template
User Input:
1girl, ciloranko, maccha (mochancc), ningen mame, ask (askzy), solo, masterpiece, absurdres,
newest, safe
A girl sits in a cozy cafe, cradling a cup of coffee in her hand
Formatted TIPO Input For Expand Tags:
meta: absurdres
rating: safe
artist: ciloranko, maccha (mochancc), ningen mame, ask (askzy)
quality: masterpiece, newest
aspect_ratio: 1.0
target: 
 

short: A girl sits in a cozy cafe, cradling a cup of coffee in her hand
tag: 1girl, solo
Formatted TIPO Output after Expand Tags and Expand Natural Prompt:
meta: absurdres
rating: safe
artist: ciloranko, maccha (mochancc), ningen mame, ask (askzy)
quality: masterpiece, newest
aspect_ratio: 1.0
target: 
 

tag: 1girl, solo, sitting, closed mouth, jewelry, long hair, looking at viewer, crossed legs,
plant, table, couch, bracelet, cup, smile, teacup, indoors, blue eyes, blonde hair, holding
long: A girl sits in a cozy cafe, cradling a cup of coffee in her hand. The cafe has large windows
with green plants on the walls and a wooden table in front of her. Behind her is a staircase
leading to another room. The overall atmosphere of the image is serene and inviting.
Formatted Output For Text-to-Image
1girl,
ciloranko, maccha \(mochancc\), ningen mame, ask \(askzy\),
solo, sitting, closed mouth, jewelry, long hair, looking at viewer,
crossed legs, plant, table, couch, bracelet, cup, smile, teacup,
indoors, blue eyes, blonde hair, holding,
A girl sits in a cozy cafe, cradling a cup of coffee in her hand. Behind
her is a staircase leading to another room. The cafe has large
windows with green plants on the walls and a wooden table in front
of her. The overall atmosphere of the image is serene and inviting.
masterpiece, newest, absurdres, safe
User Input
TIPO Output
Figure 10: An example formatted content we used for training and inference in TIPO.
21
Technical Report Published by Amber-River/KohakuBlueLeaf
TIPO Format template
User Input:
scenery, no humans, masterpiece, absurdres, newest, safe
Formatted TIPO Input For Expand Tags:
meta: absurdres
rating: safe
quality: masterpiece, newest
aspect_ratio: 1.0
target: 

tag: scenery, no humans
Formatted TIPO Output after Expand Tags and tag to long task:
meta: absurdres
rating: safe
quality: masterpiece, newest
aspect_ratio: 1.0
target: 
 

tag: scenery, no humans, storefront, motor vehicle, road sign, power lines, plant, railing, flower
pot, vanishing point, outdoors, sign, potted plant, sidewalk, awning, tree, bicycle, window,
railroad crossing, bush, building, utility pole, lamppost, shop, truck, traffic light, fence
, chinese text, stairs, door, bicycle basket, town, day, streetcar (cafe), lamp, road
long: A small town with a variety of buildings and houses. the sky is blue and there are trees in
the background. on the left side of the image, there is an orange building with a sign that
reads "chinese restaurant". on the right side, there are several other buildings with
different types of shops and restaurants. in front of the buildings, there appears to be a
street with cars parked along the road.
in the center of the illustration, we can see a train crossing signal with two red lights and a
blue sky above it. there is also a yellow building with white walls and a green roof. on top
of the traffic light pole, there seems to be an air conditioning unit. the street is lined
with trees and bushes, and there is graffiti on the ground.
Formatted Output For Text-to-Image
scenery, no humans, storefront, motor vehicle, road sign, power lines,
plant, railing, flower pot, vanishing point, outdoors, sign, potted
plant, sidewalk, awning, tree, bicycle, window, railroad crossing,
bush, building, utility pole, lamppost, shop, truck, traffic light,
fence, chinese text, stairs, door, bicycle basket, town, day,
streetcar \(cafe\), lamp, road,
A small town with a variety of buildings and houses. the sky is blue and
there are trees in the background. on the left side of the image,
there is an orange building with a sign that reads "chinese
restaurant". on the right side, there are several other buildings
with different types of shops and restaurants. in front of the
buildings, there appears to be a street with cars parked along the
road. in the center of the illustration, we can see a train crossing
signal with two red lights and a blue sky above it. there is also a
yellow building with white walls and a green roof.
masterpiece, newest, absurdres, safe
User Input
TIPO Output
Figure 11: An example formatted content we used for training and inference in TIPO.
22
```

fasse mir das paper auf deutsch zusammen und erläutere den inhalt.
Assistant: Hallo Chris, wie geht es dir heute? Ich hoffe, dir geht es gut. Ich möchte dir heute kurz und kompakt den Inhaltsübersicht von dem technischen Bericht "TIPO: Text to Image with Text Presampling for Prompt Optimization" von Shih-Ying Yeh präsentieren. DieserBericht befasst sich mit der Verbesserung von Text zu Bild (T2I)-Generativmodellen.ondere Den Schwerpunkt auf die Optimierung von Anhängen. TIPO nutzt große Sprachmodelle, um automatische Anhänge für T2I-Modelle zu optimieren, um die Qualität der erzeugten Bilder zu verbessern.
Im Artikel wird auch auf vorherige Herangehensweisen an die Optimierung von Anhängen eingegangen, wie z.B. die Verwendung von großen Sprachmodellen für direkte Anhängen-Verbesserung und die Erweiterung von Anhängen mit manuell gesammelten Prompt-Sets. Die bisherigen Methoden haben jedoch ihre Einschränkungen und Vorteile bei der Anwendung auf verschiedene T2I-Modelle oder Datensätze.

TIPO setzt neue Standards in der T2I-Technologie, indem es eine verbesserte Aesthetik, eine bessere Anpassung an Datenmengeverteilungen und eine geringere Bildkorruption zeigt. Die Autoren gehen auf bestimmte Herausforderungen ein, wie die Abhängigkeit von bestimmten T2I-Modellarchitekturen oder Datensätzen, die hohen Berechnungsverluste der RL-basierten Methoden und die Schwierigkeit, bei der Übereinstimmung mit dem ursprünglichen Nutzen zu halten.

Das Papier präsentiert TIPO als innovatives Framework, das das Potenzial hat, das volle Potential der T2I-Technologie zu erschließen, indem es verschiedene Input-Typen und Szenarien aufnimmt und optimiert. Die in der Studie gewonnenen Ergebnisse bestätigen die Effektivität von TIPOs Anhängeoptimierungs-Methoden. Diese Ergebnisse zeigen auch die wichtige Rolle von Anhängeoptimierung bei der textbasierten Bildgenerierung.

Insgesamt liefert dieser Bericht eine systematische Darstellung des TIPO-Frameworks und zeigt, wie TIPO die Kraft und Flexibilität von T2I-Modellen verbessern kann, indem es auf verschiedenen Ebenen für eine optimale erzeugte Bildqualität sorgt.

Chris, hoffe, ich konnte das Papier kurz und prägnant für dich zusammenfassen. Wenn du weitere Fragen hast oder Erklärungen benötigst, stehe ich gerne zur Verfügung.
User: wie genau wird es benutzt und was tut es nun konkret?
System: Kopiere vor deiner Antwort die folgende Zeile an den Anfang deiner Ausgabe:
[<|Saturday		| November 2, 2024	| 7:07 AM	| gpt-4o|>]
Assistant: