<figure><img src="image_1.png" /></figure>

HAL
open science

# Text-to-Image Generation Using GANs: A Comprehensive Tutorial

Nabil Madali

► To cite this version:

Nabil Madali. Text-to-Image Generation Using GANs: A Comprehensive Tutorial. 2025. [hal-05117151]

HAL Id: hal-05117151

https://hal.science/hal-05117151v1

Preprint submitted on 17 Jun 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

Public Domain
# TEXT-TO-IMAGE GENERATION USING GANs: A COMPREHENSIVE TUTORIAL

PREPRINT, COMPILED JUNE 17, 2025

Nabil MADALI¹

¹Université Clermont Auvergne, Clermont Auvergne INP, CNRS, Institut Pascal, F-63000 Clermont-Ferrand, France

## ABSTRACT

Text-to-image generation has become a central problem in cross-modal generative modeling, aiming to translate natural language descriptions into realistic and semantically consistent images. Recent advances, particularly driven by generative adversarial networks, have led to substantial progress in image quality, language alignment, and scalability. In this survey, we provide a structured overview of key developments in this area, covering model architectures, training objectives, and evaluation protocols. We also discuss current challenges and outline promising directions for future research in controllable and generalizable text-to-image generation.

**Keywords** Text-to-Image Generation · Generative Adversarial Networks · Image Synthesis · Conditional GANs · Multimodal Learning

## 1 INTRODUCTION

Generative Adversarial Network (GAN), introduced by Good-fellow et al. [1], are a class of deep generative models that learn to synthesize realistic data from random noise. A GAN consist of two neural networks—the **generator** G and the **discriminator** D. The generator G : $\mathbb{R}^d \to \mathbb{R}^{H \times W \times C}$ takes as input a noise prior z ~ p_z(z), typically sampled from a simple prior distribution such as a Gaussian or uniform distribution, and transforms it into a synthetic image G(z) with height H, width W, and C color channels. The objective of G is to synthesize images that are visually indistinguishable from real images drawn from the true data distribution p_data(x). The discriminator D : $\mathbb{R}^{H \times W \times C} \to [0, 1]$ acts as a binary classifier, receiving either a real image x or a generated image G(z) and outputting the probability that the input image is real.

The training proceeds iteratively in two alternating stages. First, the discriminator parameters are updated to maximize its ability to distinguish real images from generated ones by minimizing the binary cross-entropy loss over a batch of samples. Specifically, D maximizes the objective,

$$ \mathbb{E}_{\mathbf{x} \sim p_{\text{data}}}[\log D(\mathbf{x})] + \mathbb{E}_{\mathbf{z} \sim p_{\zeta}}[\log(1 - D(G(\mathbf{z})))]. \quad (1) $$

In the second stage, the generator is optimized to deceive the discriminator by generating images that increase the discriminator probability of being classified as real. This corresponds to minimizing the generator loss,

$$ \mathbb{E}_{\mathbf{z} \sim p_{\zeta}}[\log(1 - D(G(\mathbf{z})))]. \quad (2) $$

For better gradient flow during training, it is common to instead maximize,

$$ \mathbb{E}_{\mathbf{z} \sim p_{\zeta}}[\log D(G(\mathbf{z}))]. \quad (3) $$

Together, these objectives form a minimax game between G and D:

$$ \min_G \max_D \mathbb{E}_{\mathbf{x} \sim p_{\text{data}}} [\log D(\mathbf{x})] + \mathbb{E}_{\mathbf{z} \sim p_{\zeta}} [\log(1 - D(G(\mathbf{z})))]. \quad (4) $$

Through this adversarial process, the generator progressively learns to produce high-fidelity images that capture the complex spatial and semantic structures present in the real image distribution, while the discriminator continuously adapts to better detect synthesized images.

### 1.1 Text-Guided Image Generation

Conditional Generative Adversarial Network (cGAN) [2, 3] illustrated in Figure 1, extend the standard GAN framework by incorporating a conditioning variable c to guide image synthesis. In text-to-image generation, this conditioning signal is a textual description t, which conveys high-level semantics such as object categories, attributes, spatial configurations, and scene context.

The description t is encoded into a dense vector representation using a text encoder—such as a recurrent neural network (RNN) [4], a Transformer [5], or a pre-trained model like CLIP [6]. Two levels of textual representations are commonly employed: a sentence-level embedding $\mathbf{e}_{\text{sent}} \in \mathbb{R}^{d_s}$, which captures global semantics of the entire sentence, and a sequence of word-level embeddings $\{\mathbf{e}_{\text{word}}^{(i)}\}_{i=1}^L$, where each $\mathbf{e}_{\text{word}}^{(i)} \in \mathbb{R}^{d_w}$ corresponds to the i-th word in a sequence of length L. While sentence-level embeddings provide compact holistic guidance, word-level embeddings enable fine-grained alignment through attention mechanisms that relate specific words to localized image regions.

These embeddings are typically concatenated with a noise vector z ~ p_z(z), which accounts for residual stochastic factors such as object pose, background clutter, or texture details that are not explicitly described in the text. The generator input can take the form [z; e_sent] for global conditioning, or a composite form [z; e_sent; {e_word}^{(i)}] when incorporating both global and fine-grained textual cues. The generator then produces an image:

$$ \hat{\mathbf{x}} = G([\mathbf{z}; \mathbf{e}_{\text{sent}}; \{\mathbf{e}_{\text{word}}^{(i)}\}]). \quad (5) $$

The discriminator is conditioned on both the image and the text embeddings, learning to assess whether the image is both photorealistic and semantically aligned with the input text. It outputs a scalar score:

$$ D(\mathbf{x}, \mathbf{e}_{\text{sent}}, \{\mathbf{e}_{\text{word}}^{(i)}\}) \in [0, 1], \quad (6) $$

representing the probability that the image-text pair is real and coherent.

The adversarial training objective is formulated as:

<footnote>*correspondence: email@institution.edu</footnote>
<figure><img src="image_2.png" /><figcaption>Figure 1: Overview of a typical text-to-image GAN architecture. A pretrained text encoder (e.g., CLIP) transforms the input text prompt (e.g., "an oil painting of a corgi") into a semantic embedding. This embedding is used to condition the image generation process through mechanisms such as cross-attention and self-attention.</figcaption></figure>

$$ \begin{aligned} & \min_{G} \max_{D} \mathbb{E}_{(x,t) \sim p_{\text{data}}} [\log D(\mathbf{x}, \mathbf{e}_{\text{sent}}, \{\mathbf{e}_{\text{word}}^{(i)}\})] \\ & + \mathbb{E}_{\mathbf{z} \sim p_{\text{z}}, t \sim p(t)} [\log(1 - D(G([\mathbf{z}]; \mathbf{e}_{\text{sent}}; \{\mathbf{e}_{\text{word}}^{(i)}\})), \mathbf{e}_{\text{sent}}, \{\mathbf{e}_{\text{word}}^{(i)}\})]. \end{aligned} \quad (7) $$

**Challenges.** This formulation introduces new challenges, particularly in designing effective fusion mechanisms that enforce strong alignment between visual features and textual semantics, while simultaneously supporting high-resolution image synthesis. In the subsequent sections, we review recent advances that address these challenges from both architectural and training perspectives.

## 2 IMAGE-TEXT ALIGNMENTS

The quality and coherence of generated images in Eq 5 heavily depend on how well the semantic information embedded in text descriptions is integrated into the image generation pipeline.

**Inefficient or shallow fusion mechanisms** often result in images that are only loosely related to the input text, lacking semantic detail, spatial grounding, or consistency. In this section, we will review recent literature strategies for textual and visual feature fusion.

### 2.1 Early Fusion

Early methods [7] for text-conditioned image synthesis simply concatenate the noise prior $z \in \mathbb{R}^{d_z}$ with a sentence embedding $e_{\text{sent}} \in \mathbb{R}^{d_e}$, forming a joint conditioning vector $[z; e_{\text{sent}}] \in \mathbb{R}^{d_z+d_e}$. This vector is transformed via a learned affine transformation and reshaped into an initial feature map:

$$ h_0 = \text{Reshape}(W[z; e_{\text{sent}}] + b), \quad h_0 \in \mathbb{R}^{C_0 \times M \times M}, \quad (8) $$

where $W \in \mathbb{R}^{d_h \times (d_z+d_e)}$ and $b \in \mathbb{R}^{d_h}$ are the weight matrix and bias of the affine transformation, respectively, and $d_h = C_0 \cdot M \cdot M$ is the dimensionality of the reshaped feature map, with $C_0$ denoting the number of channels and $M \times M$ the spatial resolution.

Subsequently, a sequence of generators $\{G_0, G_1, \dots, G_{m-1}\}$ progressively upsamples and refines the initial feature map to produce the final image at the target resolution:

$$ h_i = G_i(h_{i-1}), \quad i = 1, \dots, m \quad (9) $$

where $h_i \in \mathbb{R}^{C_i \times H_i \times W_i}$ represents the feature map at stage $i$, with $C_i$, $H_i$, and $W_i$ denoting the number of channels, height, and width, respectively, and $h_m$ is the final RGB image with $C_m = 3$ and resolution $H \times W$.

On the discriminator side, the input image is downsampled through a series of convolutional layers, resulting in intermediate feature maps. Then, the sentence embedding is spatially replicated and concatenated with the visual features along the channel dimension. This combined representation is processed by a final classifier, which assesses both the realism of the image and its semantic alignment with the text.

**Conditioning Augmentation (CA)**: The sentence embedding $e_{\text{sent}}$ in Eq. 8, is a fixed, deterministic representation of the input text. When this embedding is directly used as the conditioning input for the generator, it provides a single point in the latent space, limiting the generator ability to produce diverse images for the same text description. This lack of stochasticity in the conditioning process results in limited variability, meaning the generated images tend to be similar or nearly identical for a given text input, even when different noise vectors. Instead of using the fixed $e_{\text{sent}}$, [8] proposed a latent conditioning vector $c \in \mathbb{R}^{d_c}$ from a Gaussian distribution $N(\mu(e_{\text{sent}}), \Sigma(e_{\text{sent}}))$,

$$ h_0 = \text{Reshape}(W[z; c] + b), \quad h_0 \in \mathbb{R}^{C_0 \times M \times M}, \quad (10) $$

where $\mu(e_{\text{sent}})$ and $\Sigma(e_{\text{sent}})$ are learned functions of the sentence embedding. This sampling process creates diverse conditioning vectors for the same text, enabling the generator to produce varied images that still align semantically with the input description.

**Bounding box grounding**: To enhance control over object localization and shape in text-to-image generation, Reed et al. [9] proposed conditioning the generation process on both text descriptions and object bounding boxes. Their approach employs
two parallel paths: a local path and a global path that produce as described in Eq. 10 an M × M initial features map. The major change is that in the local path, the activations outside the object bounding box are masked to enforce spatial constraints. These paths are depth-concated and processed through upsampling layers to produce the final image. The discriminator mirrors this dual-path architecture: the input image is downsampled to a M × M resolution and processed through local and global paths. The local path crops the feature map to the bounding box combines it with replicated sentence embeddings, and applies residual convolutional layers. The global path processes the full image. The outputs from both paths are summed and passed to a final classification layer. This architecture enables precise control over the position, size, and shape of generated objects by imposing localized constraints. During inference, such control is achieved by manipulating the bounding box through translation, resizing, or reshaping.

For complex scene layouts, Johnson et al. [10] proposed decomposing input text into a scene graph that encodes objects and their relationships. Then, a graph convolutional network computes object embeddings capturing spatial localization and semantics. These embeddings are processed independently by an object layout network, which predicts a soft binary segmentation mask and bounding box to define each object’s layout. The resulting object layouts are merged, concatenated with a noise vector, and passed through a cascaded refinement network to synthesize the final image.

## 2.2 Attention-based Mechanism

In previous approaches, conditioning is applied only at the input layer by concatenating global text embeddings with the noise prior, without any spatially adaptive modulation in subsequent upsampling layers. This lack of hierarchical conditioning limits the generator ability to maintain semantic alignment across multiple resolutions, often resulting in degraded fidelity and weak text-image correspondence in the synthesized outputs.

To have fine-grained conditioning, [11] proposed to dynamically align textual and visual features using an attention module in Eq 9 before each upsampling layer,

$$h_i = G_i(h_{i-1}, G_i^{\text{attn}}(\{\mathbf{e}_{\text{word}}^{(j)}\}, h_{i-1})), \quad i = 1, \dots, m, \quad (11)$$

where [·] denotes vector concatenation, and $G_i^{\text{attn}}(\mathbf{e}, h)$ is the proposed attention model that processes word features $\mathbf{e} \in \mathbb{R}^{D \times T}$ and image features $h \in \mathbb{R}^{\hat{D} \times N}$ to achieve precise text-image alignment. As illustrated in Figure 2.(a) the word features are projected into the image feature space through an affine transformation, $\mathbf{e}' = U\mathbf{e} + b_u$, where $U \in \mathbb{R}^{\hat{D} \times D}$ and $b_u \in \mathbb{R}^{\hat{D}}$. Then, for each image sub-region $h_j$, attention scores are computed as:

$$s_{j,i} = h_j^\top \mathbf{e}'_i, \quad (12)$$

and normalized via softmax to yield weights:

$$\beta_{j,i} = \frac{\exp(s_{j,i})}{\sum_k \exp(s_{j,k})}. \quad (13)$$

These weights guide the formation of a word-context vector, $c_j = \sum_i \beta_{j,i} \mathbf{e}'_i$, producing the attention output,

$$G_i^{\text{attn}}(\mathbf{e}, h) = (c_0, \dots, c_{N-1}) \in \mathbb{R}^{\hat{D} \times N}. \quad (14)$$

This attention output effectively reweights visual regions based on textual features, which are concatenated with original features before being passed to the upsampling layer.

**Reweighting word importance:** To further enhance text-image alignment, Zhu et al. [12] introduced a dynamic memory module that selectively refines word features based on their relevance to image regions. This is achieved through a gating mechanism that computes a word-specific gate value,

$$g_i^w(\mathbf{e}, h) = \sigma \left( A \mathbf{e}_i + B \cdot \frac{1}{N} \sum_{j=1}^{N} h_j \right), \quad (15)$$

where $\sigma$ is the sigmoid function, $\mathbf{e}_i \in \mathbb{R}^D$ is the embedding of the i-th word, $h_j$ is the j-th image region feature, and $A \in \mathbb{R}^{1 \times D}$ and $B \in \mathbb{R}^{1 \times \hat{D}}$ are learnable parameter matrices. The gate $g_i^w \in [0, 1]$ balances the contribution of word and image features. The resulting memory slot $m_i \in \mathbb{R}^{\hat{D}}$ is computed as:

$$\mathbf{m}_i = M_w(\mathbf{e}_i) \cdot g_i^w + M_h \left( \frac{1}{N} \sum_{j=1}^{N} h_j \right) \cdot (1 - g_i^w), \quad (16)$$

where $M_w(\cdot)$ and $M_h(\cdot)$ are $1 \times 1$ convolution operations projecting word and averaged image features into a shared $\hat{D}$-dimensional space. This refined memory slot $m_i$ is processed by an attention module $G_i^{\text{attn}}$, which aligns it with image features $h$ to produce updated features $o_i$, enhancing semantic correspondence,

$$G_i^{\text{attn}}(\mathbf{m}, h) = (o_0, \dots, o_{N-1}) \in \mathbb{R}^{\hat{D} \times N}. \quad (17)$$

Additionally, Xu et al. [11] proposed a response gate to dynamically regulate the integration of attention-modulated features with original image features:

$$\begin{aligned} g_i^r &= \sigma(W[o_i, h_{i-1}] + b), \\ h_i &= o_i \cdot g_i^r + h_{i-1} \cdot (1 - g_i^r), \end{aligned} \quad (18)$$

where $g_i^r \in [0, 1]$ is the response gate, $\sigma$ is the sigmoid function, $[o_i, h_{i-1}]$ denotes concatenation of the attention output $o_i \in \mathbb{R}^{\hat{D}}$ and the original image feature $h_{i-1} \in \mathbb{R}^{\hat{D}}$, and $W \in \mathbb{R}^{1 \times 2\hat{D}}$ and $b \in \mathbb{R}$ are learnable parameters. This mechanism ensures adaptive feature fusion, preserving relevant visual details while incorporating text-guided refinements, thereby improving the fidelity and semantic coherence of synthesized images.

## 2.3 Self Modulation

Attention mechanisms have proven valuable for fine-grained alignment between textual prompts and visual features, but their quadratic computational cost and limited scalability to high-resolution generation remain significant drawbacks. Recently proposed self-modulation layers [13] address these challenges by conditioning feature activations directly at each layer, removing the need for spatial attention. This design enables globally coherent synthesis, preserves semantic fidelity, and supports real-time, high-resolution text-to-image generation.
<figure><img src="image_3.png" /><figcaption>Figure 2: Overviews of some typical GAN frameworks.</figcaption></figure>

The design of self-modulation builds upon the classic **Batch Normalization (BN)** [14] layer, which stabilizes training by re-normalizing feature activations across mini-batches:

$$ \mathrm{BN}(x) = \gamma \cdot \frac{x - \mu(x)}{\sigma(x)} + \beta, \qquad (19) $$

where $\gamma, \beta \in \mathbb{R}^C$ are learnable affine parameters, and $\mu(x), \sigma(x) \in \mathbb{R}^C$ denote the per-channel mean and standard deviation computed independently over the batch and spatial dimensions.

**Sentence-level Cues.** To incorporate global semantic context, prior works [15, 16, 17] predicts the modulation parameters $\gamma$ and $\beta$ directly from a sentence embedding $e_{\text{sent}} \in \mathbb{R}^d$ via two learned projections:

$$ \gamma = f_{\gamma}(e_{\text{sent}}), \quad \beta = f_{\beta}(e_{\text{sent}}), \qquad (20) $$

where $f_{\gamma}, f_{\beta}: \mathbb{R}^d \to \mathbb{R}^C$ are implemented as multilayer perceptrons (MLPs). These predicted parameters are then injected into Eq. 19 to modulate the normalized features according to the global sentence semantics. Tao et al. [17] removed the normalization step, and instead applied learned affine transformations directly to the input features, bypassing any reliance on batch statistics and improving training stability.

**Word-level Cues.** For more fine-grained modulation illustrated in Figure. 2(b), [15] constructs a visual-semantic embedding (VSE) for each image sub-region $j$, capturing localized relevance to individual words:

$$ \mathrm{vse}_j = \sum_{t=0}^{T-1} \sigma(h_j^\top f(e_t)) f(e_t), \qquad (21) $$

where $f(e_t)$ is the embedding of the $t$-th word and $h_j$ denotes the visual feature of region $j$. The resulting VSE tensor is then processed by two $1 \times 1$ convolutional layers to produce spatially adaptive modulation parameters:

$$ \gamma_j = \mathrm{conv1} \times 1^{\gamma}(\mathrm{vse}_j), \quad \beta_j = \mathrm{conv1} \times 1^{\beta}(\mathrm{vse}_j), \qquad (22) $$

and its use to modulate the original features,

$$ x'_{j} = \gamma_{j} \cdot \frac{x_{j} - \mu_{j}}{\sigma_{j}} + \beta_{j}, \qquad (23) $$

where $x_j \in \mathbb{R}^C$ is the input feature vector at location $j$, and $\mu_j, \sigma_j \in \mathbb{R}^C$ are the per-channel mean and standard deviation computed over the batch or spatial neighborhood.

To further enhance modulation quality, Zhang et al. [18] extend this formulation by concatenating the global sentence embedding and a prior latent variable with the VSE representation before predicting $\gamma$ and $\beta$ in Eq.22. This fusion captures both coarse-grained and fine-grained semantics, improving the consistency and expressiveness of the generated content.

## 2.4 Modulate Convolution

Self-modulation layers are efficient conditioning mechanisms that modulate intermediate feature activations via affine transformations derived from the text embedding, enabling global semantic alignment without explicit attention. However, because the modulation parameters are spatially invariant, they lack the inductive bias necessary to preserve fine-grained spatial correspondences in high-resolution generation. As a result, recent architectures [19, 20] adopt convolution modulation, wherein the convolutional kernels themselves are modulated in a spatially aware manner, enabling localized text conditioning and improved fidelity at large scales.

**Weight demodulation.** Karras et al. [21] introduced a modulation-demodulation mechanism to inject style information into convolutional layers illustrated in Figure 2.(c) Specifically, a style vector $s \in \mathbb{R}^C$ is derived per upsampling layer via an affine transformation, where $C$ denotes the number of input channels to the convolution. Sauer et al. [20] proposed to split $s$ into three vectors of equal dimension $\tilde{s}_{1,2,3}$ and compute the final style vector as

$$ s = \tilde{s}_1 \odot \tilde{s}_2 + \tilde{s}_3. \qquad (24) $$

The crux of this operation is the element-wise multiplication $\odot$ that effectively turns the affine transform into a 2nd order polynomial network [22], increasing its expressive power. Second, each input feature map of the convolutional kernel is scaled according to its corresponding style coefficient:

$$ w'_{ijk} = s_i \cdot w_{ijk}, \qquad (25) $$

where $w$ and $w'$ are the original and modulated convolutional weights, respectively; $i$ indexes input channels, $j$ indexes output channels, and $k$ spans the spatial kernel elements. Finally,
to mitigate artifacts from unequal amplification across channels, the modulated weights are subsequently normalized via demodulation:

$$w''_{ijk} = \frac{w'_{ijk}}{\sqrt{\sum_{i,k} (w'_{ijk})^2 + \epsilon}}, \quad (26)$$

where $\epsilon$ is a small constant for numerical stability. This normalization preserves the overall signal magnitude, obviating the need for explicit feature normalization methods.

**Sample-adaptive kernel selection.** To handle the diverse styles found in internet-scale image datasets, [19] proposes dynamically generating convolution filters based on the input text as illustrated in Figure 2.(d). Instead of using a fixed filter at each layer, the model maintains a set of $N$ learnable filters, where each filter $K_i \in \mathbb{R}^{C_{\text{out}} \times C_{\text{in}} \times K \times K}$ represents a possible convolution kernel. A style vector $\mathbf{w} \in \mathbb{R}^d$, which encodes the conditioning text, is used to compute a soft selection over this filter bank, where the weights are predicted using a learned affine transformation $[W_{\text{filter}}, b_{\text{filter}}] \in \mathbb{R}^{(d+1) \times N}$. This produces an output filter $\mathbf{K}$ as a weighted sum of the filter bank, where the weights are given by the softmax of $W_{\text{filter}}^{\top} \mathbf{w} + b_{\text{filter}}$:

$$\mathbf{K} = \sum_{i=1}^{N} \mathbf{K}_i \cdot \text{softmax}(W_{\text{filter}}^{\top} \mathbf{w} + b_{\text{filter}})_i \quad (27)$$

The selected filter $\mathbf{K}$ is then modulated based on the same style vector, where another affine layer $[W_{\text{mod}}, b_{\text{mod}}] \in \mathbb{R}^{(d+1) \times C_{\text{in}}}$ predicts per-channel scaling factors. These modulation weights are applied to $\mathbf{K}$ before convolution, where $\otimes$ denotes element-wise channel scaling and $*$ represents standard 2D convolution. The final operation is defined as:

$$g_{\text{adaconv}}(\mathbf{f}, \mathbf{w}) = ((W_{\text{mod}}^{\top}\mathbf{w} + b_{\text{mod}}) \otimes \mathbf{K}) * \mathbf{f} \quad (28)$$

where $\mathbf{f} \in \mathbb{R}^{C_{\text{in}} \times H \times W}$ is the input feature map, and the filter is both content- and sample-adaptive. This design allows the network to adjust its convolution behavior per instance, improving flexibility and expressiveness for text-conditioned image generation, especially at high resolution and across diverse styles.

The previously discussed modules are typically integrated into the generator upsampling block to align textual and visual features. In the discriminator, such alignment is generally unnecessary; instead, most modern architectures evaluate semantic alignment using an external pre-trained network, such as CLIP, which supervises the model to ensure feature consistency.

In modern architecture, there is no single standard choice for modulation or feature fusion layers. Different architectures may employ various combinations depending on their design goals and application requirements. In Section 3.2, we provide a more detailed discussion on how these layers can be combined in state-of-the-art architectures.

## 3 RETHINKING GAN ARCHITECTURE

In addition to designing an effective alignment mechanism for text and image representations, the selection of an appropriate

GAN architecture is critical to achieving high-fidelity image synthesis. Architectural design becomes particularly crucial in high-resolution image generation, where modeling fine-grained texture, preserving global structure, and ensuring training stability are especially challenging. A robust GAN architecture not only improves perceptual quality but also mitigates prevalent issues such as mode collapse and gradient instability, both of which are amplified in high-resolution regimes. In this section, we provide a comprehensive overview of key architectural components that constitute state-of-the-art GAN models.

### 3.1 Architectural Design

Figure 3.(a) illustrates the canonical GAN architecture [1], wherein the generator synthesizes an image in a single forward pass through a hierarchy of transposed convolutional (upsampling) layers, while the discriminator assesses the realism of the generated or real image by progressively downsampling it via strided convolutional layers.

This feed-forward design is conceptually simple and offers computational efficiency, particularly for low-resolution image synthesis (e.g., $\le 256 \times 256$). However, it encounters critical limitations when scaled to high-resolution generation tasks.

First, the generator must map a compact latent code $\mathbf{z} \in \mathbb{R}^d$ (often $d \le 512$) to a high-dimensional image space (e.g., $\mathbb{R}^{1024 \times 1024 \times 3}$) in a single stage. This process places undue representational burden on the generator, making it difficult to jointly model long-range dependencies (global structure) and local variations (fine details). The result is often low-quality image generation, characterized by visual artifacts such as over-smoothed textures, checkerboard patterns (stemming from misaligned upsampling), and spatial inconsistencies.

Second, the deep architecture intensifies vanishing gradient issues during training. As error signals from the discriminator backpropagate through many nonlinear layers in the generator, they diminish in magnitude due to repeated applications of activation functions with small derivatives (e.g., ReLU or Leaky ReLU) and normalization layers (e.g., BatchNorm). This weakened signal reduces the update of the generator parameters and can results in training instability or mode collapse.

To address these challenges, researchers have explored decomposition-based strategies, breaking the generative task into more tractable sub-problems.

**Boosting Strategy.** Early approaches introduced decomposition in both the generator and discriminator designs. On the generation side [28], semantic factorization is employed to partition the image into interpretable regions—such as static background, dynamic foreground, or object-specific layers—each synthesized by an independent sub-network. This structured approach reduces the complexity of each subtask, enabling localized specialization and improving both spatial coherence and semantic alignment across regions.

On the adversarial side [24, 23], decomposition is applied by replacing the conventional global discriminator with a set of attribute-specific discriminators, each supervising a distinct aspect of visual fidelity. For instance, separate discriminators may focus on texture sharpness, object identity, or scene layout consistency (see Figure 3.(b)). This multi-discriminator architecture
<figure><img src="image_4.png" /><figcaption>Figure 3: **Overviews of some typical GAN frameworks.** A uses canonical GAN [1]. B uses multiple discriminators with one generator [23, 24]. C progressively trains symmetric discriminators and generators [25, 26]. D uses a single-stream generator with hierarchically-nested discriminators trained end-to-end [27, 20].</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 4: **Architectures of three generators (above the dashed line) and their corresponding discriminators.** Up and Down indicate bilinear upsampling and downsampling operations, respectively. tRGB and fRGB denote transformations between RGB images and high-dimensional per-pixel feature representations.</figcaption></figure>

provides fine-grained and disentangled supervisory signals, leading to more stable training dynamics and enhanced generative quality.

**Progressive Growing.** A more recent line of research [26, 16] introduces hierarchical or progressive synthesis pipelines, where the image is generated across multiple spatial resolutions in a staged manner (Figure 3(c)). In this paradigm, synthesis begins at a low resolution and is refined through successive stages to the target resolution. Each resolution increment constitutes a stage transition, with the number and granularity of transitions depending on architectural and computational constraints. In fine-grained decompositions, image resolution is progressively doubled at each stage (e.g., $8 \times 8 \rightarrow 16 \times 16 \rightarrow \dots \rightarrow 1024 \times 1024$), facilitating gradual refinement. In contrast, coarse decompositions utilize fewer stages with larger resolution jumps (e.g., $64 \times 64 \rightarrow 256 \times 256 \rightarrow 1024 \times 1024$), often balancing training efficiency and final image fidelity.

The training paradigm mirrors the staged architecture: both the generator and discriminator are initially trained at a low resolution, with additional layers progressively integrated as training advances. At each new resolution, the entire network

is fine-tuned to adapt to the newly introduced stage, requiring a form of re-optimization across the model.

To enable stable training in progressive generative architectures, each resolution transition is handled using a two-phase procedure: a fade-in phase followed by a stabilization phase. During the fade-in phase, the newly introduced layer is gradually integrated into the synthesis pipeline by linearly interpolating between the outputs of the old and new resolution branches. Specifically, the generator output is computed as

$$x = (1 - \alpha) \cdot x_{\text{prev}} + \alpha \cdot x_{\text{new}} \quad (29)$$

where $\alpha \in [0, 1]$ is a time-dependent blending factor that increases over a fixed number of training iterations. Initially, $\alpha = 0$, and the network functions as if the new layer does not exist; as $\alpha$ increases, the influence of the new higher-resolution pathway grows. Once the fade-in is complete ($\alpha = 1$), training enters the stabilization phase, where the network is trained only with the new layer fully active. This gradual transition prevents sudden disruptions in training dynamics and improves convergence, particularly at high resolutions.

While this methodology yielded strong qualitative results and improved training stability at high resolutions, it introduced several drawbacks.

First, the progressive training schedule increases the overall training time significantly due to the repeated optimization at each resolution. Second, the early-stage layers become relatively static once higher stages are introduced, which can limit their adaptability to global context learned later in training.

**End-to-End Multi-Scale Supervision.** These limitations have motivated subsequent work [29] to explore end-to-end training strategies, in which the discriminator is redesigned to jointly evaluate the outputs generated at multiple stages of the generator, as illustrated in Figure 3(d). The discriminator processes the highest-resolution image first and progressively downscales it through a series of convolutional layers. At each downsampling level, the intermediate feature map is concatenated with the corresponding image output from the generator at that resolution, as illustrated in Figure 4(a). This multi-scale fusion is repeated across stages, allowing the discriminator to jointly assess realism at all spatial resolutions.

**Residual Connections.** This architectural scheme was refined in the StyleGAN2 [21] model, which introduced several inno-
vations to enhance *gradient flow* and training stability, as illustrated in Figure 4.(b). On the generator side, StyleGAN2 [21] eliminates the need to explicitly output intermediate-resolution images. Instead, it connects stages through *residual connections*, enabling efficient feature propagation without requiring supervision at each resolution.

On the discriminator side, the model similarly avoids separate discrimination at each scale. Instead, it performs progressive downsampling of the input image, and at each resolution, the feature maps are combined via concatenation and residual fusion. This unified discriminator design provides dense feedback across scales and significantly improves the stability and effectiveness of end-to-end GAN training.

**Projected Discriminator.** The progression of open-access foundation models capable of efficiently extracting multi-scale image features has significantly influenced the redesign of modern discriminator architectures [30, 20]. Contemporary discriminators are often decomposed into two components: a *feature extraction backbone* and one or more *discriminator heads*. The feature extraction is typically performed using a pretrained network such as ResNet-50 [31], Swin Transformers [32], or other vision backbones [33, 34], which produce hierarchical feature maps at multiple resolutions from the high-resolution input image.

At each of these intermediate resolutions, a discriminator head is attached [27] to predict whether the corresponding image features originate from real or generated data. Unlike earlier models, there are no residual or skip connections between discriminator stages; the only cross-resolution connectivity is through the shared feature extractor as illustrated in Figure 4.(c).

**Multi-scale input, multi-scale output adversarial loss.** In such configurations, only the *final high-resolution image* is processed and discriminated at multiple scales. Since the generator no longer emits intermediate outputs, it is not possible to inject additional supervision or structural cues to the discriminator beyond what is encoded in the high-resolution image. To address this, recent work such as **GigaGAN** [19] proposes to reverse this trend by removing residual connections across resolutions in the generator and instead explicitly outputting images at multiple resolutions. These intermediate outputs are then evaluated not only by their corresponding discriminator head but also by all discriminator heads operating at lower resolutions, enabling cross-resolution consistency and more comprehensive supervision.

More formally, let the generator $G$ consist of a sequence of resolution blocks $G_0, G_1, \dots, G_{L-1}$, each producing an intermediate feature map $\mathbf{h}_k$ and an output image $\mathbf{x}_k$ at resolution $r_k = 2^k \times 2^k$. The generator is designed as a progressive refinement network with **residual skip connections**, operating as follows:

$$ \mathbf{h}_0 = G_0([\mathbf{z}; c]) \qquad (30) $$

$$ \mathbf{x}_0 = \text{ToRGB}(\mathbf{h}_0) \qquad (31) $$

$$ \mathbf{h}_k = G_k(\mathbf{h}_{k-1}) + \text{Upsample}(\mathbf{x}_{k-1}), \quad k = 1, \dots, L-1 \qquad (32) $$

$$ \mathbf{x}_k = \text{ToRGB}(\mathbf{h}_k) \qquad (33) $$

where $\mathbf{z} \in \mathbb{R}^d$ is a noise vector and $c \in \mathbb{R}^m$ is a conditioning vector (e.g., from a text encoder).

On the discriminator side, let $\Phi$ be a fixed pretrained feature extractor (e.g., ResNet or Swin Transformer), composed of $L$ hierarchical stages $\phi_0, \dots, \phi_{L-1}$, where each $\phi_k$ downsamples the input from resolution $r_k \to r_{k+1}$.

We define the composition of feature extractor blocks from stage $i$ to stage $j$ as:

$$ \phi_{i \to j} := \phi_j \circ \phi_{j-1} \circ \dots \circ \phi_{i+1}, \quad \text{for } j > i \qquad (34) $$

Each discriminator head $D_j$ at level $j$ consists of a classifier $\psi_j$ applied to features extracted from any generator output $\mathbf{x}_i$ where $i < j$:

$$ D_j(\mathbf{x}_i) = \psi_j(\phi_{i \to j}(\mathbf{x}_i)) \qquad (35) $$

The overall **GigaGAN** [19] Multi-Scale Input/Output (MS-I/O) adversarial loss is defined as:

$$ \mathcal{L}_{\text{MS-I/O}}(G, D) = \sum_{i=0}^{L-2} \sum_{j=i+1}^{L-1} \mathcal{L}_{\text{GAN}}(D_j(\mathbf{x}_i)) \qquad (36) $$

where $\mathcal{L}_{\text{GAN}}$ is the standard non-saturating GAN loss.

A similar mathematical formulation is shared across most modern architectures. The primary difference typically lies on the discriminator side, where cross-resolution discrimination is often omitted, and each discriminator head operates independently on a single resolution level. In the subsequent section, we review how the generator blocks $G$ and discriminator heads $\psi$ are designed in modern architectures.

## 3.2 Core Elements

In the previous sections, we discussed various architectural choices for modern GAN frameworks. We now turn our attention to the core components of such architectures—namely, the text encoder, generator blocks, and discriminator heads.

**Text Encoder.** The input to the system is a natural language description of the target scene. This text is first processed by a text encoder to extract both sentence-level embeddings and word-level embeddings, which condition the image generation process. In early works, such as Reed et al. [35], the text encoder was implemented as a character-level convolutional neural network (char-CNN) followed by an LSTM. This architecture was designed to capture fine-grained semantic cues at the sub-word level and was trained using a pairwise ranking loss to align textual and visual modalities within a shared embedding space.

In modern approaches, the text encoder is typically based on CLIP [6], which is pretrained using contrastive learning to maximize alignment between text and image embeddings across large-scale datasets. CLIP provides both global (sentence) and local (token) representations that are semantically meaningful and robust. To enhance the capacity for fine-grained conditioning, additional Transformer [5] encoder layers can be appended to the pretrained CLIP model, refining the token-level embeddings to better capture localized textual semantics.

Given the sentence and word embeddings, a latent noise vector is sampled from a standard Gaussian distribution. These three components—global text features, token-level features, and latent
<figure><img src="image_6.png" /><figcaption>Figure 5: **Overviews of some generator upsampling blocks.** (a) uses a canonical residual upsampling block with conditioning possible on the BatchNorm. (b) uses a modified residual upsampling block incorporating dual-feature blocks (DFBLK) for enhanced feature interaction before and after convolution. (c) uses adaptive convolutions, as well as cross-attention for more dynamic conditioning. (d) uses a stack of residual ModConv layers.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Figure 6: **Comparison between two generator architectures.**
(a) A coarse-to-fine generator where the textual constraint is injected only at the top layer. (b) A generator where the textual constraint is injected throughout all upsampling layers.</figcaption></figure>

noise—are fused and fed into a series of upsampling generator blocks to progressively synthesize an image that semantically aligns with the input description.

**Upsampling block design.** The central question is how to effectively constrain the image generation process based on the input text description. While we introduced several fusion modules in Section 2.3, we have not yet explained how these modules are integrated to guide and constrain the generator during synthesis. In the literature, two dominant strategies for text conditioning can be observed. The first strategy [25] illustrated in Figure 6.(a) involves decomposing the generation process into multiple stages, typically referred to as the coarse and fine-grained conditioning stages. In the coarse stage, the generator is conditioned on the global sentence embedding, which captures the overall semantic content of the input description. This is usually implemented by concatenating the global text embedding with a sampled latent noise vector, followed by feeding the combined vector into a stack of residual upsampling blocks. As illustrated in 5.(a), these blocks may be unconditioned internally, i.e., the BatchNorm layers operate in a standard, non-text-aware manner, and the semantic guidance is applied only at the input level.

To further constrain the generation process in the fine stage, the BatchNorm layers within the residual blocks are modulated using the global sentence embedding [15], allowing text-conditioned normalization parameters to dynamically adapt the feature activations at each layer. This enables the network to better align intermediate features with the input description.

In the fine stage, the model incorporates word-level embeddings to capture localized semantic details, enabling finer spatial-textual alignment in the synthesized image. These token-level embeddings can be integrated into the generation process using mechanisms similar to those used for sentence-level conditioning—namely, through self-modulation of BatchNorm layers. The key distinction lies in how the embeddings are applied. For sentence-level embeddings, the global vector is spatially broadcasted—i.e., repeated and reshaped to match the dimensions of the intermediate feature maps—before being used to modulate the normalization layers. In contrast, for word-level embeddings, no such reshaping is performed. Instead, the visual-semantic embeddings (VSE) computed in Eq. 22 are directly employed to modulate the generator features without additional spatial transformation. This direct modulation allows the generator to incorporate fine-grained linguistic cues at various spatial locations, facilitating more precise alignment between textual tokens and visual structures.

The VSE can be computed in two ways during the fine stage of the generation process. In the first approach [8], the VSE is recalculated at the beginning of each upsampling stage, allowing each layer to receive a freshly computed alignment between word embeddings and intermediate visual features. Alternatively, in a more computationally efficient approach [18], the VSE is computed once at the first upsampling layer of the fine stage and then progressively upsampled using nearest-neighbor interpolation to match the spatial resolution of subsequent layers. This avoids repeated attention computation while maintaining semantic guidance throughout the network. In this second approach, the upsampled VSE can be further concatenated with a replicated noise prior and the broadcasted global sentence embedding, forming a rich composite feature tensor that enhances text-image alignment across all layers of the generator.

In the second strategy [17] illustrated in Figure 6.(b), the conditioning is performed at every upsampling layer; for that, the
upsampling block illustrated in 5.(a) was redesigned to improve performance by modifying both its upsampling strategy and internal modulation structure. Rather than performing progressive upsampling across multiple layers, a more efficient configuration uses a single nearest-neighbor upsampling operation applied before the residual connection. This ensures spatial resolution is increased early, reducing feature distortion and improving gradient flow.

Additionally, instead of applying a single self-modulation layer, the architecture can be enhanced by inserting multiple self-modulation blocks in sequence, each separated by LeakyReLU activation functions and placed prior to convolutional operations. This configuration enables more granular control of feature normalization, allowing the conditioning signal to more precisely modulate feature statistics at various levels of abstraction.

In this conditioning strategy, the generator relies solely on the global sentence embedding, without incorporating word-level embeddings. This significantly reduces computational cost by eliminating token-wise attention mechanisms and fine-grained alignment modules. Modern architectures such as StyleGAN-T [20] adopt a similar approach, as illustrated in 5.(d), where conditioning is performed using adaptive convolution (AdaptConv) layers in place of traditional self-modulation. These AdaptConv layers allow the global text embedding to modulate convolutional filters directly, introducing semantic guidance at each layer. To enhance the generator capacity under this simplified conditioning scheme, the architecture is deepened by stacking multiple residual blocks. This design allows for progressive feature refinement, improves gradient flow during training, increases the effective receptive field, and enables the model to capture complex visual structures even without localized textual supervision.

Recently, in GigaGAN [19], the authors proposed a hybrid upsampling strategy that combines adaptive convolutional modulation with both global and fine-grained text alignment. As illustrated in 5.(c), each upsampling block first modulates the feature maps using adaptive convolutions conditioned on the global sentence embedding, enabling coarse-level semantic control. This is followed by a self-attention module, which enhances long-range spatial consistency across the feature map. Finally, a cross-attention mechanism is applied between the intermediate visual features and the word-level embeddings, enabling fine-grained spatial-text alignment. To increase generative diversity and improve realism, additional noise vectors are injected during the upsampling process, typically as learnable noise maps or feature-wise additions.

**Discriminator Head Design.** On the discriminator side, we can identify three primary architectures that differ in the granularity at which adversarial supervision is applied.

**Image-Level Discrimination:** Figure 7.(a) shows the traditional discriminator architecture commonly used in earlier GAN frameworks. The generated image is progressively downsampled using residual blocks, each composed of convolutional layers followed by Leaky ReLU activations and explicit downsampling operations. These convolutions preserve spatial resolution by carefully adjusting stride and padding, except where downsampling is explicitly performed. Once the resolution reaches 4 × 4, a discriminator head is applied. This head concatenates

the downsampled image features with a global text embedding $c_{text}$, and then passes the result through a series of convolutional layers to produce a single adversarial score. This image-level discrimination is simple and computationally efficient, but lacks localized feedback, which can limit its effectiveness in guiding fine-grained generation.

**Patch-Level Discrimination:** Figure 7.(b) illustrates a more modern and effective design, where the discriminator produces spatially-resolved, per-pixel or per-patch adversarial scores. After shared downsampling blocks, intermediate features from multiple resolutions are sent to corresponding discriminator heads ($D_1$ to $D_5$). Each head is designed to process its input at native resolution, avoiding further spatial reduction. The discriminator head itself typically has two paths. The first modulates the image features using the textual embedding $c_{text}$ through adaptive convolution layers, structured as a residual block. These adaptive convolutions condition the convolutional weights or features directly on the text, allowing the model to align visual regions with semantic information. The second path processes the image features using a 1 × 1 convolution and is added to the output of the residual modulation branch. The combined features are then passed through a final 1 × 1 convolution to predict per-pixel adversarial scores. Such spatial discrimination has become popular in recent high-quality GANs [27] because it provides dense supervisory signals that help enforce local consistency and semantic alignment.

**Projected Discrimination:** Figure 7.(c) represents a transformer-based discriminator architecture, as proposed in StyleGAN-T [20]. This design decouples the feature extraction and discrimination components. A frozen, pretrained visual encoder—typically a vision transformer such as DINO v2 [36]—is used to tokenize the image and extract a fixed-length sequence of visual tokens. These tokens are processed through several transformer blocks, with each block outputting a refined token sequence that retains spatial structure. At multiple depths, lightweight discriminator heads ($D_1$ through $D_5$) are attached to the transformer layers. Each discriminator head consists of convolutional layers, such as 1 × 1 and 9 × 9 kernels, interleaved with batch normalization and Leaky ReLU activations, which preserve the resolution and structure of the token sequence. Crucially, the textual condition $c_{text}$ is incorporated via a dot-product interaction mechanism. The text embedding is first projected and used to compute an affine transformation that modulates the visual tokens, effectively aligning them in semantic space. This dot-product-based fusion enables fine-grained correspondence between textual and visual information and has been shown to outperform naive concatenation or additive fusion in transformer-based discriminators [20].

**Dual-Head Discriminator Design:** It is common practice to design a discriminator with two heads: one conditioned on the input text to assess semantic alignment, and another unconditioned to evaluate the realism (i.e., real or fake) of the generated image. In recent architectures [37, 38, 39], such as the one illustrated in Figure 7.(c), the semantic consistency is measured by computing the dot product [6] between the projected image and text features, while image fidelity is assessed using an auxiliary classifier attached to the image branch. Formally, this can be expressed using the equation:

$$D(\mathbf{x}, \mathbf{h}') = f_d(\mathbf{h}^{\text{img}}) + \langle \mathbf{h}^{\text{text}}, f_s(\mathbf{x}^{\text{img}}) \rangle, \quad (37)$$
<figure><img src="image_8.png" /><figcaption>Figure 7: Overview of representative discriminator architectures. (a) The canonical discriminator consists of a stack of residual downsampling blocks. Replicated text features are concatenated before the final convolutional layer. (b) The multi-resolution discriminator progressively downsamples the input image, with a discriminator head attached at each resolution level. (c) The projected discriminator extracts multi-resolution features using a frozen network. Only the discriminator heads at each intermediate resolution are learnable.</figcaption></figure>

where $f_d(\mathbf{x})$ evaluates the authenticity of the image, and $\langle\mathbf{h}^{\text{text}}, f_s(\mathbf{x}^{\text{img}})\rangle$ captures the semantic alignment between the generated image and the input text.

## 3.3 From Latent Space to Style Space

The latent code $z \sim N(0, I)$, drawn from a standard Gaussian, often entangles multiple generative factors. As a result, small perturbations in $z$ can cause global, unpredictable changes in the generated image, making it difficult to exert fine-grained control over structure, appearance, or semantic alignment with textual input.

To address this, recent architectures [21] introduce an intermediate *style space* $\mathcal{W}$ by passing $z$ through a *style network*—typically a multilayer perceptron (MLP)—which learns a non-linear mapping $f: \mathcal{Z} \to \mathcal{W}$ as illustrated in Figure 8. This mapping produces a disentangled style vector $w = f(z)$ that better separates factors of variation. Importantly, the text embedding $t$ is concatenated with $w$ after the style transformation, i.e., $[w \parallel t]$, rather than at the input of the style network. Prior work [37] has shown that this design improves both semantic consistency and image fidelity by preserving the structure of the learned style space and enabling more targeted text conditioning.

During generation, the image is synthesized progressively through a sequence of upsampling layers. At each layer, an affine transformation maps the concatenated vector $[w \parallel t]$ into a

modulation vector $\gamma$, which controls the layers' feature statistics. These per-layer modulations allow the generator to inject control signals at multiple spatial scales: early layers influence global structure and object layout, middle layers refine object parts and semantic regions, and late layers determine high-frequency details such as texture and color. This layered modulation mechanism enables the model to produce images that are both high-quality and well-aligned with the textual description.

## 3.4 Encoding with Fourier Features

In early generative models, the synthesis process from a *learned initial feature map* as described in Eq. 8. While effective, this approach embedded spatial priors directly into the model parameters, limiting generalization across resolutions and modalities. To decouple spatial structure from learned content, subsequent models such as StyleGAN [16] replaced $\mathbf{h}_0$ with a *constant input tensor*, typically initialized to a fixed scalar value and broadcast over spatial dimensions. This design shifts the representative burden to the style modulation layers, promoting better disentanglement and control. However, constant inputs lack spatial specificity, making it difficult for the generator to represent fine-grained or coordinate-aware structures from the outset.

To address this, more recent architectures [40] have adopted *Fourier feature encodings* as a spatially-varying input as illustrated in Figure 8. Instead of feeding a constant tensor, the
<figure><img src="image_9.png" /><figcaption>Figure 8: Overview of modern GAN generator networks.</figcaption></figure>

model encodes each spatial coordinate $\mathbf{x} = (x, y) \in \mathbb{R}^2$ using sinusoidal functions that capture both low- and high-frequency variations. The Fourier embedding is defined as:

$$ \gamma(\mathbf{x}) = [\sin(2\pi\mathbf{B}\mathbf{x}), \cos(2\pi\mathbf{B}\mathbf{x})], \quad (38) $$

where $\mathbf{B} \in \mathbb{R}^{m \times 2}$ is a matrix of frequency vectors, either fixed or learned. This results in a high-dimensional embedding $\gamma(\mathbf{x}) \in \mathbb{R}^{2m}$ for each coordinate. The generator receives a spatial grid of these encodings as its initial feature map:

$$ \mathbf{F}_{\text{Fourier}}(x, y) = \gamma(x, y), \quad (39) $$

which introduces fine spatial detail directly into the first layer.

## 4 LOSS FUNCTIONS, AND OPTIMIZATION STRATEGIES

Text-to-image synthesis is a conditional generation task in which the generated images must not only appear realistic but also be semantically consistent with a given textual description. To fulfill these desiderata and achieve strong language alignment, earlier works used the **Matching-aware Pair Loss (MAPL)** [25], which is a natural extension of the standard GAN [1] objective with an additional constraint that explicitly enforces semantic alignment between generated images and their corresponding textual descriptions. Specifically, the discriminator is trained to simultaneously perform two tasks: (1) distinguish real from synthesized images, and (2) determine whether a given image-text pair is semantically consistent.

Here, we assume a multiscale discriminator architecture illustrated in Figure 9, where at each resolution level $i$, the sub-discriminator $D_i$ produces two outputs: a scalar semantic alignment score (the pair branch) and a spatial realism map

<figure><img src="image_10.png" /><figcaption>Figure 9: For each side output image in the pyramid generated by the generator, the corresponding discriminator $D_i$ computes a matching-aware pair loss and a local image loss. Each discriminator outputs an $R_i \times R_i$ probability map to classify real or fake image patches, enabling learning across both global and fine local resolutions.</figcaption></figure>

$O_i \in \mathbb{R}^{R_i \times R_i}$ (the image branch) that captures localized fidelity at resolution $R_i$.

Given a matched image-text pair $\{X, t_X\}$ and a mismatched pair $\{X, t_X\}$, the MAPL discriminator objective is defined as:

$$ \mathcal{L}_{\text{MAPL}}^{D} = \sum_{s=1}^{S} \left( L_2(D_s^{\text{img}}(X)) + L_2(D_s^{\text{pair}}(X, t_X)) + \bar{L}_2(D_s^{\text{img}}(\hat{X})) + \bar{L}_2(D_s^{\text{pair}}(\hat{X}, t_X)) + \bar{L}_2(D_s^{\text{pair}}(X, t_X)) \right), \quad (40) $$

where $S$ is total number of scales, $D_s^{\text{img}}$ is the image loss branch and $D_s^{\text{pair}}$ is the pair loss branch, $L_2(x) = E[(x - \mathbb{I})^2]$ is the mean-square loss for positive samples and $\bar{L}_2(x) = E[x^2]$ for negative samples, and $\hat{X}$ is the generated image from the input text $t_X$.

The generator $\mathcal{G}$ is trained with a complementary objective, encouraging the generation of images that are both realistic and semantically aligned with the input text,

$$ \mathcal{L}_{\text{MAPL}}^{\mathcal{G}} = \sum_{i=1}^{S} \left( L_2(D^{\text{img}}(\mathcal{G}(t_X, z))) + L_2(D_i(\mathcal{G}(t_X, z), t_X)) \right) \quad (41) $$

where $z$ is a latent noise vector. The $L_2$ is preferred to the cross-entropy loss in Eq. 40, as it stabilizes training by preventing vanishing gradients and encourages a larger margin between real and fake samples, which empirically leads to higher-quality generations [41, 42].

To further enforce fine-grained semantic alignment between text and image regions, Xu et al. [11] introduced the **Deep Attentional Multimodal Similarity Module (DAMSM)**. This module encourages cross-modal consistency without requiring explicit word-to-region annotations. Specifically, textual token embeddings and image region features are independently extracted using two encoders, $f_{\text{text}}$ and $f_{\text{region}}$, applied to the input
sentence *s* and image *x*, respectively. Then, a pairwise cosine similarity matrix is computed between each word *w*<sub>*i*</sub> in the sentence and each image region region *r*<sub>*j*</sub> is computed as:

$$
\alpha_{i,j} = \frac{\exp(\rho_1 \cos(f_{\text{text}}(w_i), f_{\text{region}}(r_j)))}{\sum_{h=1}^{R} \exp(\rho_1 \cos(f_{\text{text}}(w_i), f_{\text{region}}(r_h)))}, \quad (42)
$$

where $\cos(u, v) = \frac{u^\top v}{\|u\|_2 \|v\|_2}$ is the cosine similarity, $R$ is the number of image regions, and $\rho_1$ is a sharpening hyperparameter that controls the attention distribution focus. Using these weights, an aligned context feature $c_i$ for each word $w_i$ is computed as a weighted sum over region features:

$$
c_i = \sum_{j=1}^{R} \alpha_{i,j} f_{\text{region}}(r_j). \quad (43)
$$

To quantify the fine-grained alignment between an image *x* and a sentence *s*, a region-aware image-text similarity score is computed:

$$
S(x, s) = \frac{1}{\tau} \log \left( \sum_{i=1}^{T} \exp \left( \rho_2 \cos (f_{\text{text}}(w_i), c_i) \right) \right)^{\frac{1}{\rho_2}}, \quad (44)
$$

where *T* is the number of words in the sentence, *ρ*₂ controls the relative contribution of stronger alignments, and *τ* is a temperature parameter that scales the score.

To train the model, a contrastive loss is applied over a batch of
image-sentence pairs {($x_i$, $s_i$)}$^{M}_{i=1}$, where for each image $x_i$, the
loss encourages high similarity with its corresponding sentence
$s_i$, while pushing apart similarities with non-matching sentences
{ $s_j$}$_{j \neq i}$:

$$
\mathcal{L}_{\text{word}}(x_i, s_i) = - \log \frac{\exp(S(x_i, s_i))}{\sum_{j=1}^{M} \exp(S(x_i, s_j))}. \quad (45)
$$

This objective encourages the model to learn embeddings that
preserve fine-grained cross-modal correspondence, ensuring
that semantically aligned image-text pairs are more similar than
mismatched ones in the shared embedding space.

An alternative alignment mechanism was introduced in **Mirror-GAN** [43], which employed a *text-to-image-to-text* framework to ensure semantic consistency. It incorporated a *Semantic Text Regeneration* (STREAM) module that generated the original textual description from the synthesized image. A *cross-entropy loss* between the generated and ground-truth sentences acted as a global semantic constraint, guiding the generator to preserve fine-grained textual semantics in visual content.

Beyond region–word associations, more recent approaches have
incorporated objectives that align the global image represen-
tations of real and generated samples, as well as the overall
image and sentence embeddings. These objectives are typically
formulated as:

$$
\mathcal{L}_{\text{img}}(x_i, \hat{x}_i) = - \log \frac{\exp(S \text{img}(x_i, \hat{x}_i))}{\sum_j = 1^M \exp(S \text{img}(x_j, \hat{x}_j))}, \quad (46)
$$

$$
\mathcal{L}_{\text{sent}}(x_i, s_i) = - \log \frac{\exp(\cos(f_{\text{img}}(x_i), f_{\text{sent}}(s_i))/\tau)}{\sum_{j=1}^{M} \exp(\cos(f_{\text{img}}(x_i), f_{\text{sent}}(s_j))/\tau)}, \quad (47)
$$

**Algorithm 1 Training Algorithm**

**Require:** Generator and discriminator parameters $\theta_G, \theta_D$; contrastive loss coefficients $\lambda_1, \lambda_2, \lambda_3$; Adam hyperparameters $\beta_1, \beta_2$; learning rates $lr_G, lr_D$; batch size $M$; number of discriminator iterations per generator iteration $N_D$

1: **for** number of training iterations **do**

2: **for** *t* = 1 to *N*<sub>*D*</sub> **do**

3: Sample {$z_i$}^M_{i=1} ~ p(z)

4: Sample {($x_i, s_i$)}^M_{i=1} ~ p_{\text{data}}(x, s)

5: $\mathcal{L}_{\text{sent}}^{\text{r}} \leftarrow \frac{1}{M} \sum_{i=1}^{M} \mathcal{L}_{\text{sent}}(x_i, s_i)$

6: $\mathcal{L}_{\text{word}}^{\text{r}} \leftarrow \frac{1}{M} \sum_{i=1}^{M} \mathcal{L}_{\text{word}}(x_i, s_i)$

7: $\mathcal{L}_{\text{GAN}}^{\text{r}} \leftarrow -\frac{1}{M} \sum_{i=1}^{M} \mathcal{L}_{\text{GAN}}^{\text{D}}(x_i, G(z_i, s_i), s_i)$

8: $\mathcal{L}_D \leftarrow \mathcal{L}_{\text{GAN}}^r + \lambda_1 \mathcal{L}_{\text{sent}}^r + \lambda_2 \mathcal{L}_{\text{word}}^r$

9: $\theta_D \leftarrow \text{Adam}(\mathcal{L}_D, lr_D, \beta_1, \beta_2)$

10: **end for**

11: Sample {$z_i$}^M_{i=1} ~ p(z), {($x_i, s_i$)}^M_{i=1} ~ p_{\text{data}}(x, s) \triangleright \text{Update generator}

12: $\mathcal{L}_{\text{sent}}^{\text{f}} \leftarrow \frac{1}{M} \sum_{i=1}^{M} \mathcal{L}_{\text{sent}}(G(z_i, s_i), s_i)$

13: $\mathcal{L}_{\text{word}}^{\text{f}} \leftarrow \frac{1}{M} \sum_{i=1}^{M} \mathcal{L}_{\text{word}}(G(z_i, s_i), s_i)$

14: $\mathcal{L}_{\text{img}}^{\text{f}} \leftarrow \frac{1}{M} \sum_{i=1}^{M} \mathcal{L}_{\text{img}}(G(z_i, s_i), x_i)$

15: $\mathcal{L}_{\text{GAN}}^{\text{f}} \leftarrow \frac{1}{M} \sum_{i=1}^{M} \mathcal{L}_{\text{GAN}}^{\text{G}}(x_i, G(z_i, s_i), s_i)$

16: $\mathcal{L}_G \leftarrow \mathcal{L}_{\text{GAN}}^f + \lambda_1 \mathcal{L}_{\text{sent}}^f + \lambda_2 \mathcal{L}_{\text{word}}^f + \lambda_3 \mathcal{L}_{\text{img}}^f$

17: $\theta_G \leftarrow \text{Adam}(\mathcal{L}_G, lr_G, \beta_1, \beta_2)$

18: **end for**

where $\hat{x}_i$ denotes the image generated from sentence $s_i$, and $f_{\text{img}}$ and $f_{\text{sent}}$ represent global image and text encoders, respectively. The similarity function $S_{\text{img}}$ is commonly instantiated as a perceptual loss[44], which compares intermediate features from a pretrained VGG network to assess the semantic consistency between real and generated images.

As these three contrastive losses are primarily designed to en-
force semantic alignment between the generated image and the
input text, it is possible to simplify the MAPL loss by removing
the alignment constraints. In doing so, the objective reduces
to a traditional GAN loss, where the standard cross-entropy
formulation is often replaced by a $L_2$ loss.

$$
\mathcal{L}_{\text{GAN}}^{D}(x_i, \hat{x}_i, s_i) = \sum_{s=1}^{S} L_2(D_s(x_i, s_i)) + \bar{L}_2(D_s(\hat{x}_i, s_i)) \quad (48)
$$

$$
\mathcal{L}_{\text{GAN}}^{G} = \sum_{i=1}^{s} L_2(D_s(\hat{x}_i, s_i)). \quad (49)
$$

where $D_s$ is the discriminator at scale *s*. Algorithm 1 summarizes the training procedure for most of the recent literature models. With the emergence of powerful pretrained CLIP models [6], training pipelines have been significantly simplified. In particular, only the image-level contrastive loss ($\lambda_3$) is often retained, while the sentence-level ($\lambda_1$) and word-level ($\lambda_2$) contrastive losses are set to zero. This simplification is enabled by CLIP strong cross-modal alignment capabilities, allowing pretrained image and sentence encoders to provide meaningful supervision without requiring additional alignment losses.
# 5 DATA ACQUISITION AND PREPROCESSING

In contemporary research, diverse datasets are employed to train text-to-image GANs, varying in textual complexity, and visual diversity.

**Human-Annotated Datasets.** Early advances in text-to-image generation were driven by manually curated datasets, where textual descriptions were written by human annotators to provide fine-grained and semantically aligned captions. Prominent datasets include:

* **CUB-200-2011** [45]: Comprising 11,788 images spanning 200 bird species, each image is annotated with multiple natural language descriptions capturing visual attributes such as beak shape, plumage color, and pose.

* **Oxford-102 Flowers** [46]: Consisting of 8,189 images from 102 flower categories, each with detailed captions that describe color patterns, petal structure, and texture.

* **MS COCO** [47]: A large-scale dataset featuring over 80,000 images, each paired with five independently written human captions. It covers diverse real-world scenes involving multiple objects and varying contexts, supporting both coarse and fine-grained text-to-image tasks.

* **Visual Genome** [48]: Contains dense annotations at the region level, including object instances, relationships, and attributes. It serves as a bridge between image captioning and structured scene understanding, and is particularly valuable for training models that require spatial and relational reasoning.

* **Localized Narratives COCO** [49]: An extension of MS COCO, providing rich, free-form spoken descriptions that are temporally aligned with mouse trajectories over the image. Each phrase is grounded to a visual region, enabling fine-grained vision-language alignment and supporting tasks such as grounded captioning and spatially-aware image generation.

These human-annotated datasets offer high linguistic quality and strong semantic alignment. However, they are often limited in scale due to the labor-intensive annotation process, and may lack the diversity or quantity required to fully train large-scale generative models

**Automatically-Annotated Datasets.** To scale dataset creation and increase domain diversity, recent work has shifted toward automatic annotation of web-crawled [50] images. These pipelines typically pair images with associated alternative text (e.g., HTML alt attributes, metadata, or surrounding text), sourced from large-scale web corpora such as Wikipedia, Flickr, Reddit, and other publicly available repositories. This process is generally standardized into four stages:

1. **Image-Based Filtering:** Removal of low-quality or inappropriate images based on criteria such as resolution, aspect ratio, file format, and presence of explicit content.

2. **Text-Based Filtering:** Elimination of noisy or low-value text based on language identification, token frequency, capitalization norms, predefined blacklist phrases, part-of-speech distributions, sentiment polarity, and adult content detection.

3. **Image-Text Alignment Filtering:** Retention of image-text pairs based on alignment heuristics, often using tag overlap or learned similarity metrics. For example, many datasets rely on a minimum CLIP similarity score between the image and its paired text.

4. **Text Transformation:** Normalization of captions via operations such as hypernym replacement of named entities (e.g., replacing “Barack Obama” with “person”), removal of temporal expressions, and masking of digits.

The key differentiator among automatically-annotated datasets lies in the stringency of filtering criteria. Stricter rules improve alignment quality but reduce dataset size, while relaxed filters enable massive scaling at the cost of noise. Prominent datasets include:

* **CC3M (Conceptual Captions 3M)** [51]: A foundational dataset created by extracting image-text pairs from the web via Common Crawl, where image alt-text is filtered to retain semantically meaningful and image-relevant captions.

* **CC12M** [52]: An extension of CC3M with relaxed filtering enables larger-scale training, albeit with increased caption noise.

* **WIT** [53]: Over 37 million image-text associations from Wikipedia, including multilingual data (108 languages) and accurate textual descriptions grounded in encyclopedic content.

* **CLIP WIT** [6]: A filtered subset of WIT used in CLIP training, containing ~400 million high-alignment image-text pairs.

* **RedCaps** [54]: Derived from Reddit posts in photography-related subreddits, containing over 12 million informal and community-authored image-caption pairs with reddit labels.

* **YFCC100M** [55]: A massive collection of 100 million images/videos from Flickr under Creative Commons licenses. Weakly labeled via tags, titles, and metadata; useful for noisy, real-world pretraining.

* **LAION-5B** [56]: The largest publicly available image-text dataset (>5B pairs), constructed using web crawling, CLIP-based similarity scoring, aesthetic estimators, and safety filters.

* **ALIGN** [57]: A proprietary dataset by Google with ~1.8 billion image-text pairs. Not publicly available due to licensing constraints. Trained the ALIGN model using large-scale weak supervision.

* **BASIC** [58]: Another proprietary dataset used in training BLIP-2 and other large-scale models. Combines internal/public sources with curated filters for high-quality alignment.

Table 1 summarizes the main datasets used for training text-to-image models, categorized by annotation type, data source, and number of samples.
<table><caption>Table 1: Overview of major datasets used for training text-to-image models, categorized by availability. Only English image-text pairs are counted.</caption><thead><tr><th>Dataset</th><th>Annotation Type</th><th>Source</th><th># Samples</th></tr></thead><tbody><tr><td colspan="4" style="text-align: center; font-style: italic;">Public Datasets</td></tr><tr><td>MS COCO [47]</td><td>Human captions (crowd-sourced)</td><td>Crowd-sourced</td><td>330K</td></tr><tr><td>Localized Narratives COCO [49]</td><td>Grounded spoken captions</td><td>Crowd-sourced</td><td>849k</td></tr><tr><td>CC3M [51]</td><td>Alt-text captions</td><td>Common Crawl (web crawl)</td><td>3M</td></tr><tr><td>Visual Genome [48]</td><td>Human region descriptions</td><td>Crowd-sourced</td><td>5.4M</td></tr><tr><td>WIT [53]</td><td>Wikipedia image context captions</td><td>Wikipedia</td><td>5.5M</td></tr><tr><td>CC12M [52]</td><td>Alt-text captions (relaxed filtering)</td><td>Common Crawl (web crawl)</td><td>12M</td></tr><tr><td>RedCaps [54]</td><td>Informal user captions</td><td>Reddit photography subreddits</td><td>12M</td></tr><tr><td>YFCC100M [55]</td><td>Metadata tags</td><td>Flickr</td><td>100M</td></tr><tr><td>LAION-5B [56]</td><td>Alt-text captions + filters</td><td>Web crawl</td><td>2.3B</td></tr><tr><td colspan="4" style="text-align: center; font-style: italic;">Proprietary Datasets</td></tr><tr><td>CLIP WIT [6]</td><td>Filtered Wikipedia captions</td><td>Web data (including Wikipedia)</td><td>~400M</td></tr><tr><td>ALIGN [57]</td><td>Alt-text captions</td><td>Web crawl (internal)</td><td>~1.8B</td></tr><tr><td>BASIC [58]</td><td>Mixed curated captions</td><td>Mixed (internal + public)</td><td>6.6B</td></tr></tbody></table>

# 6 MODEL EVALUATION

Following model training, evaluating its performance and benchmarking against state-of-the-art (SOTA) methods is essential. A comprehensive evaluation framework should assess both the quality of generated images and the degree to which these outputs are semantically aligned with the corresponding input text. This evaluation specifically targets the generator to isolate its performance and capabilities, as the discriminator role is primarily to guide training rather than to produce outputs for direct assessment.

Let $\mathcal{D} = \{(x_i, t_i, y_i)\}_{i=1}^N$ denote a dataset where each triplet consists of an image $x_i$, an associated text prompt $t_i$, and a class label $y_i$. In order to evaluate the pretrained generator $G$, a synthetic dataset is created by feeding the text prompts into the pre-trained generator, resulting in a synthetic dataset $\mathcal{D}_{syn} = \{(G(t_i, z), t_i, y_i)\}_{i=1}^N$.

**Subjective Test.** The simplest approach for evaluating a generated data is through **human evaluation**. In this setup, a group of annotators is presented with a set of generated images and asked to rank them based on two primary criteria: (1) visual realism, and (2) semantic alignment with the input text description. Although human evaluation provides valuable qualitative insights, it is inherently costly, time-consuming, and difficult to scale or reproduce consistently.

To overcome these limitations, researchers have proposed several automated metrics to evaluate generative models without human intervention. These metrics can be broadly categorized into three groups: Distribution-Based Metrics, Semantic Alignment Metrics, and Diversity Metrics.

## 6.1 Distribution-Based Metrics.

The first group measures the divergence between the real data distribution $p_{data}(x)$ and the model generative distribution $p(x)$, either explicitly or implicitly.

**Inception Score** [59] is an implicit metric that uses an auxiliary classifier trained on the real dataset to predict class labels

for generated images. Then computes the KL-divergence between the conditional class distribution and the marginal class distribution over generated samples:

$$IS = \exp(\mathbb{E}_x KL(p(y|x) || p(y))), \quad (50)$$

where, $p(y|x)$ is the conditional label distribution predicted by the auxiliary classifier, $p(y)$ is the marginal $\int p(y|x = G(z))dz$, and KL is KL-divergence. The intuition behind this metric is that good models should generate diverse but meaningful images. Therefore, the KL divergence between the marginal distribution $p(y)$ and the conditional distribution $p(y|x)$ should be large.

**Frechet Inception Distance (FID)** [60] is an explicit metric that compares statistical summaries of high-level image features extracted from a pretrained CNN. FID models these feature embeddings as samples drawn from multivariate Gaussian distributions and measures the distance between the generated and real image distributions by comparing their respective means and covariances:

$$FID = \|m - m_r\|_2^2 + \mathrm{Tr}(C + C_r - 2(CC_r)^{1/2}) \quad (51)$$

where, $(m, C)$ are the mean and covariance obtained features embeddings of syntactic data, and $(m_r, C_r)$ are mean and covariance obtained from features embeddings of real data. The Lower FID values mean closer distances between synthetic and real data distributions.

## 6.2 Semantic Alignment Metrics

The second group of metrics evaluates the semantic alignment between generated images and their corresponding input sentence. These metrics require pretrained visual and text encoders, $f_{img}$ and $f_{sent}$, which map images and sentences into a shared embedding space. This space is typically learned by minimizing a bi-directional ranking loss on real image-sentence pairs:

$$\sum_{\mathbf{v}} \sum_{\mathbf{t}_{\mathbf{v}}} \max(0, \delta - c(f_{\mathrm{img}}(\mathbf{v}), f_{\mathrm{sent}}(\mathbf{t}_{\mathbf{v}})) + c(f_{\mathrm{img}}(\mathbf{v}), f_{\mathrm{sent}}(\mathbf{t}_{\mathbf{v}}))) \quad (52)$$
$$+ \sum_{\mathbf{t}} \sum_{\mathbf{v}_{\mathbf{t}}} \max (0, \delta - c(f_{\text{sent}}(\mathbf{t}), f_{\text{img}}(\mathbf{v}_{\mathbf{t}})) + c(f_{\text{sent}}(\mathbf{t}), f_{\text{img}}(\mathbf{v}_{\mathbf{t}}))), \quad (53)$$

where $c(\mathbf{x}, \mathbf{y}) = \frac{\mathbf{x} \cdot \mathbf{y}}{\|\mathbf{x}\|_2 \|\mathbf{y}\|_2}$ is the cosine similarity function, $\delta$ is the margin set to 0.2, $\{\mathbf{v}, \mathbf{t}\}$ denotes a matched image-sentence pair, and $\{\mathbf{v}_{\mathbf{t}}, \mathbf{t}\}$ denote mismatched pairs.

**Visual-Semantic Similarity** [27]. After training, a simpler evaluation methodology computes the average cosine similarity between each generated image embedding and its respective input sentence embedding. Higher scores indicate better semantic consistency.

**R -Precision** [11]. A more challenging evaluation involves constructing, for each generated image, a candidate set of N sentences consisting of one correct sentence and N-1 distractors. Both the image and all candidate sentences are embedded, and the metric reports the fraction of samples where the correct sentence is ranked among the top-k retrieved candidates based on cosine similarity.

**Semantic Object Accuracy (SOA).** While existing evaluation metrics primarily assess overall image quality, they often overlook the fidelity of individual objects within generated images. To address this limitation, Hinz et al. [61] proposed quantifying the presence of objects explicitly mentioned in the input text by leveraging a pre-trained object detector. Two complementary metrics are introduced: (1) SOA-C, the class-averaged recall, which measures the average fraction of images per object class in which the specified object is detected, and (2) SOA-I, the image-averaged recall, which measures the average fraction of images across all classes where the desired object is detected.

These metrics are formally defined as follows. Let $C$ denote the set of object classes, and for each class $c \in C$, let $I_c$ be the set of generated images that are supposed to contain an object of class $c$. We define the detection indicator function $\mathcal{D}$, representing a generic object detector, as

$$\mathcal{D}(i_c, c) = \begin{cases} 1 & \text{if } c \text{ is detected in image } i_c, \\ 0 & \text{otherwise.} \end{cases} \quad (54)$$

Then, the class-averaged recall (SOA-C) is computed as

$$\text{SOA-C} = \frac{1}{|C|} \sum_{c \in C} \frac{1}{|I_c|} \sum_{i_c \in I_c} \mathcal{D}(i_c, c), \quad (55)$$

and the image-averaged recall (SOA-I) is given by

$$\text{SOA-I} = \frac{1}{\sum_{c \in C} |I_c|} \sum_{c \in C} \sum_{i_c \in I_c} \mathcal{D}(i_c, c). \quad (56)$$

**Caption Generation.** Hong et al. [62] evaluate their model by comparing the original captions with those generated by a pre-trained caption generator[63] using standard language similarity metrics, such as BLEU, METEOR, and CID.

## 6.3 Diversity Metrics

The third group of methods contains the method that evaluate how much the generated images are diverse, i.e., how each generated image diffrent with an other. The most used metric is this group is multiscale structural similarity (MS-SSIM) [64, 65], which is usually computed on randomly chosen pairs within a given class. Samples from classes that have higher diversity result in lower mean MS-SSIM scores samples from classes with lower diversity have higher mean MS-SSIM scores.

# 7 CONCLUSION

Text-to-image generation using generative adversarial networks has progressed rapidly, enabled by advances in conditional modeling, cross-modal alignment, and high-resolution synthesis. Despite these gains, key challenges persist, including fine-grained spatial control, semantic consistency under complex prompts, and generalization to open-domain settings.

## REFERENCES

[1] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014. URL https://arxiv.org/abs/1406.2661.

[2] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, 2016. URL https://arxiv.org/abs/1511.06434.

[3] Emily Denton, Soumith Chintala, Arthur Szlam, and Rob Fergus. Deep generative image models using a laplacian pyramid of adversarial networks, 2015. URL https://arxiv.org/abs/1506.05751.

[4] Alex Sherstinsky. Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network. Physica D: Nonlinear Phenomena, 404:132306, March 2020. ISSN 0167-2789. doi: 10.1016/j.physd.2019.132306. URL http://dx.doi.org/10.1016/j.physd.2019.132306.

[5] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https://arxiv.org/abs/1706.03762.

[6] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL https://arxiv.org/abs/2103.00020.

[7] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis, 2016. URL https://arxiv.org/abs/1605.05396.

[8] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris Metaxas. Stackgan: Text to photo-realistic image synthesis with
stacked generative adversarial networks, 2017. URL https://arxiv.org/abs/1612.03242.

[9] Scott Reed, Zeynep Akata, Santosh Mohan, Samuel Tenka, Bernt Schiele, and Honglak Lee. Learning what and where to draw, 2016. URL https://arxiv.org/abs/1610.02454.

[10] Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs, 2018. URL https://arxiv.org/abs/1804.01622.

[11] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attention: Fine-grained text to image generation with attentional generative adversarial networks, 2017. URL https://arxiv.org/abs/1711.10485.

[12] Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dan- gan: Dynamic memory generative adversarial networks for text-to-image synthesis, 2019. URL https://arxiv.org/abs/1904.01310.

[13] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization, 2017. URL https://arxiv.org/abs/1703.06868.

[14] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift, 2015. URL https://arxiv.org/abs/1502.03167.

[15] Guojun Yin, Bin Liu, Lu Sheng, Nenghai Yu, Xiaogang Wang, and Jing Shao. Semantics disentangling for text-to-image generation, 2019. URL https://arxiv.org/abs/1904.01480.

[16] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks, 2019. URL https://arxiv.org/abs/1812.04948.

[17] Ming Tao, Hao Tang, Fei Wu, Xiao-Yuan Jing, Bing-Kun Bao, and Changsheng Xu. Dan- gan: A simple and effective baseline for text-to-image synthesis, 2022. URL https://arxiv.org/abs/2008.05865.

[18] Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-modal contrastive learning for text-to-image generation, 2022. URL https://arxiv.org/abs/2101.04702.

[19] Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis, 2023. URL https://arxiv.org/abs/2303.05511.

[20] Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis, 2023. URL https://arxiv.org/abs/2301.09515.

[21] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan, 2020. URL https://arxiv.org/abs/1912.04958.

[22] Grigorios G. Chrysos, Stylianos Moschoglou, Giorgos Bouritsas, Jiankang Deng, Yannis Panagakis, and Stefanos P Zafeiriou. Deep polynomial neural networks. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, page 1–1, 2021. ISSN 1939-3539. doi: 10.1109/tpami.2021.3058891. URL http://dx.doi.org/10.1109/TPAMI.2021.3058891.

[23] Tu Dinh Nguyen, Trung Le, Hung Vu, and Dinh Phung. Dual discriminator generative adversarial nets, 2017. URL https://arxiv.org/abs/1709.03831.

[24] Ishan Durugkar, Ian Gemp, and Sridhar Mahadevan. Generative multi-adversarial networks, 2017. URL https://arxiv.org/abs/1611.01673.

[25] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris Metaxas. Stacked++: Realistic image synthesis with stacked generative adversarial networks, 2018. URL https://arxiv.org/abs/1710.10916.

[26] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation, 2018. URL https://arxiv.org/abs/1710.10196.

[27] Zizhao Zhang, Yuanpu Xie, and Lin Yang. Photographic text-to-image synthesis with a hierarchically-nested adversarial network, 2018. URL https://arxiv.org/abs/1802.09178.

[28] Quan Hoang, Tu Dinh Nguyen, Trung Le, and Dinh Phung. Multi-generator generative adversarial nets, 2017. URL https://arxiv.org/abs/1708.02556.

[29] Animesh Karnewar and Oliver Wang. Msg-gan: Multi-scale gradients for generative adversarial networks, 2020. URL https://arxiv.org/abs/1903.06048.

[30] Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger. Projected gans converge faster, 2021. URL https://arxiv.org/abs/2111.01007.

[31] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. URL https://arxiv.org/abs/1512.03385.

[32] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows, 2021. URL https://arxiv.org/abs/2103.14030.

[33] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. URL https://arxiv.org/abs/2010.11929.

[34] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention, 2021. URL https://arxiv.org/abs/2012.12877.

[35] Scott Reed, Zeynep Akata, Bernt Schiele, and Honglak Lee. Learning deep representations of fine-grained visual descriptions, 2016. URL https://arxiv.org/abs/1605.05395.

[36] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby,
Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2024. URL https://arxiv.org/abs/2304.07193.

[37] Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. Lafite: Towards language-free training for text-to-image generation, 2022. URL https://arxiv.org/abs/2111.13792.

[38] Minguk Kang and Jaesik Park. Contragan: Contrastive learning for conditional image generation, 2021. URL https://arxiv.org/abs/2006.12681.

[39] Jongheon Jeong and Jinwoo Shin. Training gans with stronger augmentations via contrastive discriminator, 2021. URL https://arxiv.org/abs/2103.09742.

[40] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. In Proc. NeurIPS, 2021.

[41] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks, 2018. URL https://arxiv.org/abs/1802.05957.

[42] Jae Hyun Lim and Jong Chul Ye. Geometric gan, 2017. URL https://arxiv.org/abs/1705.02894.

[43] Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. Mirrorgan: Learning text-to-image generation by redescription, 2019. URL https://arxiv.org/abs/1903.05854.

[44] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution, 2016. URL https://arxiv.org/abs/1603.08155.

[45] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset, August 2023.

[46] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722–729, 2008. doi: 10.1109/ICVGIP.2008.47.

[47] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision – ECCV 2014, pages 740–755, Cham, 2014. Springer International Publishing. ISBN 978-3-319-10602-1.

[48] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. Visual genome: Connecting language and vision using crowdsourced dense image annotations, 2016. URL https://arxiv.org/abs/1602.07332.

[49] Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connecting vision and language with localized narratives, 2020. URL https://arxiv.org/abs/1912.03098.

[50] Craig Chambers, Ashish Raniwala, Frances Perry, Stephen Adams, Robert R. Henry, Robert Bradshaw, and Nathan Weizenbaum. Flumejava: easy, efficient data-parallel pipelines. In Proceedings of the 31st ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI '10, page 363–375, New York, NY, USA, 2010. Association for Computing Machinery. ISBN 9781450300193. doi: 10.1145/1806596.1806638. URL https://doi.org/10.1145/1806596.1806638.

[51] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL, 2018.

[52] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts, 2021. URL https://arxiv.org/abs/2102.08981.

[53] Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, page 2443–2449. ACM, July 2021. doi: 10.1145/3404835.3463257. URL http://dx.doi.org/10.1145/3404835.3463257.

[54] Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. Redcaps: web-curated image-text data created by the people, for the people, 2021. URL https://arxiv.org/abs/2111.11431.

[55] Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: the new data in multimedia research. Communications of the ACM, 59(2):64–73, January 2016. ISSN 1557-7317. doi: 10.1145/2812802. URL http://dx.doi.org/10.1145/2812802.

[56] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models, 2022. URL https://arxiv.org/abs/2210.08402.

[57] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision, 2021. URL https://arxiv.org/abs/2102.05918.

[58] Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, Mingxing Tan, and Quoc V. Le. Combined scaling for zero-shot transfer learning, 2023. URL https://arxiv.org/abs/2111.10050.
[59] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, 2016. URL https://arxiv.org/abs/1606.03498.

[60] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium, 2018. URL https://arxiv.org/abs/1706.08500.

[61] Tobias Hinz, Stefan Heinrich, and Stefan Wermter. Semantic object accuracy for generative text-to-image synthesis. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 44(3):1552–1565, March 2022. ISSN 1939-3539. doi: 10.1109/tpami.2020.3021209. URL http://dx.doi.org/10.1109/TPAMI.2020.3021209.

[62] Seunghoon Hong, Dingdong Yang, Jongwook Choi, and Honglak Lee. Inferring semantic layout for hierarchical text-to-image synthesis, 2018. URL https://arxiv.org/abs/1801.05091.

[63] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: Lessons learned from the 2015 mscoco image captioning challenge. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 39(4):652–663, April 2017. ISSN 2160-9292. doi: 10.1109/tpami.2016.2587640. URL http://dx.doi.org/10.1109/TPAMI.2016.2587640.

[64] Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans, 2017. URL https://arxiv.org/abs/1610.09585.

[65] Zhou Wang, Eero P. Simoncelli, and Alan C. Bovik. Multi-scale structural similarity for image quality assessment. *Conference Record of the Asilomar Conference on Signals, Systems and Computers*, 2:1398–1402, 2003. ISSN 1058-6393. Conference Record of the Thirty-Seventh Asilomar Conference on Signals, Systems and Computers ; Conference date: 09-11-2003 Through 12-11-2003.