End-to-end Training for Text-to-Image Synthesis using Dual-Text Embeddings
Authors: Yeruru Asrar Ahmed
Anurag Mittal
Venue: Accepted by TMLR ()

Abstract: Text-to-Image (T2I) synthesis is a challenging task that requires modeling complex interactions between two modalities ( i.e., text and image). A common framework adopted in recent state-of-the-art approaches to achieving such multimodal interactions is to bootstrap the learning process with pre-trained image-aligned text embeddings trained using contrastive loss. Furthermore, these embeddings are typically trained generically and reused across various synthesis models. In contrast, we explore an approach to learning text embeddings specifically tailored to the T2I synthesis network, trained in an end-to-end fashion. Further, we combine generative and contrastive training and use two embeddings, one optimized to enhance the photo-realism of the generated images, and the other seeking to capture text-to-image alignment.  A comprehensive set of experiments on three text-to-image benchmark datasets (Oxford-102, Caltech-UCSD, and MS-COCO) reveal that having two separate embeddings gives better results than using a shared one and that such an approach performs favourably in comparison with methods that use text representations from a pre-trained text encoder trained using a discriminative approach.  Finally, we demonstrate that such learned embeddings can be used in other contexts as well, such as text-to-image manipulation.

Published in Transactions on Machine Learning Research (06/2025)
End-to-end Training for Text-to- Synthesis using Dual- Text Embeddings
Yeruru Asrar Ahmed
asrar@cse.iitm.oc
Deportment of Computer Science and Engineering Indian Institute of Technology Madras
Anurag Mittal
amittal@@se.iüämm wwwm.
Deportment of Computer Science and Engineering Indian Institute of Technology Modras
Reviewed on OpenReview: htt tps:: ew wwwp://wwee//eeee.mm..mm um ?t d= 10
HV
Abstract
Text-to-Image (T2I) synthesis is a challenging task that requires modcling complex inter- actions between two modalities (, f.e., text and image). A common framework adopted in recent state-of-the-art approaches to achieving such multimodal interactions is bootstrap the learning process with pre-trained image-aligned text embeddings trained using con- trastive loss. Furthermore, these embeddings across various synthesis models. In contrast, we explore an approach to learning text em- beddings specifically tailored to the T2I synthosis network, trained in an end-to-end fashion. Further, we combine generative and contrastive training and use two embeddings, one opti- mized to enhance the photo-realiste text-to-image alignment. comprchensive set of experiments on three text-to-image bench- mark datasets (Oxford-102, Caltech-UCSII and MS-COCO) reveal that having two separate embeddings gives better results than using a shared one and that such approach performs favourably in comparison with methods that use text representations from pre-trained text encoder trained using discriminative approach. Finally, we demonstrate that such learned embeddings can be used in other contexts as well, such as texl-to-image manipulation.
typically trained generically and reused
the generated images, and the other seeking to capture
1 Introduction
Visualizing images for any textual statement is elemental to human understanding of the world. Intelligent systems' ability to generate images from the text for human understanding has wide range of applications such as information sharing, computer-aided design, text-to-image search, and photo editing, Image syuthesis from text is challenging task due to complex interaction and ambiguous association of the text modality with the image modality. For instance, multiple textual descriptions can describe the same image and vice versa. In addition, finer details of the images may not always be well captured in textual descriptions. this domain, Generative Adversarial Networks (GANs) (Goodfellow et al. 2014) were the go-to method generate realistic images. Conditional GANs (Mirza & Osindero 20044 Odena et al. 2017 Miyato & Koyamna 2018) allow
generate real images semantically coherent with text (Reed et al., 2016cc Dash al. 2017 Reed et al. 2016b) by conditioning the generation process global sentence embeddings.
Though GANs have been shown to generate mcaningful images, naively generating high-rosolution images from text leads to subpar visual results and training instability due to the complex nature of the task. A set methods attempts to solve this problem by advancing the visual generation part of the model. For instance, StackGAN (Zhang et al., (017b) employs a héerarchical stage-wise training of GANs from a low- resolution a high resolution and conditions the generator at every stage by images generated from the previous stage generator.

---

Published in Transactions on Machine Learning Research (06/2025)
Image
Image
Text
smadl wi bread
Teet
Irage
breatand
Contrastive Lec6
Lons
Mdack urems
Dreast
Tea
Text
Generatel ated
Image
Text
M
Figure (a) Text and image are projected into a shared embedding space enhance mutual informa- tion capturing discriminative features (Discriminative Fmbeedingg), (b) Word embeddings are trained by generating images capturing semantic details (Generative Embeddings) and (c) Our Dual Text Embedding approach combining both generative and discriminative embeddings.
Another set of methods (Xu et al. 2018 Zhu et al. 2019 Liang et al., 2020 addresses high-quality image generation from a text by improving the compatibility between text and image modalities. achieved by semantically aligning the visual features image sub-regions with the pro-trained word emboddings through allention. These pre-trained embeddings are trained by projecting lext and image features into shared embedding space and maximising mutual information between text and image features using contrastive loss [uu et al. 2018 Radford et al.] 2021). [. Furthermore, these embeddings are typically trained generically and
reused across various synthesis models [u et al. 2018 Liang et al. 2020 Tao et al. 2022 Liao et al. 2022
Ramesh et. al. 2021 2022 Gafni et al. 2022 Rombach et al. 2021 ). This work explores a new direction for learning text representations within the Text-to-Image (T21) framework through unified end- aud-to-end training, as illustrated in Figure [ to enhance compatibility between image and text modalities. Additionally, we distinct. embeddings: one optimized for
combine generative and contrastive training while utilizing enhancing the photorealism of generated images and the other focused on capturing accurate text-to-image alignment shown in Figure [ We evaluate the proposed approach three datasets: 1) Oxford-102 Nilssback & Zisserman 2008). 2) Caltech-
Birds 200 (CUB) (Welinder et al., 2010), and 3) MS- COCC [Lin et al. (2144b). We use three metrics to assess the generated images: Inception Score (IS) (Salimans et al. 2016), Frechet Inception Distance (FID) (Heusel et al. 2017) for image quality, and R-precision (Xu et al. 2018 to measure lext-image alignmentt Our model reduces the FID score from 14.06 13.67 on CUB and
40.31 30.07 on Oxford-102 For MS-COCO, the FID drops from 35.49 to 25.17, outperforming AttnGAN [u et al. 2018), which uses embeddings trained generically in discriminative approach using contrastive loss. Furthermore, we observe that employing separate embeddings gives superior results compared to a shared embedding approach as verified in Section 4.3.4 4 Finally, we demonstrate that such learnt dual-text. embeddings can be used in other contexts as well, such as for text-to-image manipulation.
The contributions this paper are summarized as follows:
We propose novel approach for learning text embeddings specifically tailored to the requirements of Text-to-Image synthesis networks, optimized through end-to-end training paradigm to enhance synthesis performance.
We introduce dual-embedding framework that integrates generative and contrastive training paradigms, with one embedding optimized for photo-realism and the other for robust text-to-image alignment.
Our approach demonstrates competitive performance compared to methods that utilize text rep- resentations derived from pre-trained text encoders optimized using discriminative training paradigm.
We show the application of such learnt embeddings for additional tasks, such as text-to-image manipulation.
2

---

Published in Transactions on Machine Learning Research (06/2025)
Dual Text
Embedding Setup
Generated Image
Bo
Diecrimnntto-al
Multi- Modal
Text Encoder
$D
Contrastive Loss
Concal
yellow with and has long, pointy
Generator-side
2-N00.1
Adversarial
Wi
(G)
Discriminator (D)
Real Image
X
Conditioning Augmentation
Modulation parameters
FC with reshape
Upsampling CEN
Downsampling
Gradient
blocks
Figure 2: Overview of DTE-GAN architocture. DTE-GAN consists of three core components: singlo-stage generator G (Section 3.2 ii) discriminator D (Section 3.3 and iii) dual lext embedding setup (Section 3.4. In the Figure, Wa generator-side word embeddings, SG generator-side sentence embedding, WD discriminator-side word emboddings, SD = discriminator-side sentence embedding. The model is optimised using two objective functions: 1) adversarial loss, and 2) multi-modal contrastive loss.
2
Related Work
In this section, some of the relevant. works in the literature relating to this paper discussed briefly.
Generative Adversarial Networks: In past few years, GANs (Goodfellow et al. 2014 had been the
go-to method for generating images and class-specific images (Mirza & Osindero 2014 Odena et al. 2017
Miyato & Koyama, 2018) on small datasets such as MNIST (LeCun & Cortes 2010) and CIFAR (Krizhevsky
et al.). However, GAN training is highly unstable when used to generate images on large datasets such as
ImageNet (Deng al. 2009p. Researchers have explored fix this training instability by re-framing GAN
loss and regularisation (Arjovsky et al. 2017 Gulrajani
2017 Mao al., 2017 Miyato al.) 2018
Brock al. 2017 generate high-resolution images large datasets (Krrras al. 2018 Brock et al.
2019).
Text-to-Image synthesis: GANs conditioned on global sentence-level embeddings are known to generate meaningful images at low resolutions (Reed et al.) 2016b Dash et al. 2017 Reod et al.] 2016c). StackGAN (Zhang et al. 2017b) generales high-resolution images in stage-wise approach, where the generator at each stage conditioned by the image generated from the previous stage. Unlike StacKGAN, HDGAN (Zhang et al. 2018) trains single generator and multiple discriminators for each resolution. AttnGAN (Xu et al. 2018 uses text embeddings to fine-tune image features and also introduccs multimodal contrastive loss (DAMSM loss) to bridge the gap between generated images and words. DM-GAN (Zhu et. al. 2019 refines words and image features using a memory module. MirrorGAN (Qiao et al. 2019b) generates a caption for the generated images that improves the text ws. image semantic consistency. SD-GAN (Yin et al. 2019
introduces a Siamese structure for the generator that uses Conditional Batch Normalization (CBN) (Chen
et al. 2019) improve the alignment of text-image. CPGAN (Liang et al., (020) learns memory-attended
text encoder by attending salient features in images for each word and Lnn-grriied discriminator (Li et al. 2019). DTGAN (Zhang & Schomaker 2020) applies channel and spatial attention, conditioned
sentence vector to focus important features for each textual representation. XMC-GAN (Zhang et al, 2021] maximises the mutual information between text and image using intra-modality and inter-modality contrastive losses. DF-GAN (Tao et al.) 2022) uses deep affine transformed global sentence embedding to condition the geand theator and the matching aware discriminator. In this space text-to-image semantic alignment-baseo methods, pro-trained emboddings are an inherent prerequisite. These embeddings are only trained by the discriminative approach. Unlike these mothods, our apporach (DTE) attempts learn text embeddings that capture generative and discriminative properties.
3

---

Published in Transactions on Machine Learning Research (06/2025)
Generative embedding learning: Some methods attempt to learn embeddings (text / visual) end-to-end 8s part of a generator. For example, Qvan den Oord et al. 2017 Esser et al.) 2020) learn discrete embeddings for visual representation and show substantial improvement in the performance of text-to-image synthesis (Ramesh et al.] 2021 Ding et al.] 2021). Further better and compact representation are learned to improve the quality of image generation Razavi et al. 2019: Lee et al. 2022 Gafni et al., 2022). Unlike these works that consider only the generation process while learning embeddings, DTE explores the capture of different perspectives of generation and discrimination process by learning dual text embeddings.
Large Scale Text-to-Image Synthesis: Denoising Diffusion Probabilistic models (Sohl-Dickstein et al. 2015 have currently achieved remarkable success in image generation (Ho ct al.) 2020 Nichol & Dhariwal
2021
Dhariwal & Nichol 2021) by revensing the forward Markovian process with noise removal in multiple
steps. Though diffusion- models are able to generate images with complex and varied interactions for
text and generate high-quality images (Ramesh et al. 2022 Saharia et al., 2022 Gu et al., 2022), these approaches require largo-scale training and exploit pre-trained discriminative language models like CLIP
[Radford et. al., 2021). Further, CLIP-based language and image encoders are used as a bootstrapping
approach for predicting conditional representations in large-scale GAN-based approaches (Tao et al 2023
Kang et al. 2023 Zhou et al. 2021). Unlike CLIP, which is trained to capture text-image alignment only,
DTE is proposed to learn text-image alignment and text representation toimprove image realism.
3 Methodology
In this section; an end-to-end framework called "Dual Tert Embedding GAN' (DTE-GAN) is formulated for learning lext embeddings learn tailored to the T21 synthesis network in an end-to-end manner while capturing different representation for text for improving photo-realism and capturing text-image alignment. In the following sub-scctions, the overall architecture is introduced followed by specific details on generator and discriminator architectures respectively; then the final sub-section focuses on learning dual text embeddings.
3.1 Model overview
DTE-GAN consists of three core components: 1) novel dual text embodding sctup, 2) singlo-stage generator G, and 3) discriminator D. The overall architecture is shown in Figure] 2
First, the given text T is passed through the novel dual text embedding procedure to encode the text into two types of sentence emboddings, namely: 1) Generator-side sentence embedding Sc, and 2) Discriminator- side sentence embedding Sn. The dual text embedding setup consists of two separate Bi-LSTM (Schuster) & Paliwal 1997p text encoders (Generator-side and Discriménator-sten and their own independent word embeddings (Wa & Wp). As the name suggests, the Generator-side word embeddings Wa and its en- coder are intended to be trained from the image-generation process (generator G) and its losses, while the Diseriminator-side word embeddings WD and its encoder are optimised by the contrastive loss at the dis- criminator D. Such a decoupling between these two parts of embedding adds flexibility in capturing different natures of image generations and discrimination processes. Specifically, the image generation process war- rants learning representation for creating an image, while the discrimination process strives to learn features for improving text-image aligument. Further, the scparation of these two emboddings allows We to learn from noisy gradients of G independently (as G's gradients are initially noisy due to fake image sentence pairs), while WD learns from stable gradients of D (real image sentence pairs).
During training, Wo, the generator-side text encoder, and Generator G are trained from gradient signals of the generation process i.e., Adversarial loss of fake images faae and multi-modal contrastive loss bet ween the generated image Lake and the given text T. Next, WD and the discriminator-sinn text encoder are trained from the gradient signals of the multi-modal contrastive loss between the real image Treat and the given text T. Further, discriminator D is trained from Adversarial loss (Ifake, Ireat) and multi-modal contrastive loss between the real image Treat and the given text. T.
4

---

Published in Transactions on Machine Learning Research (06/2025)
3.2 Generator
As opposed to other methods that use slacks of GANs or hierarchical GAN, single-stage generator is employed that can generate an image at any resolution owing to its simpler design and easier training procodure. The generator G takes three inputs: i) a noise voctor dimension d, from a Standard Gaussian Distribution (000 1) with the Truncation Trick (Brock et al., 2019 Tao et al., 2022 Zhang Schomaker 2020), ii) the generator-side sentence embedding Sg dimension dsc, and iii) the discriminator-side sentence embedding SD of dimension dsp. Next, the two sentence embeddings (Sc, Sp) together are passed through the conditioning augmentation (Zhang al. 2017b) to get the conditional vector C. which is concatenated with a noise vector sampled from a Standard Gaussian Distribution ((0, 1) to form the input vector Jc and passed through fully connected layer with reshape to create low-resolution spatial feature map. It may be noted that, in this step, Sp is detached from the gradient flow of G to avoid getting gradients from the generation process (refer to ablation studies Section 4.3.4).
Further, this low-rcsolution feature map is passed through a set of upsampling blocks (UpBlock) followed by convolution layer that accepts the last high-resolution feature map and outputs the generated image Iake of dimension 3 x h X (h height, w width). Each UpBlock is formulated as residual layer consisting of bi-linear upsampling step followed by two convolution blocks (convolutional layer + Conditional Batch normalisation (CBN) Chen et al., 2019) + LeakyReLU (Maas et al., 2013)). To increase the stochastic capability of the model, scaled noise is added (similar to StyleGAN (Karras et al., 2019)) to input before passing to the convolutional layer. The modulation parameters in Conditional Batch normalisation (CBN) (Yin al.) 2019 Chen et al., 2019) Ye: and Be are calculated from fo by means of lincar projection layer. The modulation parameters Te: and Be in CBN caleulated follows:
BN Ct, (T 7o) 8- u(x)
+ (B 3c)
(1)
(T)
fG Concal 2)
(2)
%c (fc(
(3)
B. FCelfG)
The generator is trained to minimise adversarial loss (C and multi-modal contrastive loss (C ) LG Cont
is formulated as loss between the features from the generated image and the discriminator-side sentence embeddings. Mathematically, the objective functions can be written as follows:
Cldo Eanpal [-D($)]
(5)
exp (Sim (fuz.
Som (p.. = - -log
(6)
Σ exp SSm (pu
Sim(fo, Sp) cos Sp)
(7)
Here, Sim(...) .) is score function to calculate the similarity between sentence embeddings and image features, cos(u, v) = u²u //||u|||| the Cosine Similarity between features and т denotes the temperature hyper- parameter, and f. represents visual features extracted by the discriminator for the generated image lfake- We use conditioning augmentation (Zhang et al. 7b1 to sample the sentence condition from an independent Gaussian Distribution N (2 (s:), E (s:).. The regularisation term from conditioning augmentation (Cca) for combined sentence embeddings(s) is:
=CA= = (N (N (s)). Es:)) (W(O,
Here ((st) and Ess) are mean and diagonal covaraince matrices that are computed as functions of the com- bined sentence embedding. The regualarisation term is KL Divergence between the Conditioning Gaussian and a Standard Gaussian Distribution. The final loss for generator is defined as:
LG Adv + XıLAA + Gg
5

---

Published in Transactions on Machine Learning Research (06/2025)
3.3 Discriminator
The discriminator D is designed to serve two purposes: (1) to be a critic to determine whether the image is real or fake, and (2) to be a feature encoder to extract image features for multi-modal contrastive loss. The given image (Treal or fake) is passed through a series of downsampling blocks (DounBlocks), until the feature map is size 8 x 8. Next, these 8 x dimensional spatial features are passed through two separate branches: one for extracting features for the adversarial loss and the other for computing image features for multi-modal contrastive loss. For the adversarial branch, the input passed through a DownBlock, ResBlock, and fully connected layer to predict the logit to represent if the given image is real or fake. The predicted logit is used as input to an Adversarial Hinge loss (Miyato et al. 2018 L as follows:
C'd Excepasaa [max (0, 1-11111000 D
(10)
+E Egppa max D()))
Here, x and & are real (Treat) and generated faae)) images.
In the multi-modal contrastive loss branch, the features are passed through a DownBlock, ResBlock, and a lincar projection layer output visual features fu- The multi-modal contrastive loss Cooee takes as input the real image features fv, and sentence embeddings Sp, and calculates the contrastive loss to increase the mutual information in text and image follows:
exp (Sim (Sr, SD,))
C (sr, SD.) = log
(11)
Σラ= exp (Sim (Sr,, SD,))
C is the contrastive loss between real image- text pairs. The final objective function for the Discriminator is defined as:
Ln L'd + da
(12)
3.4 Dual text embedding learning
Embeddings can be viewed as memory representation learned by reducing a loss. Multiple emboddings, each learned by optimising on different losses, will capture various memory representations for the same word. The goal of the dual text embedding setup is to learn generator-side word embeddings We (along with its encoder) to capture complex representation of words to aid improve the photo-realism of the generated images and discriminator-side word embeddings Wp (along with its encoder) to capture distinctive features for words to align text-image associativity. To achieve this, we make sure that We receives only the gradients from image-generatin process whereas Wp receives gradients from the contrastive loss. Specifically, we formulate the generator-side embedding loss (LEm)) and the discriminator-side embedding loss (CPm)) as follows:
C = LG cD X3 Xg
(13)
(14)
Here, La denotes the loss function for the generator G, C. denotes the multi-modal contrastive loss between real image (Treat) features and discriminator-side sentence embedding Sp for the given text T.
4 Experiments
In this section, datasets and evaluation metrics are introduced for experiments. Futher, proposed DTE is evaluated and compared quantitatively and qualitatively with other methods in the literature. The specific training details and hyper-parameters are mentioned in the supplementary material.
Datasets: DTE-GAN evaluated on three datasots, namely, 1) Caltoch-UCSS birds (CUB) (Welineer et al. 2010 2) Oxford-11 102 flowers (Nilsback Zisserman| 2008),, and 3) MS COCO (Ln et 2014b
datasets. For CUB and Oxford-102 datasets, have similar setup to StackGAN Zhang et al. 2017b

---

Published in Transactions on Machine Learning Research (06/2025)
CAPTION
REAL
DF-GAN
DTE-GAN
CAPTION
DF-GAN
DTE-GAN
CAPTION
DF-GAN
DTE-GAN
this bird has a white belly, breast, gray orange
primarily a light
brown with
black black red
black markings head
Many horee running acrons plain
of pie sits on white piets.
sits at the and the streat.
Figure Visual comparision of the images generated by DF-GAN
(Tao et al. 2022 and DTE-GAN on
CUB Welinder et al. 2010 and Lin et al. 2014a Datasets.
CAPTION
REAL
HDGAN
DTE-GAN
CAPTION
REAL
HDGAN
DTE-GAN
CAPTION
REAL
HDGAN
DTE-GAN
this flower han bright petats with smooth tecture.
petais shaced like
perpal sepel
stalka of grass
pointed srigmm
surround
brown
Figure 4: Illustaaioon of the images generated by HDGAN Oxford-100 102 Flower Dataset [Nilsback & Zisserman 2008
ZZang et al. 2018) and those of DTE-GAN on
Ten captions are provided for each image in both the datasets (Reed et al. 2016a). The MS-COCO dataset consists of around 80k training and 40k validation images; and for every image, there are captions provided with the dataset.
CUB
COCO
Method
IS f
FID +
R% f
FID 4
R% 1
NoP +
StackGAN
Zhang et al.
2017b
3.70 + .04
AttnGAN
Xu et al. 2018)
4.36
23.98
67.82
35.49
83.82
230M
MirrorGAN iQia
al. 2019b
4.76
18.32
57.67
34.71
74.53
DM-GAN Zhu et al. 2019
4.7
16.09
72.32
32.64
88.56
46M
KT-GAN Tan et al. 2021
4.85
17.32
30.73
TIME Liu et al. 1
4.91
14.30
71.57
31.14
120M
DAE-GAN Ruan al.
2021
4.4
15.19
85.45
28.12
92.61
98M
CSM-GAN (Tan al. 2022a
4.6
20.18
33.48
DR-GAN TTan et al. 2023a
4.90
14.96
27.80
73M
ALR-GAN (Tan el al. 2023b)
4.96 4.04
15.14
77.54
29.04
69.20
76M
DF-GAN Tao et al. 2022
4.86 1.6
14.81
19.32
19M
SSA-GAN Liao et al.
2022
5.17 .08
15.61
85.4
19.37
90.6
26M
DTE-GAN
5.12 004
13.67
86.64
25.17
90.82
11M
DTE-GAN+MAGP
5.09 + .02
13.94
81.33
19.69
88.39
11M
Table 1: Quantitative comparison between DTE-GAN and other models on CUB WWelinder et al. 2010 and COCO (Lin et al. 2014a) datasets. indicates values are unreported. The best three results are marked with +underline bold, and underline, respectively. "4" indicates the higher, the better, while "1" indicates the lower, the better.
4.1 Visual comparison
The generated images are visually compared between DF-GAN (Tao ct al. 2020 and DTE-GAN the CUB and COCO datasets. In Figure 3 it evident that OF-GAN struggles depict complete bird shapes. The presented model, employing dual text embeddings, minimizes image generation loss, improving shape accu-

---

Published in Transactions on Machine Learning Research (06/2025)
racy and realistic fine-grained features in generated images. Furthermore, DTE-GAN outperforms DF-GAN in pose representation, resulting in more natural-loking images. With regard to semantic consistency be- tween images and text, DTE-GAN captures detailed structures and overall coherence compared to DF-GAN. Despite not using word embeddings for image region attention, DTE-GAN's ability to learn embeddings for both generation and discrimination allows it lo generate images with finer details. DTE-GAN produces images that resemble real images due to its learned embeddings encompassing generation and discrimination aspects. For the COCO dataset, DTE-GAN generates images of comparable quality to DI-GAN while using significantly fewer parameters. This efficiency is achieved by learning tailored text embeddings specifically for the synthesis model. In Figure 4 images for the Oxford-102 dataset are generated and compared with those of HDGAN (Zhang et al. 2018). We can observe that our model is able to capture the complex variation of the flowers and generate more realistic images than HDGAN.
Method
IS f
FID
+
SA (Zhang et al. 2017b)
3.20 + .01
51.89
StackGAN+ ++ (hangg al.
2017a
3.26 + .01
48.68
HDGAN (Zhang et al. 2018
3.45
43.17
SSTIS (Tan et al. 20226
3.37 .05
SS-TiGAN (Tan et al. 2023c
3.45 + .04
40.54
GAN (Cai et al. 2019
4.06 + .05
40.31
RAT-GAN (Ye et al. 2024
4.09
DTE-GAN DTE-GAN+MAGP
4.21 + .08
30.07
4.26 + .07
31.13
Table 2: Quantitative comparison between DTE-GAN and other models on Oxford-102 Dataset. The best three results are marked with undelin,,,,,ne, bold, and underline, respectively. "f" indicates the higher, the better, while " =1" indicates the lower, the better.' '_" indicates values are unreported.
4.2 Quantitative Evaluation
Evaluation metrics: To evaluate the quality of images generated, the following metrics used are: Inception
Score (IS) (Salimans et al. 2016) calculates the Kullback-Leibler (KL) divergence between conditional
distribution and marginal distribution for class probabilitics from Inception-v3 (Szegedy et al., 2016) model.
Higher IS suggest generated images are from higher diverse classes. Fréchel Inceplion Dislance (FID) HHuueel
et al. 2017) calculates the Fréchet Distance between two multivariate Gaussians, which are fit to the global features extracted from the Inception-v3 (Szegedy et al. 2016) model on the synthetic and generated images. Lower the FID suggest generated images closer to real images. R-precision (R%) precision evaluates texi- to-image alignment, by assessing whether generated images can be used to retrieve the text.
The performance of proposed model is compared with that of the lightweight GAN approaches (having similar training setups) for the task of text-to-inae to-inage- synthesis CUB and COCO datasets in Table ] From Table on the CUB dataset, we observe that DTF-GAN improves IS from 4.91 to 5.12, achieves the best R- precision of 86.64 and further decreases FID from 14.06 to 13.67. We also train our DTE-GAN with the proposed regularisation trick Matchingg-war Gradient Penalty (MAGP) (Tao et all 2022). which smooths out discriminator function and allows to generate more realistic images. On the CUB dataset, compared to AttnGAN (Xu et al., 20188) that employs contrastive loss-based embeddings, our model docreases FID from 23.98 to 13.67. Such an improvement illustrates the effectiveness of the end-to-end learned dual embeddings over fixed pre-trained embeddings learned on the same data. On MS-COCO (Lin et al. 2014b) dataset (in Table 1. we achieve similar performance of DF-GAN and SSA-GAN with fewer parameters, underscoring the efficiency of the proposed DTE and its ability to learn embeddings tailored specifically to the synthesis model.
achieves similar performance as that of SSA-GAN (Liao et al. 2022 on COCO dataset as SSA-GAN and DF-GAN Tao et al. 2022) both incorporating MAGP.
As shown in the Table D we have employed a network with significantly fewer parameters by reducing the width of the layers by half compared to SSA-GAN Liao et. al. (2022) and DF-GAN Tao et al. (020)) in their
8

---

Published in Transactions on Machine Learning Research (06/2025)
CAPTION
REAL
CLIP
CLIP Gamb
DTE
CAPTION
REAL
CLIP
CLIP
emb
DTE
this flower is yellow color, with petals that are skinny and oval shaped.
this flower has yellow petals red spots them, and red on the border each petal.
bird very long pointy bill orange color ,hat has green crown body
small with elight grey in coverts, secondaries and primaries.
Figure 5: Images generated using CLIP, CLIP + Learnable Generator side embeddinigs (CLIP + Gemb) and DTE on CUB and Oxford-102 datasets.
respective UpBlock and DownBlock. Despite this reduction in complexity, our model achieves comparable results by learning more effective text encoding representatious, which improve the performance of Text-to- Image synthosis
Following previous works TTa et al.
2022 Zhang & Schomaker 2020 Liao el. al. 2022 we report only
FID scores, as IS scores for the MS- COCO dataset do not reflect the quality of the synthesised images. In comparison to TIME (Liu et al.] 2021 which Iearns embeddings along with the Text-to- Image synthesis model, OTE-GAN achieves significant improvement (0.6 in FID, +15% in R-precision) demonstrating the effectiveness of dual text embeddings. In Table 2 on the Oxford-109 dataset, we use IS and FTD scores for evaluation, R-precision scores are available in the literature. In this dataset, our model improves IS score from 4.06 to 4.21 over the state-of-the-art (DualAttn-GAN (Cai et al. 2019 LeicaGAN (Qiao et al. 2019ah) models and remarkably decreases FID from 40.31 to 30.07.
4.3 Additional Studies
Dataset
Embeddings
IS 1
FID +
R % f
CLIP
4.53
21.33
74.12
CUB
CLIP+
4.51
19.36
78.35
DTE
5.12
13.67
86.64
CLIP
3.72
38.36
71.78
Oxford
CLIP+Geml
3.81
35.93
73.87
DTE
4.21
30.07
83.19
Table 3: We compare quality of '21 generation of our proposed DTE approach with that of models trained using CLIP (Raforrd et al. 2021) and CLIP+G.mb. "1" indicates the higher, the better, while "J" indicates the lower, the better.
4.3.1 DTE vs CLIP:
We compare DTE with pre-trained CLIP embeddings by training a GAN for text-to-image generation. Additionally, we train another GAN model resembling the DTE setup. This model uses learnable Generator- side embeddings and pre-trained CLIP embeddings for the discriminator (referred as CLIP+ Gemb). Table compares image quality using different CUB and Oxford-100 1022 datasets embeddings. CLIP+Gens. improves over just CLIP. In Figure 5 CLIP images differ from reality, while CLIP+Gemt matches better text and real images due to Icarned generative embeddings. DTE's combined approach creates images closer to real images. Furthermore, we demonstrate that this approach can be integrated with pre-trained vision-language models, specifically CLIP+Gemb where only the generator-side embeddings are learned. This method of focusing solcly on generator-side embeddings can be soamlessly incorporated into current diffusion-baser Text-to-Imagge models (Rombach al. 2021 Ramesh al.] (002)) within the existing training frameworks and emphasises learning embedding needed for synthesis model.

---

Published in Transactions on Machine Learning Research (06/2025)
4.3.2 Generalisation ability of DTE:
To assess how well the DTE setup applies to other architectures, we integrate into AttnGAN Xu et all 2018), now called ATTE In AttnGAN, pre-trained text embeddings (DAMSM embeddings) that are generic is used training synthesis model. Instead, we train embeddings scratch using DTE. Incorporating DTE into AttnGAN involves modifying its discriminators to include a dual branch (adversarial loss and multi- modal contrastive loss) after reaching 8x8 feature size. As AttGAN uses words for aligmment in the attention layer, we introduce a word-contrastive loss (Xu et al. 2018: "Zhang et al. 2021) in the final discriminator as an additional branch when the feature size is 8 x 8, aimed at reducing the semantic gap between words and image features. We combine generator-side and discriminator-sion word embeddings to provide word features for the Generator's attention mechanism. The results Table A show that AttnGAN+DTE, can train without pre-trained embeddings and even improve the results, proving that DTE can be integrated well with other methods.
Method
IS f
FID
R%
AttnGAN
4.36 = 002
23.98
67.82
AttnGAN+DTE
4.38 = 003
21.45
71.39
DTE-GAN
5.12 .04
13.67
86.64
Table 4: Impact of DTE approach on AttnGAN (uu et al.
2018) CUB Datasel (Welinder et al. 2010
"1" indicates the higher, the better, while "I" indicates the lower, the better.
Captions
this bird bright blue crown as orange eyering
bird grey crown, grey primaries, and white belly.
the bird of speckled pattern of black, grey, tan feathers
small pudgy bird with red tones all over, black tipped wings
small bird with yellow throat, breast and belly, and brown and yellow secondaries.
this bird has brown crown that is spotted, belly and breast that is white
Source
Concept
Manipulated
Figure 6: Examples manipulated images generated by Light GAN (Li al. 2020b) using DTE- GAN pre-trained embeddings on CUB dataset. Source images are manipulated by the caption of concept images.
4.3.3 Application to Text-to-lmage manipulation task:
To demonstrate the versatility of the learned dual embeddings, we apply them to text-to-image manipulation tasks. We train dual text emboddings through DTE-GAN on the CUB datasot for text-to-image synthe- sis. We then utilise the pre-trained word embeddings (Wa, Wp) from this synthesis task for text-to-image manipulation in the Lightweight GAN for Text-to-Image manipulations (Li et al. 2020b) task the same
10

---

Published in Transactions on Machine Learning Research (06/2025)
Method
IS t
FID
MP f
MANIGAN
8.48
9.75
0.072
LWGAN
8.26
8.02
LWGAN w/ DTE-EMB
8.56
7.77
0.075
Table 5: Quantitative comparison of Iuception score, FID and Manipulative metric (MP) (Li el. al. 2020a for manipulated images on CUB dataset. We use Lightweight GAN (Li et al. 2020b) which we name LWGAN with the pre-trained emboddings using DTE-GAN (LWGAN w/ EMB). The best results are red. "1" indicates the higher, the better, while "4" indicates the lower, the better." '-" indicates values are unreported.
datasct. is important to note that these pre-trained dual text embeddings remain fixed during training of manipulation work. In Table the quantitalive performance of the model using these pre-trained embed- dings is compared with that of other text-to- text-to-image manipulation models. The model improves the Inception score from 8.48 (MANIGAN (Li et al. 2020aa) ) to 8.56 and reduces the FID score from 8.02 (Li et al. 2020bb to 7.77 on the CUB dataset. This demonstrates the ability of dual text embeddings to generalise effectively across different tasks. Figure Ö illustrates few visual examples of the text-to-image manipulations.
Emb.
Components
CUB
Oxford-102
type
LG
CD
Se +
IS f
FID
R% t
IS t
FID 4
R% f
Shared
X
4.54 + .04
15.81
85.63
3.52 + .06
33.57
81.73
Shared
/
4.27 + .06
18.38
82.73
3.32 + .05
34.83
76.15
Dual
/
14.93
63.79
3.84 .03
32.98
54.97
Dual
/ /
4.25 + .05
18.01
69.38
3.28 + .04
35.41
70.48
Dual
5.12 + .04
13.67
86.64
4.21
.08
30.07
83.19
Table 6: Quantitative comparison of DTE with its variants. Here, La = generalorr loss, C.me = multi- modal contrastive loss between real image text pairs, SD - whether the generator has access to the discriminator-side sentence embedding, Sc - D whether the discriminator has access the generator- side sentence embedding. "1" indicates the higher, the better, while " indicates the lower, the better.
4.3.4 Importance of dual text embeddings
To verify the effectiveness of the proposed Dual Text Embeddings (DTE) setup, different ways of organising the word embeddings between generator G and discriminator D are evaluated on CUB and Oxford-109 102 datasets and results shown in Table Specifically, four variants of organising the embeddings are compared, namely: i) A shared word embedding layer between G and D that trained only with multi-modal contrastive loss CD between real image-text pairs it is trained only to capture distinctive features (Table row 1). 4) A shared word embedding layer between G and D that is trained using both the generator loss LG and real image-text. pair contrastive loss CD to capture distinctive and intricate appearance features in single embeddings (Table 6 row 2). dit) A dual embedding setup where G doesn't 't have access to the discriminator-sidee sentence embedding Sp (TTable G row 3). iv) A dual embedding setup where both G and D have access to the generator-side sentence embeddings Sg and discriminator-side sentence embeddings Sp (Table 6 row 4). The shared embedding model trained only with C to capture distinctive features (Table row achieves similar R-precision scores as those of the proposed DTEE-AN, but there is significant drop in IS and FID scores suggesting that there is drop in the quality of generated images. Next, the shared embedding model trained with both La and C performs inferior to the one trained only with C It proves that capturing generator-side noisy gradient signals into the same word embeddings degrades the performance. Further, the dual embedding model with independent generator- and discriminator-siide embeddings achieves better IS and FID scores compared to those of shared embedding models but has a significant drop in R-precision, suggesting that the images generated are realistic but do not capture text- to-image alignment. Further, allowing the discriminator significantly drops the performance (Table B row 4). As the Discriminator-side embeddings learned with
have access to the generator-side embeddings
11

---

Published in Transactions on Machine Learning Research (06/2025)
Epoch
Components
CUB
Oxford-100
Count
SD ++G
SG
IS
FID J
R%
IS 1
FID
R% 1
0
4.21 + .04
18.33
69.81
3.34 + .05
36.18
71.13
100
4.59 = .06
16.24
74.46
3.56
.05
33.27
75.68
300
4.82 + 044
14.96
78.57
3.91
.04
31.38
78.92
Table Quantitative comparison of of (Sc - D) whether the discriminator has access to the generator-side sentence embedding from the epoch count. SD - G represents generator has access the discriminator-side sentence embedding. "f" indicates the higher, the better, while "4" indicates the lower, the better.
ground-truth real image-text pairs, it is found beneficial to allow Generator to have access (ú.e., sneak peek) to Discriminator-side embeddings (Table 6 row 5). Diseriminator-side embeddings capture distinct features and Generator-side feature captures intricate details to improve photo-realism; providing both the information to Generator allows to generate more realistic and text-aligned images. On the other hand, Generator-side embeddings are learned using noisy gradients from fake images and allowing the discriminator access them introduces an adverse effect and decrcases the performance (Table row 4).
4.3.5 Generator-side embeddings to Discriminator
When generator-side embeddings are provided to the discriminator, we observe significant drop in overall image generation quality, as reported in Table G To further evaluate this effect, we have conducted an experiment in which generator-side embeddings are supplied to the discriminator after a specified number of training epochs, with the results presented in Table 7 Specifically, generalor-side Sg embeddings are combined with Sp using summation from the start of training (epoch 0), after 100 epochs when the generator began producing images with plausible structure, and after 300 epochs when more realistic images were generated. The low noisy quality of generated images during the initial stages affects the learning of generator-side embeddings; consequently, providing these embeddings to the discriminator negatively impacts the overall image generation quality.
Epoch Count
loss
CUB
Oxford-102
cont
LG
IS
FID J
R%
IS f
FID
R. +
0
4.27 + .06
18.38
82.73
3.32 + .05
34.83
76.15
100
4.38 + .04
16.82
83.72
3.39
.04
34.04
78.69
300
4.47 +: .05
16.23
84.51
3.47
.03
33.96
79.94
4.54 + .04
15.81
85.63
3.52
.06
33.57
81.73
Table 8: Quantitative comparison of shared embeddings trained with contrastive loss (CPont and Generator's loss (LG) after the epoch count. Single emboddings without LG training achieves superior performance. "ga1 indicates the higher, the better, while "I" indicates the lower, the better.
4.3.6 Shared Embeddings with Dual Loss
As shown in Table 6 shared embeddings (or single embeddings for text) trained with both contrastive loss CDnt and the generator's loss La yield subpar results compared to embeddings trained solely with contrastive loss. We hypothesise that this is because, during the early stages of GAN training, the generator produces prodominantly noisy images, negatively impacting the learning of embeddings. To investigate the impact of the generator's loss, we trained multiple nelworks with shared embeddings and applied the generalor's loss lo these embeddings at different training stages. The results, reported in Table [ indicate that applying both losses from the carly stages training adversely affects image generation quality. Morcover, decoupling the embeddings introduces flexibility, enabling them to capture diverse representations, ultimately improving overall image generation quality.
12

---

Published in Transactions on Machine Learning Research (06/2025)
Epoch
CUB
Oxford-102
Count
IS
FID
R%
IS t
FID
R% 1
Single
5.12 = .04
13.67
86.64
4.21 + .08
30.07
83.19
Dual
5.14 + .04
13.37
87.12
4.28 + .09
29.71
83.38
Table 9: Quantitative comparison of shared embeddings trained with single and dual captions. Single embeddings with out LG training achieves superior performance. "f" indicates the higher, the better, while "4" indicates the lower, the better.
4.3.7 Dual Captions for Dual Text Encoders
Farlier approaches Text-to-Image synthesis have employed dual (in et al. 2019) or multiple captions (Cheng al., 2020) enhance semantie consistency and facilitate richer feature extraction. To analyse the impact of using different captions in dual text encoder selup, we have conducted experiments and reported the results in Table 9 Consistent with previous findings, results show marginal improvement in performance. This improvement is attributed to the ability of each text encoder to independently extract superior features, thereby reinforcing the overall semantic alignment and image quality.
5 Limitation and Future Scope
This work proposes an approach for learning vision language models through image generation. Due to computational constraints, our model is trained on a single NVIDIA 1080Ti GPU with 12 GB of memory, we limit our experiments to smaller datasets. However, we refer to a comparison with large-scale models models (Ho et al. 2020 Dhariwal & Nichol, 2021) publicly available large-scale datasets such as WebVid-1OM and DiffusionDB (Bain et al.) 2021; Wang et al. 2022). Recent advances also highlight the potential of hybrid models that combine GANs with iituusion-basedd training. In this direction, we intend to investigate the integration of our DTE setup [Dieriminator--Teadl Teacher Embedding) into large-scale GANs and hybrid models, where the generator trained using both GANs and diffusion losses (Xu et al. 2024 Sauer et al. 2024). The DTE mechanism can be directly incorporated into such architectures to enhance training efficiency and alignment.
Table 10
As part of future work, we plan to explore vision language learning using diffusion-
Model
Type
FID |
NoP 1
DALLE Ramesh et al.
2021
AutoRegressive
27.5
12B
LAFITE Zhoou et al. 2021
GAN
26.94
75M
Stable Diffusion (Rombach et al.
2021
Diffusion
9.62
0.9B
DALLE 2 (Ramesh al. 2022
Diffusion
10.39
5.5B
Make-A-Soe A-Scene Galni et a. 2022
AutoRegresse
11.84
StyleGAN-T (Sauer et al. 2023
GAN
13.90
1B
GigaGAN (Kang et al. 2023)
GAN
9.09
1B
UFOGen (Xu et al. 2024
Hybrid
12.78
0.9B
DTE-GAN (Ours)
GAN
19.69
11M
Table 10: Quantitative comparison with large-scale models FID and Number of Parameters (NoP) on COCO dataset (Lin el al. 2014a). 1_ indicates values are unreported.
6 Conclusion
This study introduces a novel approach to text-to-image synthesis by proposing Dual Text Embeddings (DTE), which learns text embeddings tailored explicitly for the synthesis task an end-to-end manner. Unlike traditional methods that depend on pre-trained, generic embeddings, DTE uses two separate embed- dings designed for specific purposes: one to improve the photo-realism of generated images and the other to
13

---

Published in Transactions on Machine Learning Research (06/2025)
ensure better alignment between text and images. DTE decouples these objectives with dedicated training techniques, leading to enhanced performance.
Our experiments on three benchmark datasets (Oxford-102, Caltech-UCSD, and MS-COCO) demonstrate that having separate embeddings yields better results than using a shared representation. Furthermore, the DTE framework performs favourably compared to methods relying on pre-trained text embeddings optimized with contrastive loss. Additionally, their versatility highlights the adaptability of learned dual embeddings to other language-base based vision tasks, such text-to-image manipulations.
Future work includes extending this dual embedding framework to other multimodal applications, including image or video captioning and visual question answering, further showcasing the potential of task-specific text embeddings in advancing language-vision interactions.
6.1 Broader Impact Statement
The proposed Dual Text Emneddings paradigm introduces new approach where text encoding schemes capture the perspective of current vision-language models and are also trained to capture generators per- spective. This approach enables lcarning text representations that enhance the image synthesis capabilities of conditional generation models and also provides new paradigm learning representation by generating images.
References
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan, 2017.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision, 2021.
Andrew Brock, Theodore Lim, M. Ritchie, and Nick Weston. Neural photo editing with introspective adversarial networks, 2017.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis, 2019.
Yali Cai, Xiaoru Wang, Zhihong Yu, Fu Li, Peirong Xu, Yueli Li, and Lixian Li. Dualattn-gan: Text to image synthesis with dual attentional generative adversarial network. IEEE Access, 1183706-1187166 2019, doi: 10.1109/ACCESS 2019.2958864
Ting Chen, Mario Lučié, Neil Houlsby, and Sylvain Gelly. On self-modulation for generative adversarial networks. In International Conference on Learning Representations, 2019. URL https://arxiv..rgg
11110000165..
Jun Cheng, Fuxiang Wu, Yanling Tian, Lei Wang, and Dapeng Tao. Rifegan: Rich feature generation for text-to-image synthesis from prior knowledge. In 2020 CCEECVV Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10908- 10917, 2020. doi: LLLL:///CVPA2200022000.... 200900.
Ayushman Dash, John Cristian Borges Gamboa, Sheraz Ahmed, Marcus Liwicki, and Muhammad Zeshan Afzal. Tac-gan- text conditioned auxiliary classifier generative adversarial network, 2017.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis.
A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Informa- tion Processing Systems, 2021. URL htttps://openrevies www.:nttt/ooummddAdmoooommm
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. Cogview: Mastering text-to-image generation via transformers. In M. Ranzato, A. Beygelzimer, Dauphin, P.S. Liang, and J. Wortman Vaughan (ods.), Advances in Neural Information Processing Systems, pp. 19822-1111 19835. Curran Associates, Inc., 2021.
14

---

Published in Transactions on Machine Learning Research (06/2025)
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-rcsohution image synthesis. http:iii.rrr/abb//22.009232
CoRR, abs/2012.09841,, 2020. URL
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene:
Scene-based text-to-image generation with human priors, 2022. URL https:://rxxv..rg///aa wwtp:bs////ab.//////
8131
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27, 2014.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-lo-image synthesis. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10696-10706, June 2022.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems. Curran Associates, Inc., 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (ods.), Advances in Neural Information Processing Systems, pp. 6840 6851. Curran Associates, Inc., 2020.
Minguk Kang, Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition IentttionnCCPP (CVPR), pp. 10124-10134, June 2023.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lchtinen. Progressive growing of gans for improved quality, stability, and variation, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVE CCVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. /ttp///tpp//aaaw.orr///aa///
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). wwwp:/ssssttoooooooooooooomoooo. URL http:://ww. cs, toronto.... www.oto..euo-krrttt.oo www. html
Yann LeCun and Corinna Cortes. MNIST handwritten digit databasc. 2010. URL httt:://yan. hteeexddmmmmit
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11523 11532, June 2022.
Bowen Li, Xiaojuan Qi. Thomas Lukasiewicz, and Philip Torr. Controllable text-to-image generation. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Aleh E. Fox, and R. Garnett (eds.), Advances in Newral Information Processing Systems. Curran Associates, Inc., 2019.
15

---

Published in Transactions on Machine Learning Research (06/2025)
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip H.S. Torr. Manigan: Text-guided image manipula- tion. In Proceedings of the TEEE/CVE Conference on Computer Vision and Poltern Recognation (CVPR), June 2020a.
Bowen Li, Xiaojuan Qi, Philip Torr, and Thomas Lukasiewicz. Lightweight generative adversarial networks for text-guided image manipulation. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, pp. 22020 22031. Curran Associates, Inc., 2020b.
Jiadong Liang, Wenjie Pei, and Feng Lu. Opgan: Content-parsing generative adversarial networks for text- to-image synthesis. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.); Computer Vision ECCV 2020, pp. 491-508, Cham, 2020. Springer International Publishing. ISBN
978-3-030-585418-
Wentong Liao, Kai Hu, Michael Ying Yang, and Bodo Rosenhahn. Text to image generation with semantic- spatial aware gan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recog- nition (CVPR), pp. 18187-18196, June 2022.
Tsung-Yi Yi Lin, Michacl Maire, Serge Belongic, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014, 2014a.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongic, James Hays, Pictro Perona, Dova Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014, pp. 740-755, Cham, 2014b. Springer International Publishing. ISBN 978-3-319-106022
Bingchen Liu, Kumpeng Song, Yizhe Zhu, Gerard de Melo, and Ahmed Elgammal. Time: Text and image trtuall-rannlltionn adversarial networks. Proceedings of the AAAI Conference on Artificial Intelligence, 35(3):2082-22) 2090, May 2021. URL. httpss///oss aaai .aai..rg/ /eng/indx..... /ttp:/aaa...aaaaaaaaaaaa////a/aa
Andrew Maas, Awni Y. Hanmun, and Andrew Y. Ng. Rectifier nonlincarities improve neural network acoustic models. In in ICML Workshop on Deep Learning for Audio, Speech and Language Processing, 2013.
Xudong Mao, Qing Li, Haoran Xie, Raymond Y.K. Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets, 2014.
Takeru Miyato and Masanori Koyama. egans with projection discriminator, 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for gen-
erative adversarial networks. In International Conference Learning Representations, 2018. URL
https://openreevier wwtp://wwww......ooooooooo/ddeee
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models, 2021. URL https: //axxiv.orr/abb/2122...g
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary clas- sifier GANs. In Doina Precup and Yee Whye Tch (ods.), Proceedings of the 94th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 2642-2651 PMLR, 06-11 Aug 2017.
16

---

Published in Transactions on Machine Learning Research (06/2025)
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d' d'Aché-Buu,, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. Learn, imagine and create: Text-to-image generation from prior knowledge. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Newral Information Processing Systems. Curran Associales, Inc., 2019a.
Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. Mirrorgan: Learning text-to-image gener- ation by redescription. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019b.
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URI. http://////aavi.org/aa//////////
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and llya Sutskever. Zero-shot text-to-image generation. CoRR, abs/ abs/2102.12092, 2021. URL https://arxiv.org
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents, 2022. URL https://arxiv. wwt.//////////////ee. 0albb//224.00111
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vaevae-2. In H. Wallach, H. Larochelle, A. Beygelzimer, F. Alchh-BBu,, E. Fox, and R. Garnett (eds.), Advances in Newral Information Processing Systems. Curran Associates, Inc., 2019.
Scott Reed, Zeynep Akata, Honglak Lec, and Bernt Schicle. Learning deep representations of fino-grained visual descriptions. In Proceedings of the IEEE conference on compuler vision and pallern recognilion, pp. 49-58, 2016a.
Scott Reed, Zeynep Akata, Santosh Mohan, Samuel Tenka, Bernt Schiele, and Honglak Tee. Learning what and where to draw, 2016b.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1060-10000 1069. PMLR, 2016c.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2021.
Shulan Ruan, Yong Zhang, Kun Zhang, Yanbo Fan, Fan Tang, Qi Liu, and Enhong Chen. Dae-gan: Dynamic aspect-aware gan for text-to- text-to-image image synthesis. In Proceedings of the IEEE/CVE /CVF International Conference on Computer Vision (ICCV), pp. 13960-13969, October 2021.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala. Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. 2022. URL
11487
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS'16, pp. 2234-2242, Rod Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.
17

---

Published in Transactions on Machine Learning Research (06/2025)
Axel Saner, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis. volume abs/2301.09515,, 2023. URL https://arxii: org/ http//bb/2221..000
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. European Conference on Computer Vision, pp. 87-103. Springer, 2024.
M. Schuster and K.K. Paliwal. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11):2673 2681, 1997. doi: 10.1109/78.650093
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2250222522222 2265, Lille, France, 07-09 Jul 2015. PMLR. URL https://prrceeddnng mlr. www.pprss/v33//e.emm sohl -dickstein15.htm
Christian Szegedy, Vincent Vanhoucke, Sergey lofle, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 2828-2826, 2016.
Hongehen Tan, Xiuping Liu, Mong Liu, Baocai Yin, and Xin Li. Kt-gan: Knowledge-trausfer generative adversarial network for lext-to-imaae synthesis. IEEE Transactions on Image Processing, 30:1275 1290,, 2021. doi: 10.1109/TTI 20200//TIP220000000 NTP.220003026722
Hongchen Tan, Xiuping Liu, Baocai Yin, and Xin Li. Cross-moda semantic matching generative adversarial synthesis. IEEE Transactions on Multimedia, 24:832-844 845, 2022a. doi: 10.110
networks for
9/TMMM20021.00002991
Hongchen Tan, Xiuping Liu, Baocai Yin, and Xin Li. Dr-gan: Distribution regularization for text-to-image gencration. IEEE Transactions on Neural Networks and Learning Systems, 31112)110009--032,, 2023a. doi: 20t./NNNNSS.202.2.3112222
Hongehen Tan, Baocai Yin, Kun Wei, Xiuping Liu, and Xin Li. Alr-gan: Adaptive layout refinement for text-to-image synthesis. IEEE Transactions on Multimedia, 25:8620 8631, 2023b. doi: 10.1109/TMM.20 23.3238554.
Yong Xuan Tan, Chin Poo Lee, Mai Neo, and Kian Ming Lim. Text-to-image synthesis with self-supervised learning. Pollern Recognilion Letters, 157:119- 126, 2022b. ISSN 0167-8655 doi: httpss//doi.oor//00.. p1pparrc.2022.04.000. URL
D1064
Yong Xuan Tan, Chin Poo Lee, Mai Neo, Kian Ming Lim, and Jit Yan Lim. Text-to-image synthesis with self-supervised bi-stage generative adversarial network. Pattern Recognátion Letters, 169:43 -9, 2023c.
ISSN 0167-8655, doi: http:://toi..rg/ll..10////// com/ catp://caaaaeertttaaaaa///676660
pat.e....................2 URL https::/www.scieecee
Ming Tao, Songsong Wu, Xiaofeng Zhang, and Cailing Wang. Defgan: Dynamic convolutional fusion gener- ative adversarial network for text-to-image synthesis. pp. 1250-1254, 11 2020. doi: 10.1109/ICIBA50661 2020.9277299.
Ming Tao, Hao Tang, Fei Wu, Xiao Yuan Jing, Bing-Kun Bao, and Changsheng Xu. Df-gan: A simple and effective baseline for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition (CVPR), pp. 16515 16525, June 2022.
Ming Tao, Bing-Kun Kun Bao, Hao Tang, and Changsheng Xu. Galip: Generative adversarial clips for text-to- image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 14214-14223 2023.
18

---

Published in Transactions on Machine Learning Research (06/2025)
Aaron van den Oord, Oriol Vinyals, and koray kavukeuoglu. Neural discrete representation learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.); Advances in Newral Information Pacioonrrrrrre Processing Systems. Curran Associates, Inc., 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Tnc., 2017. URL httpps:/prroeeedings meweeidddnnnnnnnin....ee...... paarrppss.cppppee/00////// caaper.ddt.
Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. DiffusionDB: A large-scale prompt gallery dataset for text-to-image generative models. arXic:2210.1480
[es], 2022. URL htttps://axxiiii http//axxvvoooraaaa/ wwwrg/abs/22100/ 14896
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report 2NS-00-2010-001, California Institute of Technology, 2010.
Tao Xu, Pengehuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attugan: Fine-grained text to image generation with attentional generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Yanwu Xu, Yang Zhao, Zhisheng Xiao, and Tingbo Hou. Ufogen: You forward once large scale text-to- image generation via diffusion gans. In 2024 IEEE/CVF Conference Compuler Vision and Pallern Recognition (CVPR), pp. 8196 82668206, 2024.
doi: (00000/CCRRR2333.10
Senmao Ye, Huan Wang, Mingkui Tan, and Fei Liu. Recurrent affine transformation for text-to-image synthesis. IEFE Transactions on Multimedia, 26:462-473, 2024. doi:
Guojun Yin, Bin Liu, Lu Sheng, Nenghai Yu, Xiaogang Wang, and Jing Shao. Semantics disentangling for text-to-image gencration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris Melaxas. Stackgan++: Realistic image synthesis with stacked generative adversarial networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, PP, 10 2017a. doi: 10.1109/TPAMI N.M..2018.2866251
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N. Metaxas. Stackgan: Text to reoto-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEF International Conference Computer Vision (ICCV), Oct 2017b.
Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-modal rastivee learning for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 833-842, June 2021.
Zhenxing Zhang and Lambert Schomaker. DTGAN: dual attention generative adversarial networks for text-to-image generation. CoRR, /bs//201.02709, 2020. URL http:://aarxi..or/////////1.111/
Zizhao Zhang, Yuanpu Xie, and Lin Yang, Photographic text-to-image synthesis with hierarchically-nnsteed adversarial network. In Proceedings of the IEEE Conference Compuler Vision and Pollern Recognation (CVPR), June 2018.
Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. Lafite: Towards languago-free training for text-to-image generation. arXie preprint wrXiv:2111.13790 2021.
Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative adversarial networks for text-to- -maaee synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
19

---

Published in Transactions on Machine Learning Research (06/2025)
A DTE-GAN with MA-GP
2022 have significantly improved quality of synthesised images on COCO dataset [Lin et al. 2014b). Their improved performance may be associated with the Matching- Aware zero-centered Gradient Penalty (MA-GP) term adopted in these methods. MA-GP is applied to Discriminator with real images and its corresponding text to smooth the loss lunction that allows the Generator to synthesise more realistic images, incorporating MA-GP into DTE-GAN by changing the Discriminator to conditional Discriminator (towards one-way output for gradient penalty). Mismatch pairs not used DTE-GAN has multi-modal contrastive branch for text-image alignment. For conditional prediction, following similar setup of DF-GAN (Tao et al. 2022 of replicating sentence features and concalenating with image features to predict logit values for adversarial loss, the modified adversarial loss function of Discriminator for conditional loss with MA-GP is:
Recent methods (Tao et al. 2022 Zhang & Schomaker 2020 Liao et al.
L Ex [max 1- Sn)))
+ Es-Po [max(0, D(2,
(15)
kBg-Paan
+ + Sp)))"
Here, k and p prper-parameters (we use the same hyper-parametar values from DF-GAN (Tao et al, 2022 For training discriminator-side word embeddings and their sentence encoder from real image-text pairs, we do not update the weights using the gradient of fake conditional prediction from the adversarial loss. Conditional Adversarial loss for Generator is:
Adv = Ri-ro (D(,, L.
-
(16)
The final objective function for the Generator and Discriminator is defined as:
LG L + >iLCs + X2
(17)
Ln GAN + *2g.
(18)
B Text Encoding Scheme
(997) for text encoding, following the popular DAMSM Xu et al. (2018) embeddings commonly employed in lightweight GAN models Xu et al. (2018): 2019); Liao et al. 2022 Tao et al. (2020) The DAMSM embeddings are trained to learn Zhu et al.
We have used single-stage Bi-LSTM Schuster & Paliwal
discriminative features by distinguishing between instances, ensuring fair comparison focused on design principles rather than simply increasing the number of parameters. Additionally, we have conducted an ablation study by replacing the Bi-LSTM (Schuster Paliwal 1997 with a 4-layer Transformer encoder Vaswani al. (2017) in both the generator and discriminator text encoders, and we have reported the results
in Table
Dataset
Encodings
IS
FID 4
R %
Bi-LSTM
5.12
13.67
86.64
CUB
Transformer Encoder
5.19
13.12
87.9
Bi-LSTM
4.21
30.07
83.19
Oxford
Transformer Encoder
4.27
29.61
83.94
Table 11: We compare quality of T21 generation using Bi-LSTM and layer Transformer Encoder text encoding scheme and report the results on CUB and Oxford-102 datasct.
20

---

Published in Transactions on Machine Learning Research (06/2025)
this flower has petals that yellow and folded together
this flower has small white petals long yellow stamens.
the bird yellow throat, breast belly with black covering his coverts and retrices.
small yellow and brown bird with long tarsus and medium length black beak.
Figure 7: Images generated via sentence interpolation between captions (for reference images), with interpo- lation applied only to Discriminator-side embeddings Sp (top row), only to Generator-side embeddings Sç (middle row) and to both embeddings Sp&SS (bottom row) shown for flowers and birds using DTTE-ANN
C Sentence Interpolation
Figure i illustrates sentence interpolation using different sets of embeddings. We perform interpolation on (1) discriminator-side embeddings alone (top row), (ii) generator-side embeddings alone (middle row), and (iii) both embeddings logether (bottom row).
We observe that interpolating only the discriminator-side embeddings trained to capture text-image align- ment results in images that exhibit smooth transitions in high-level concepts such as shape and object features. In contrast, interpolating only the generator-side embeddings (while keeping the discriminator-siide fixed) primarily affects the style and visual appearance, where the first image gradually adopts stylistic clements of the second. Finally, when both generator and discriminator embeddings are interpolated, the generated images show more comprehensive changes concept and style, producing more coherent, seman- tically rich outputs and closer to realistic images.
21

---

Published in Transactions on Machine Learning Research (06/2025)
Upsampling
5*3 Convolution
Lincar
LeakyReLL
1M1 Convolution
3'3
CBN
Linear
LeskyRaLL
Figure 8: UpBlock used in Generator of D'EE-AN.
For sentence interpolation, we follow the Text-to-Image synthesis approach proposed by Reed et al. 2016cc where an interpolated sentence embedding SInT is computed between two embeddings S1 & S2, derived from different captions, using a weighted sunmation as follows (with a E [0, 1)):
SINT = (1- aS²
(19)
D Details of the Proposed Architecture
In this section, elaborate internal architecture details of the DTE-GAN. Proposed model is implemented using Pytorch (Paszke et al. 2019) framework. DTEEGAN architecture consists of dual text embedding setup (Section D.0.1 single-stage Generator (Section D.0.2) and a Discriminator (Section D.0.3
D.0.1 Dual Text Embeddings
In the Dual Text Embeddings setup, bi- bi-Diroctional LSTM (Schuster & Paliwal
1997 used text encoder both generator-side and discriminator-side For each direction in the LSTM, hidden layer size is set 128. The size of word embeddings Wp and Wa is set to 256. The sentence embeddings Sc, Sp are encoded from the output of last hidden state of respective text encoders. For both the text encoders, sentence embedding size is sot to 256.
D.0.2 Generator
Single-stage generator G' is used to generate 256 x 256 resolution images with base channel dimension of 64. Details of G's architecture are shown in Table 12 The generator G takes noisc along with generator-side sentence embeddings Sa and the discriminator-sidee sentence embedding Sp and passes them through a set of linear layers followed by a set of upsampling blocks (UpBlocks). UpBlock at each stage is utilised for up sampling spatial features as shown in Figure X SG and Sp are also used to calculate modulation parameters for Conditional Batch Normalisation (Yin al. 2201)). Features are passed through self modulation convolution and 1 convolution resulting in generation of final image of dimension 3 x 256 256
22

---

Published in Transactions on Machine Learning Research (06/2025)
zeIR 100 NO0, 1) Sg I256, Sp IR256
WG IR260, WD R256
Linear(512) 512
Conditional Augmentation (512) 200
Lincar(200+100) - (8 ch) 4 x 4 UpBlock + (8 ch)
8
UpBlock - (8 * ch) X 16 x 16
UpBlock + (4 * ch) 32 32
UpBlock - (2 chi x 64 x 64
UpBlock (2 ch) 128 128
UpBlock -> ch x 256 x 256
Self Modulation Convolution ch x 256 x 256
x 1 Convolution - 3 x 256 256
Table 12: Generator architecture of DTF-GAN Base channel dimension ch 64.
3'3 Convolution
1'1 Convalution
LeakyReLU
3*3 Convolution
2'2 Average Pooling
LeakyReLU
2'2 Average Pooling
Add
Figure 9: DownBlock used in Discriminator of DTE-GAN.
D.0.3 Discriminator
Discriminator D is utilised to provide adversarial loss and also act as fcature extractor for multi-modal contrastive loss (as shown in Table 13 Unlike multiple / multi-slage discriminator setup, presented model with a single discriminator, is easy to train and not having a cumbersome training procedure. The dis- criminator D takes image of dimension 3 x 256 256 and passos it through a set of down sampling blocks (DownBlocks- as shown in Figure 9) followed by two branches, one for the adversarial loss and the other multi-modal contrastive loss.
23

---

Published in Transactions on Machine Learning Research (06/2025)
RGB images x 256 x 256, SD e IR256 Wp e R256
DownBlock ch x 128 x 128
DownBlock -> (2 * ch) x 64 64
DownBlock - (4 # ch) x 32 x 32
DownBlock (4 * ch) x 16 16
DownBlock (4 ch) x 8 x 8
DownBlock (8 # ch)
DownBlock - (8 + ch) x
ResBlock - (8 ch) x x4x4 4
ResBlock - (8 ch) x 4 x 4
Linear (8 ch) x 4) -1
Linear((8 * ch) 4 4) 256
Adversarial Loss
Multi-Modal Contrastive Loss
Table 13: Discriminator architecture of DTE-GAN. Base channel dimension ch 64.
D.1 Implementation Details
Implementation of the models is done using the Py Torch framework (Paskke et al. 2019) and optimising the network using Adam optimiser (Kingma & Ba, 2015 with the following hyper parameters: 3 = 0.5, B2 = 0.999, batch size = 24, lcarning rate = 0.0002, A = X2 = and X3 1. Spectral Normalisation (Miyato al. 2018) used for all convolutions and fully connected layers in generator and discriminator.. The model is trained for 600 epochs on CUB and Oxford-102 datasets (takes +4 days in NVIDIA 1080Ti GPUs) and 120 epochs for COCO dataset (takes ~7 days in 2 NVIDIA 1080Ti GPUs). During inference, report results with exponential moving average weights, with decay rate of 0.999.
24


Reviews:
[{"review_id": "PYSMyLrrhH", "reviewer": "Reviewer_BMR5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n* The framework is well ablated and has a motivation. It also paves a way forward for the researchers to think about whether text embeddings should be trained/optimized for text-to-image synthesis.\n* It uses simple text encoders, discarding the computationally costly ones. \n* It shows the merits of training an image generator along with a decoupled text embedding learning strategy. \n\n## Weaknesses\n\n* The paper mentions the use of CLIP in modern diffusion-based T2I models. However, that is not true. Many models like PixArt-Alpha [1], SANA [2], etc., don't use it and instead rely on T5 [3] and Gemma [4] for text embedding. Imagen [5] even suggests scaling up the size of text encoders for more photorealistic generation.\n* It doesn't use any metrics that are known to assess text-image alignment and image manipulation (ClipScore [6] and directional similarity [7], respectively). \n* It is unclear what training dataset was used for DTE-GAN.\n\n## References\n\n[1] PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis; Chen et al.; 2023; https://arxiv.org/abs/2310.00426\n\n[2] SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers; Xie et al.; 2024; https://arxiv.org/abs/2410.10629\n\n[3] Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer; Raffel et al.; 2019; https://arxiv.org/abs/1910.10683\n\n[4] Gemma: Open Models Based on Gemini Research and Technology; Gemma Team; 2024; https://arxiv.org/abs/2403.08295\n\n[5] Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding; Saharia et al.; 2022; https://arxiv.org/abs/2205.11487\n\n[6] CLIPScore: A Reference-free Evaluation Metric for Image Captioning; Hessel et al.; 2021; https://arxiv.org/abs/2104.08718\n\n[7] InstructPix2Pix: Learning to Follow Image Editing Instructions; Brooks et al.; 2022; https://arxiv.org/abs/2211.09800", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose DTE-GAN, a GAN-based framework. It is for text-to-image synthesis where the text embeddings are trained along with the image generator in an end-to-end manner. Furthermore, DTE-GAN decouples the learning of text embeddings in a dual framework that optimizes photorealism and text-image alignment.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Could the authors ablate the design choice of the Bi-LSTM-based text encoder used in DTEGAN?\n* What captions were used in Figure 6?\n* What is $\\lambda_{3}$ in eqn. 12?\n* How does the loss combination ensure learning of photorealism and text-image alignment? What is the intuition behind it? \n* The benchmark datasets CalTech and Oxford don't have captions for the images. How were they generated?", "broader_impact_concerns": "No ethical concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "22JJOteeM3", "reviewer": "Reviewer_FnYf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\nA notable strength of this work is the explicit separation of embeddings into two distinct roles: one for image quality and the other for semantic alignment. This separation promotes each embedding to specialize in its respective objective, leading to improved performance across standard benchmarks. The end-to-end training of these embeddings within the synthesis framework avoids reliance on fixed, pre-trained models.\n\n\n### Weaknesses\n\nWhile the current experimental results suggest that the dual embedding approach contributes to disentangling the roles of image quality and text-image alignment, this claim is currently supported indirectly through ablation studies. The paper would benefit from additional experiments that more explicitly analyze or visualize the distinct functionalities of each embedding. For instance, perturbation-based analyses (such as injecting controlled noise into each embedding) or latent space walking could offer deeper insights into their respective contributions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes Dual Text Embeddings (DTE) to enhance the image quality and text-to-image alignment in text-to-image (T2I) models. Unlike previous works, the proposed DTE-GAN method breaks apart from using pre-trained, generic embeddings by training two distinct text embeddings within the neural network: one optimized for improving image realism and the other for better text-to-image alignment, which is integrated into the GAN architecture via adversarial and multimodal contrastive losses. The model is evaluated on three benchmark datasets: Oxford-102, Caltech-UCSD, and MS-COCO, demonstrating superior or competitive performance in terms of Inception Score, FID, and R-precision.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It would strengthen the paper and benefit the readers if the red, green, blue colors used in the tables are substituted with bold, underline, etc.", "broader_impact_concerns": "Currently, there are no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oZPzNsbheu", "reviewer": "Reviewer_Wz7H", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The evaluation section should be improved. The comparison to SOTA models actually does not contain SOTA T2I models, such as stable diffusion. A good work on this domain should include comparison to latest diffusion models too, but this work only covers more GAN works.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Authors explored an approach to learning text embeddings specifically tailored to the T2I synthesis network, trained in an end-to-end fashion. Their dual text embedding approach combining both generative and discriminative embeddings (Fig1).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "What is the resolution of generated images? e.g. COCO. I highly recommend the authors to include comparisons to latest T2I generations such as Stable Diffusion, given that it is open sourced. \nEven more stronger and more impactful is the application of the proposed method on recent diffusion models.\"This method of focusing solely on generator-side embeddings can be seamlessly incorporated into current diffusion-based Text-to-Image models\"", "broader_impact_concerns": "no concern at the moment", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all acknowledge the interest and value of separating the role of the text embeddings in text-to-image generation. They initially expressed a few concerns regarding the empirical evaluation of the work. These concerns, however, were cleared by the author's responses and revisions of the paper. As such, all reviewers recommended acceptance in their final recommendation.
Decision: Accept (poster)