Blending Two Styles: Generating Inter-domain Images with MiddleGAN
Authors: Collin MacDonald
Zhendong Chu
John Stankovic
Huajie Shao
Gang Zhou
Ashley Gao
Venue: Accepted by TMLR ()

Abstract: From celebrity faces to cats and dogs, humans enjoy pushing the boundaries of art by blending existing concepts together in new ways. With the rise of generative artificial intelligence, machines are increasingly capable of creating new images. Generative Adversarial Networks (GANs) generate images similar to their training data but struggle to blend images from distinct datasets. This paper introduces MiddleGAN, a novel GAN variant that blends inter-domain images from two distinct input sets. By incorporating a second discriminator, MiddleGAN forces the generator to create images that fool both discriminators, thus capturing the qualities of both input sets. We also introduce a blend ratio hyperparameter to control the weighting of the input sets and compensate for datasets of different complexities. Evaluating MiddleGAN on the CelebA dataset, we demonstrate that it successfully generates images that lie between the distributions of the input sets, both mathematically and visually. An additional experiment verifies the viability of MiddleGAN on handwritten digit datasets (DIDA and MNIST). We provide a proof of optimal convergence for the neural networks in our architecture and show that MiddleGAN functions across various resolutions and blend ratios. We conclude with potential future research directions for MiddleGAN.

Published in Transactions on Machine Learning Research (12/2024)
Blending Two Styles: Generating Inter-domain Images with MiddleGAN
Collin MacDonald
emacdonald01 @wwm. edu
Deportment of Computer Science William & Mary
Zhendong Chu
zc:uuy Deviiiinin edu
Department Computer Science University of Virginia
John Stankovic
jasgg@virginia edu
Department of Computer Science University of Virginia
Huajie Shao
hshao @umm. edu
Department of Computer Science William & Mary
Gang Zhou
gzhou Aakko edu
Department of Computer Science Willimm & Mory
Ye Gao
ugao edu
Department of Compuler Science William Mary
Reviewed on OpenReview: https: openreview. net/ forum? id= CHuWCH
Abstract
From celebrity faces cats and dogs, humans enjoy pushing the boundaries of by blend- ing existing concepts together in new ways. With the rise of generative artificial intelligence, machines are increasingly capable of creating new images. Generative Adversarial Networks (GANs) generate images similar to their training data but struggle to blend images from distinct datasets. This paper introduces MiddleGAN, a novel GAN variant that blends inter-domain images from two distinct input sets. By incorporating a second discriminator, MiddleGAN forces the generator create images that fool both discriminators, thus cap- turing the qualities of both input sets. We also introduce blend ratio hyperparameter to control the weighting of the input sets and compensate for datasets of different complex- itics. Evaluating MiddleGAN generates images that lie bet the distributions of the input sets, both matherna ically and visually. An additional experiment verifies the viability of MiddleGAN on handwritten digit datasets (DIDA and MNIST). We provide a proof optimal convergence for the ncural networks in our architecture and show that MiddleGAN functions across various resolutions and blend ratios. We conclude with potential future research directions for MiddleGAN.
the CclebA dataset, demonstrate that it successfully
"Equal contribution. "Corresponding author.

---

Published in Transactions on Machine Learning Research (12/2024)
1 Introduction
Humans have always loved to create, especially visually. Using advanced photo manipulation tools such Photoshop, digital artists have produced highly realistic blended faces of well-known celebrities (Emma Taggarth. Simultancously, with the rise of generative artificial intelligence (AT), computers are now "creating" images and other artistic works as well (Cetinic & She). One such method of image creation is Generative Adversarial Networks (GANs). GANs are machine learning models that are trained on set of images and learn to create new images (from random noise) that are similar to the training images. More formally, GANs learn generate images that are in distribution with the images of the input domain. The use of GANs to create images has been extensively explored over the last decade. However, traditional GANs are not designed to blend two distinct sets of images together.
In this paper, we present MiddleGAN, novel variation of the traditional GAN, which takes as input two domains of images and learns to create images that lie between the input domains. In essence, MiddleGAN aims to create blended inter-domain images that contain features from both input domains. For example, given the input domains male and female faces, MiddleGAN can produce images of human faces with both masculine and feminine qualities, as shown in Figure 1 Compared existing state-of- state-of-the-art multi- discriminator GAN variations, such FairGAN (Xu et al.} and D2GAN [Nguyen et al.), the dual-dataset input and blended-image objective of MiddleGAN is novel. Instead of mamually blending the faces of female and male celebrilies, an artist could use model such MiddleGAN to create entirely new faces al the click of button. Artists and designers can use the blended features to create unique and original artworks. By generating faces or figures that combine male and female traits, they can explore new aesthetic possibilities or challenge traditional gender representations in art and media. In addition, industries like gaming and animation, blending features from different domains (e.g,, male and female) can allow for more nuanced character creation. It could help generate characters with androgynous features or those that evolve across different gender traits dynamically.
To achieve our goal of inter-domain image generation, we ask three overarching research questions:
RQI:
How can we create inter-domain images that have qualities of both input domains, and validate this methodologically via t-SNE?
Is feasible to generate images that lie in the middle of the feature space of the two domains, or can we extend our methodology to place unequal emphasis on each input domain?
RQ2:
RQ3: Are the generated images consistently high quality across multiple image sizes and blend ratios?
In order to answer these research questions, we make three contributions:
First, we propose novel variation of GAN, MiddleGAN, which leverages two discriminators and one generator lo create blended inter-domain images from two inpul datasets.
Second, we extend the initial functionality of MiddleGAN introduce Blend Ratio, which controls the amount of emphasis placed on each input domain's corresponding discriminator.
Third, through extensive evaluation on the CelebA dataset, we show that images generated by MiddleGAN are within the expected distribution (via t-SNE) and have both masculine and feminine qualities when visually examined.
The remainder of this paper is organized as follows: Section 2 describes related work and the context of this project. Section | describes the thcoretical basis of MiddleGAN and its model architecture. Section discusses our implementation of MiddleGAN. Section g details our extensive evaluation of MiddleGAN. Section 7 discusses MiddleGAN and outlines opportunities for future work. Section concludes our paper.
2

---

Published in Transactions on Machine Learning Research (12/2024)
(a) Generated (Male-leaning)
(b) Generated (Middle)
(c) Generated (Female-leaning
Visualaation
Vossiltation
Vissliiaation
-10-
5VE Denemvon1
USKC menton)
155E Dmncsoon
(d) t-SNE (Male-leaning)
(e) t-SNE (Middle)
(f) t-SNE (Female-leaning)
Figure 1: MiddleGAN can generate images which are blends of two domains. In this figure, we show the results of images generated with MiddleGAN at a resolution of 128px. The t-SNE visualizations support our claim that the blended images fall within the two input distributions (male and female).
2 Related Work
In this section, we present an overview of the original generative adversarial network (GAN) as well as several subsequent GANs as they relate to our current work.
Gencrative Adversarial Nets: Since the introduction of the first generative adversarial network over a decade ago, many novel variations of GANs have been introduced. However, at the start of them all is the work of Goodfellow et al. the paper "Generative Adversarial Nets' (Goodfellow et al.). The original GAN contained one discriminator and one generator. These models are trained in parallel. The generator learns to take random noise input and produce an image as output. The discriminator learns to distinguish between real images and those created by the generator. By engaging in minimax game, the generator learns to produce better images while the discriminator learns to better differentiate between real and generated images. The goal of training is to create a generator model that can produce images, from uniform random noise, that are within the distribution of the original training images. In other words, 'GANs are a framework for teaching a deep learning model to capture the training data distribution so we can generate new data from that same distribution' (Tutorials).
GANs have been used in wide range of applications, ranging from image-to-image style transfer to domain transfer tasks. CycleGAN focuses on 'unpaired imago-to-image translation,' while GP-GAN focuses on high- resolution composite image blending (W et al.; Zhu et al.). The work of Sankar et al. focuses on the use of GANs for domain adaptation, and the work of Rahman et al. focuses on how GANs can be leveraged for domain generalization.
DCGAN: While an important first step, the original GAN as proposed by Goodfellow et al. is notoriously hard to train and often fail to converge (Nie & Patel). Due to these challenges, many improved GAN
3

---

Published in Transactions on Machine Learning Research (12/2024)
variations have been proposed over the course of the last decade. One such proposed improvement is the Deep Convolutional Generative Adversarial Network (DCGAN) (Radford et al.). Radford et al. modified the original GAN to use Convolutional Neural Networks (CNN), which at the time were gaining popularity in computer vision tasks. DCGAN forms the basis for our MiddleGAN model, with its use of 'convolutional and convoluiional--ransposee layers in the discriminator and generator, respectively' (Tutorials). DCGAN helped to prove the viability of CNN-based GANs and improved the training stability of the model when compared to the original GAN (Radford et al.).
WGAN & WGAN-GP: The Wasserstein GAN (WGAN), introduced by Arjovsky et al. in 2017, builds upon the success of DCGAN, making use of CNN-based architecture (Arjovsky et al.). The novelty of the WGAN is two-fold. First, WGAN trains the discriminator (termed "critic") at a higher rate than the generator. As such, "training WGANs does not require maintaining a careful balance in training of the discriminator and the generator, and does not require a careful design of the network architecture either" (Arjovsky et a..). Socond, and more importantly, the loss function for the WGAN is based on the Earth Mover's (Wasserstein-l) distance, which differs from traditional GANs and provides 'a meaningful loss metric that correlates with the generator's convergence and sample quality" (Arjovsky et al.).
Gulrajani et al. proposed WGAN-GP as a direct improvement of the traditional Wasserstein GAN Gulrajani et al.)) Instead of using weight clipping to enforce the Lipschitz constraint on the discriminator /critic, which is required by WGAN, WGAN-GP 'penalizes the norm of the gradient of the critic with respect to its input" (Gulrajani et al.pp We use the loss function of WGAN-GP as the basis for the loss function of MiddleGAN.
GANs: Beyond the above works, several other proposed GAN variations make use of dual discriminators. In "Generative Multi-Adversarial Networks," Durugkar et al. propose the use of multiple discriminators for a variely of use cases, including serving as a 'formidable adversary" and a 'forgiving teacher" (Durugkar et al.). Nguyen et al. attempt to mitigate issues of mode collapse through the use of dual discriminators in "Dual Discriminator Generative Adversarial Nets" (Nguyen et al.). More concretely, Xu et al. make use of two discriminators to improve the fairness of generated images with FairGAN (Xu et al.). MD-GAN proposes the use of multiple discriminators as an approach to distributed learning (Hardy et al.). Lastly, PATE-GAN uses multiple discriminators as applied to the field of differential privacy (Jordon & Yoon
Multi-Discrimint
Given the introduction of GANs in 2014, they are somewhat dated. One might be tempted to ask why we focused on a GAN-based architecture as opposed to a more modern architecture, such as Diffusion Models. Diffusion Models, introduced in 2015, leverage idcas from 'nonequiliriim ther- modynamics" to train a generative AT model (Sohl-Dickstein et al.p. Unlike GANs, diffusion models are not adversarial. Instead, diffusion models work by 'systematiiaaly and slowly destroying structure in a data distribution through an iterative forward diffusion process... [and] then learning a reverse diffusion process that restores structure in data, yielding highly flexible and tractable generative model of the data" (Sohl- Dickstein et al.). In other words, a diffusion model first learns to convert an image to seemingly random noise, and then learns to reverse this process, taking random noisc and converting it into an in-distribution image. This process cannot be easily extended to support multiple input datasets, which prevents rraditional diffusion models from forming the basis for an inter-domain image generation model. While more recent work such as "Blended Latent Diffussin'' has leveragod diffusion models to mask and blend individual images, this work does not overcome the inherent single-domain limitation of diffussin models (Avrahami et al.).
Diffusion Models:
When compared to previous works, MiddleGAN is distinct in its goal. Unlike traditional GANs, MiddleGAN utilizes two discriminators and one generator. Even when compared to other multi-discriminator GANs such as MD-GAN and FairGAN, the objective of MiddleGAN is noticeably different, as MiddleGAN aims to generate images that are in between the distributions of the two input datasets. Despite the existence of more modern generative AI architectures such as diffusson models, MiddleGAN is still relevant as it leverages the adversarial nature of GANs to generate images in between two distributions, which diffusion models cannot do.
4

---

Published in Transactions on Machine Learning Research (12/2024)
3 Theory
In Section 4 we going to provide detailed information exactly how we achieve to obtain the inter- domain. In this Section (Section 3), we explore what is the potential usage of the inter-domain. To wit, the inter-domain, whose similarity to the two original domains is showcased by being equally distant (measured via Wasserstein distance) to the two original domains, can be used as an intermodiate domain shown to lacilitate better domain adaplation and generalization: existing works Na et al. (2021); Wang et al. (2022) have already showcased that empirically, adapting from one original domain (source) another (target) is useful. Here, we theoretically prove that the inter-domain generated by MiddleGAN can indeed lower the error difference over distribution and domain shift if adapt from original domain to the inter-domain then to the other original domain, compared with the strategy in which the domain adaptation allapaation/geneerllaaaaiii /generaliiiation happens directly from one original domain (that is used as the source) to the other original domain (that is used as the target). The purpose of this proof is to showcase a potential usage for the generated synthetic inmain
3.1 Preliminaries
Let X' and y represent the input and output spaces, respectively. Let X and Y be random variables taken from the input and output spaces. For given domain (e.g., female faces), the distribution is denoted as μ over X' y. When considering only the sample distribution and not the joint sample-label distribution, we denote the sample distribution μ over the input space X as p(X).
Assumption 1 (Bounded Input Space). A compact input space is bounded in the d-dimensional unit & ball:
X C E R :|10||2 $1}
Definition (p- Wassessen Distance). Consider z and over C IRd, For any p 2 1, the p- D-Waaserstein distance is defined as:
1/v
inf / d(I, v) dy(x,)))) (ETlan sxs
Wo(mvv)
where VV) is the set of all measures over S x S with marginals equal to μ and respectively,
3.2 Error Difference over Distribution & Domain Shift Assumption (R-Lipschitz Classifier). Let H be R-Lipschitz in &a norm, i.e.:
Va, E x: [f(x) - h(x') S R||x x'112
Assumption 3 (p-Lipschitz Loss). é is p-pischhitz, i.e., Vy, 3/ e >:
((y, (0yl-((y))))) s plyy v'll2
(((:8) - 1100-((-100 v() < 2lly - 1/12
In the settings MiddleGAN, consider u,v and the measures for the middle domain m. Lemma (Error difference over shifted domains):
Population loss (h) 6m (h)) Im (h) cv(hi) S PVR R + 1W W)(,))

---

Published in Transactions on Machine Learning Research (12/2024)
Proof:
(h) -m(hh] B [e(h(1),,)) Br wr (e(h(a'), of
f ((b()), v) du - / ((b(z'), v/) dra
(m) (h)]
|E w ((((x')))))))))))(()))))))))(() 3/)1 10 v1"
/ e(M(a'). dm / (((((=)))))) du
(h) - cm (A)]++emmm))) cv(h) = 1/ ((h(z),)) du du- j e(h(zz)) (
((h(z)) dm - e(h(z"), y0) du of +/////
Let be an arbitrary coupling of and m (i.c., a joint distribution of μ. m). Similarly, we define Tm.er
(1 - cc((h)] = =1ʃ [f(h(t)).u) - e(h(('),, dmmm
Triangle inequality < f [((h(x),)) ((((('),,)) v0)0 drm..
(& p-Lipschitz)
|f [((((o).)) - ((h(x').))) drmmm $ $ /(ll0(()) - M(O'Il) + plly s/11) d ((oi(())>-oo)1)))
(h is R-Lipschitz)
S ((llb()))-(((1100 + ply - 2/1l) dram
S sS / pRlli - I'll + plly d7mm
(R 0)
$ / pV R2 +1(11x x'11 11 - s/11) drp.
S SinG / PV R2 1(1|x x'll + 1l alll) d
=PV R2 (11((,,) S sVrRe + 1W
Similarly:
ICm (h) - (h)l S PV R + 1Wp(m,))
(h) - (m (h)) + lem (h) - ch)))) S pV R - (W, + PV R (W
Since Wp(n, m) Wp(m, v) = - [ v):
(1) em (7:) 1e (h) <c(hh) 2pv R + (W) S PV Re + (W,(H,

---

Published in Transactions on Machine Learning Research (12/2024)
Remark: Existing work Wang et al. (2022) already proves that (h) cx(h) I pVR2 (Wp(u,)) Here, we prove that if there's s a middle domain, m, then (cx(h) <m (h)) and - cz(h) are both I. 1pv R2 + 1Wp(H, v). Therefore, adapting from to m and m to v breaks the original adaptation task from to v in two subtasks, with each subtask equipped with smaller population loss on error difference over shifted domains.
4 Model Architecture & Design
In this section, we provide overview of the model architecture well the design of MiddleGAN. Additionally, we provide a theoretical basis for MiddleGAN, building on lop of the original GAN. We show show that there exists optimal parameters for both discriminators as well as optimal solution for the parameters of the generator.
Before we discuss MiddleGAN, we need to discuss the original GAN on which MiddleGAN is based. In the original GAN (Goodfellow al.), the generator and the discriminator engage in a minimax game which G tries to minimize value objective (GG, D) whereas D tries maximize it. V(G, D) is defined in Equation
D in which is the distribution of the real samples and 9 is the distribution of the noise. A key observation obtained from Equation D is that G's effort is to generate G(z) whereas z is input noise such that G(2) will be in-distribution with the distribution of the real samples
min max V(G, D) p(p)(() [o)(D(x))
D
(1)
+ Ez- Dog((1 - D(G(z)))
Based on the key observation that we obtain from Equation in MiddleGAN we propose to employ two discriminators, Da and D,. Each discriminator only aware of one domain of images. The first discriminator, termed Da, only knows of the images in Domain A. Similarly, the second discriminator, termed D,, only knows of the images in Domain B. Neither discriminator knows of the existence of the other domain of images, and is solely responsible for determining if an image came from its corresponding domain or not. The generator G engages in a two-way minimax game with the two discriminators. The samples it generates will be in the middle of the feature space of both input domains. Below, we empirically prove that the generated samples in Pm are represented by the features that are invariant across the input domains.
Formally, the objective function of Da: Ds, and G described by Equation 2
min max V(G, D., D₂) G D.. D.
- = Ex up (x.)) [log( D(Ia)) + Esng(E) (o D.
(2)
+ Ex log( (xx))) + Ex [log(1 Dz(G(=))))
In the previous paragraphs we have described how to generate samples that are similar to samples from both input domains. Figure 2 outlines the duel-discriminator architecture of MiddleGAN and how it used to generate domain agnostic samples.
4.1 Optima for the Discriminators and Generator
We first discuss the two discriminators Da and D3 given fixed G. We propose Theorem 4.1 regarding the optimal values for D. and Da, represented as D and Dj
Given Pms the distribution of samples generated by a fixed generator G, the optimal values for the parameters of Da and D, are D = P. and DJ
Pe Pu+
Proof. The value objective V(G, Da, Db) be expanded.

---

Published in Transactions on Machine Learning Research (12/2024)
Inputs
MiddleGAN Training
Generated Images
Dataset A
LDA
DA
Random Noise
Generator
Generator Loss
Dg
Dalaset B
LDB
In this figure describe how to the MiddleGAN to generate fake, domain agnostic samples. As shown in the diagram, neither discriminator is directly aware of the others existence the existence of a second dataset. This is importance, as it allows cach discriminator to focus solely on diffcrentiating between images in it's own dataset and any other images, regardless of source. Black arrows indicate the forward flow of information, while red arrows indicate backwards propagation. The dashed green line indicates that the generated images come from the trained generator.
Figure 2:
[ (o(oa))og((P,(F))dd) /
V(G, Da. D₂)
((2) log(1 - Da(G(=))) - Do(G(2)))dd
(3)
[
+ Pm (m)) log(1 - Da(xa)))) (dr
+ / po(xx) (og(DD((())))) + Pm (16) log(1 Da(rs))drs
We observe that Pa; Ps and Pm belong in R. For the domain A discriminator Da, any pair of Pa and Pm in the form of Pa log(y) + Pm (I (1- log()), Pa log(y) Pm (1 - log(v)) achieves its maximum value at Pa+ (Goodfellow al.). Similarly, for the domain discriminator Ds, any pair of ps and pm in the form of Ps log(y) + Pm log(y), p6 log(y) + Pm (1 log(y)) achieves its maximum value at Put
Now we bring forth Theorem 4.1 which proposes that there exists an optimal solution for the parameters of not only Da and D,, but also G. There exists a global minimum for the virtual training criterion C(G) defined as
mar V(G, Db). C(G)
(4)
D.. D.
In other words, there exists an optimal solution for the parameters of the generator G.
Proof. Goodfellow et al. proved that, in the original GAN where there is only one discriminator D and one generator G, the virtual training criterion can be written as the following:
Coriginat (G) max DG
(5)
p P P+Pm KL(pm ||
P+ Pm
-lo + KL(p) ||
2
2
in which p the distribution of the real samples and Pm the distribution of generated fake samples, and KL the Hallback-Leible divergence. With two discriminators, our virtual training criterion C(G) can be rewritten
8

---

Published in Transactions on Machine Learning Research (12/2024)
Pa + Pa+ Pm
Po + Pm
C(G) 1og(4) [L(pa
+ KL(m m
1og(4) + LLpp
Pb + +Pm + [L ||
+ Pm
(6)
2 2l)g(4) + 2JSD(pa Pm) 2JSD(pb |I Pm)
In Equation
JSD the Shaannn divergence. To find the global minimum, M(G), we want 6
obtain
M(G) = argmin - 24og(4) + 2JSD(pp Il Pm.) + 2JSD(pp II Pm)
(7)
Pm
We observe in Equation that we are looking for the optimal value of the JSD centroid defined as Centroid" ory min E (S; Q) in which P. and Q are distributions. We can see that the generator is essentially
looking for the JSD controid of the source domain distribution Pa and the target domain distribution pb- The convexity of the problem has been proved in Nielssnn 2020).
4.2 Beyond the "Middle" Feature Space
With two discriminators, the generator G faces an additional challenge, as its success is determined by its ability to fool both discriminators. In order to do this, the generator must learn to produce images that have features of both input domains. Simply put, the loss of the generator is the averaged loss from both discriminators, as shown in Equation B
LG = (LDa + LDD)
(8)
2
In Equation the losses of both discriminators are weighed equally. This aims to ensure that the generator G learns to produce images that are in the middle of both input domains. However, this is an artificial limitation, as the weights for the losses from each discriminator do not have to be evenly weighted. To expand the flexibility of MiddleGAN, we introduced a new blending hyperparameter, the blend ratio, into our model architecture.
La L= Ln, * (BR) + LDg * (1- BR)
(9)
The updated loss for the generator is shown in Equation 9 The blend ratio can range from 0 to 1, and directly controls the percent of weight placed on the loss of DA A blend ratio of 0.5 equally weighs the losses from both discriminators and replicates our initial implementation of MiddleGAN. A blend ratio of 0 would completely ignore Domain A and cause MiddleGAN to act like a traditional GAN, using only Domain B as input. A blend ratio of 0.75 would result in the loss of discriminator A carrying three times the weight of the loss of discriminator B. The blend ratio vastly increases the range of images that MiddleGAN can learn to produce.
4.2.1 Choosing the Blending Ratio.
In many generative models, the influence of domains is affectod not only by the blend ratio but also by factors such as the underlying data distributions and the number of samples. However, in MiddleGAN, the blend ratio directly determines how the losses from the two discriminators tematically blend features from both domains without relying on assumptions about the data distributions. Since the discriminators are trained independently on their respective datasets, the blend ratio effectively serves as a weighted "vole" between the loss functions, providing more precise control over the blending process than the raw characteristics of the data alone.
weighted, enabling us to sys-
We agree that manually selecting the appropriate blend ratio can be inefficient. To address this, we suggest using grid search to systematically optimize the blend ratio. Our aim is to minimize metrics like Kernel
9

---

Published in Transactions on Machine Learning Research (12/2024)
Inception Distance (KID) and Fréchet Inception Distance (FID), making grid search logical approach. By exploring different blend ratio values, we can objectively determine the best ratio that balances the contributions of both domains and enhances image quality.
However, an important point to note is that while FID and KID are typically used to assess the similarity between real and generated images, MiddleGAN's goal is different. Instead of producing images that perfectly match a single real dataset, we aim to generate images that intentionally blend features from both domains. This means that our generated images are intentionally distinct from either of the real datasets. For details the FID and KID scores after grid search, please refer to Table 2 in Section 6.2.1
4.3 Novelty of MiddleGAN
At first glance, MiddleGAN might appear similar to traditional GAN, but with two discriminators instead of onc. However, the second discriminator plays a pivotal role that sets MiddleGAN apart from standard GANs. While many GAN variants utilize multiple discriminators, MiddleGAN is uniquely designed to generate inter-domain images that blend features from two distinct datasets, which is not the primary focus of typical GANs. The second discriminator is not merely an ineremental addition; it is crucial for ensuring that the generated images effectively combine characteristics from both input domains. This enables MiddleGAN to function in a space between domain translation and image synthesis, providing a solution for generating images that exist between two feature distributions something that traditional GANs, like DCGAN or StarGAN, are not built to do. Additionally, the blend ratio hyperparameter enhances the model's ficxibility, allowing precise control over the influence of each domain. This feature makes MiddleGAN particularly valuable in scenarios where a balance between characteristics from two domains is required, representing a significant advancement over conventional GAN approaches.
5 Implementation
Our implementation of MiddleGAN builds upon PyTorch's official implementation of DCGAN (Tutorials) This implementation was designed for use with the CelebA dataset, which sped up our initial development time. However, this implementation had several limitations, most notably a fixed input image size of 64px. By dynamically computing the number of layers required by both the generator and discriminator models based on the requested image size, we enabled our implementation of DCGAN to support any power-of-two image size.
Next, we modified our impletentation of DCGAN to add additional discriminator and support two input domains. These changes formed the basis for MiddleGAN. We modified the model architecture so that in each training epoch, images from both domain A and domain B were seen by the model. To account for differences in the sizes of the input domains, we leveraged the itertools library's Cycle method to ensure that the length of the smaller dataset would appear to match that of the larger dataset (pyt a)- With the addition of the second discriminator, we simply adjusted the loss function for the generator take into account the losses of both discriminators when evaluated on generated images. At this stage, our implementation of MiddleGAN could generate images, albeit with low quality.
With the basic MiddleGAN implementation complete, we began preliminary testing and evaluation of the model. We observed high training instability of the GAN and poor quality of generated images. After making these observations, we began to look into ways to stabilize the training of the GAN.
In parallel, we were exploring the idea of transformer-based implementation of MiddleGAN, based on TransGAN (Jiang et al.). While the paper authors provided PyTorch implementation of TransGAN (git) b) (which itself was based on their earlier work, AutoGAN ggit a.) opted for cleaner re-implementation of TransGAN (Sarigün). Using TransGAN as our basc model, we developed a low resolution (32px) transformer- based version of MiddleGAN. This implementation utilized the loss function from WGAN-GP (Gulrajani et al.) to stabilize the training process. A sample of the images produced with our transformer based implementation of MiddleGAN is shown Figure [al] Figure 3e and Figure 3b were very intermixed, showing that generated samples were not clearly between the input distributions. Given these results, along with significantly longer training times we observed, we ultimately declined to pursue development
10

---

Published in Transactions on Machine Learning Research (12/2024)
Visualization
famal:
mide
Ierale
(a) Generated Images
(b) L-SNE Visualization
(c) I-SNE Visualization
Figure 3: A sample of images generated with transformer-bsedd MiddleGAN implementation as well associated tSNE visualizations. The images were generated at resolution of 32px by 32px with blend ratio of of 0.50. As shown by the t-SNE visualizations, the generated images fell within the distribution of both input domains (as opposed to between it).
of a transformer-based MiddleGAN for this project. We leave it to future research to better evaluate the viability and effectiveness of a transformer- based MiddleGAN.
While we ultimately declined to pursuc a Transformer-basedd implementation of MiddleGAN, that work was not in vain, as we did observe that the loss function being used in TransGAN was leading to more stable model training. The loss function, WGAN-GP loss, is based the work of Gulrajani et al. in "Improved Training of Wasserstein GANs" (Gulrajani et al.).. When implemented WGAN-GP loss based MiddleGAN, we observed more stable training performance. Taking inspiration from the original WGAN paper (Arjovsky et. al.), we also modified our model to train the discriminalors al a higher frequency then the generator, which also led to improved quality in the output images.
DCGAN-
6 Evaluation
In this section we describe our dataset, experimental setup, and results for MiddleGAN across multiple datasets.
6.1 Experimental Setup
which was proposed by Liu et al. in 2015. The CelebA dataset contains over 200,000 annotated images of celebrity faces. We selected this dataset due its large size, extensive use prior works, and separability into two domains (*Male" and "Female"). To create the two domains we used the boolean "Male" identifier present in the original dataset annotations[] By splitting the "Male" identifier created two smaller datasets, which we designated 'Male' (with approx. 84,000 images) and "Female" (with approx. 118,000 images). In all experiments, the "Female" dataset was assigned domain A, and the 'Male' dataset was assigned domain B. These dataset-domain assignments were based on alphabetical ordering, but given the order-an aggostic structure of MiddleGAN with regards to input domains, we would anticipate identical results if the dataset-domain assignments were reversed.
Datasct: We primarily evaluated MiddleGAN using the ColebA datasct [Liu ot al. 2015
Image Generation: To evaluate MiddleGAN, used the "Male" and "Female" datasets to train nine different MiddleGAN models using combination of three blend ratios (0.25, 0.50, and 0.75) and three image sizes (32px, 64px, and 128px). Aside from the blend ratio and Image Size, all models used the same hyperparameters as shown in Table I If our selection of a hyperparameter was heavily influenced by one
I All annotations in the CelebA datasct are booleans. As such, made the assumption that any image that had value of "I" for the "Male" identifier was male, value 9-1" was female. While this may he 100% accurate, "-1" value is technically "Not. Male" opposed "Female", do not expect that this assumption significantly impacted our results.
11

---

Published in Transactions on Machine Learning Research (12/2024)
Table I: MiddleGAN Hyperparameters
Hyperparameter
Description
Values
Source
Batch Size
Batch size during training.
128
DCGAN
NZ
Latent space for generator.
128
Epochs
Epochs during training.
200
LRD
Learning rate of discriminators.
0.0001
TransGAN
LRG
Learning rate of generator.
0.0001
TransGAN
B1
Beta for Adam optimizers.
0.0
TransGAN
B2
Beta for Adam optimizers.
0.999
DCGAN
N Critic
Discriminator generator training ra-
WGAN
tio.
GP Weight
Gradient penalty weight, (used in loss 100
function).
Blend Ratio
The blend ratio between input domains. 0.25, 0.50, 0.75
Image Size
The size of the generated images.
32px, 64px, 128px
(a) Domain A Images (F)
(b) Domain Images (M)
(c) Generated Images
Figure 4: This figure shows the results of images from both input domains as well as images generated by MiddleGAN. We used an image size of 128px and a blend ratio 0.5. The generated images have both masculine and feminine qualities.
prior works, those works are noted in the "Souree Paper" column. All models were trained NVIDIA A100 GPU, with an average per-model training time of around 12 hours. We generated 1000 images per model for use in our experiments.
We elected to use the pre-trained ResNet101 model provided by PyTorch as the basis of our feature extractor (pyt) Φ He et al.). We then fine-tuned the model to provide a binary classification for male and female faces from the original CelebA dataset. This resulted in fine-tuned 1esNet101 models, one per Image Size. From there, were able to extract feature vectors (length=2048) from the second-to-last layer of the ResNet101 model. We extracted feature vectors for each of the nine sets of images generated by MiddleGAN, speaking, this means we extracted 3000 feature vectors for each of the nine models, which were evenly split between domain A images, domain B images, and generated images. Leveraging principal component analysis (PCA), we reduced the length of each extracted vector to 50 (sk] b). After performing PCA, we input the feature vectors into t-SNE in order better understand the relationship between the input images and generated images.
Feature Extraction & t-SNE:
well as matching-size sets of images from both input domains. Roughly
12

---

Published in Transactions on Machine Learning Research (12/2024)
(a) Epoch 50
(b) Epoch 100
(c) Epoch 150
(d) Epoch 200
Figure 5: Generated images on fixed noise across 200 epochs. We used an image size of 128px and a blend ratio of 0.50. Image quality continues to improve until at least the 200th epoch.
6.2 Experimental Results
We achieved positive results on all nine trained versions of MiddleGAN, across variety of blend ratios and image sizes. Figure g shows a sample of images from both input domains well as images generated by MiddleGAN. An informal evaluation by the authors of the paper found the generated images to have both masculine and feminine qualities, which one would expect given the input domains of male and female faces. Beyond the mix of masculinc and feminine qualitics, we noted a diverse range of skintones and ethnicities present both the input images and generated images, which further shows the diversity of images that MiddleGAN can generate. This is also an indicator that MiddleGAN did not suffer from mode collapse, which is when "the generator starts producing the same output (or small set of outputs) over and over again" (pro).
While the training loss for the generator slowly increases over the course of the training, we continued to see improvements in the resulting imagery, as shown in Figure which shows samples generated by MiddleGAN on fixed noise across training epochs. For this project, all training stopped after 200 epochs. We leave it to future work to evaluate the quality of images generated by MiddleGAN could continue to improve after 200 epochs.
6.2.1 KID and FID Scores to Evaluate the Generated Images
The FID score is commonly used to measure how closely the distribution of generated images aligns with that of real images by comparing feature representations extracted from a pre-trained Inception network. It evaluates both the realism and diversity of the generated images, with lower FID values indicating a better match to real images. The KTD score, contrast, is an alternative metric that does not assume Gaussian distribution for the features and is particularly beneficial when working with smaller datasets, as it avoids the bias that FID can introduce with limited sample sizes.
It's important to emphasize that FID and KTD are typically applied in contexts where the goal is to generate images that closely mirror the real image distribution. However, our objective in this work is different. MiddleGAN is designed to generate blended doter- images that incorporate features from two distinct domains-man male and female faces. As a result, the generated images are expected fall somewhere wetween the two rcal distributions. Consoquently, higher FID and KID values do not necessarily indicate poor performance but instead reflect the intentional design of our model. With this understanding, we present the computed FID and KID scores in Table 2
Although the FID scores for the generated images compared to the male and female distributions are higher than what is typical for GANs designed to replicate a single domain, this is due to the specific nature of our task. Notably, FID(female, generated) < Fifefemale, male) and FID(male, generated) < FID(female, male), indicating that the generated images resemble both male and female faces more closely than male and female faces resemble each other. This demonstrates that MiddleGAN successfully captures the intermediate characteristics of both domains, producing the blended images intended. The KID scores show similar
13

---

Published in Transactions on Machine Learning Research (12/2024)
Fomale, Middle
Male, Middle
Female, Make
FID
84.50
99.85
106.34
KID
0.0655
0.0700
0.0940
Table 2: The KID and FID scores to evaluate the generated images. It is worth highlighting that FID and KID are gemerally employed in lasks where the aim is lo generate images that closely resemble the real image distribution. However, this differs from our objective in this work. MiddleGAN is specifically designed to produce blended inter-domain images that combine features from two distinet domains (male and female faces). As result, the generated images are expected to fall between these two real distributions. Therefore, higher FID and KTD scores do not necessarily signify poor performance but rather reflect the intentional design of our model.
(a) Generated Images (32px)
(b) Generated Images (6dpx)
(c) Generated Images (1 (128px)
Viuallazion
5SME Mousiliation
Vissalaation
-22
-20
-12
28
(d) t-SNE Visualization (32px)
(e) t-SNE Visualization (64px)
(I) t-SNE Visualization (128px)
Figure 6: The impact of different image sizes on the images generated by MiddleGAN, with blend ratio of 0.5. While the generated images contain masculine and feminine qualities at all resolutions, the t-SNE visualizations are noticeably different at lower resolutions. This is expected, as at low resolutions, there are not enough pixels to clearly differentiate between masculine and feminine qualities.
trend, further confirming that the generated images capture the diversity and overlap of features between the two domains, rather than merely replicating features from one domain isolation. In conclusion, while FID and KID are traditionally used assess how well generated images replicate real ones, in our case, these metries reflect the inter-domain nature of the generated images, which are purposefully designed to blend features from both male and female faces. We hope this explanation clarifies approach and the rationale behind the FID and KID results.
6.3 Image Size Experiments
The impact of different blend ratios on the images generated by MiddleGAN, with an image size of 128px. With blend ratio of 0.25, the generated images were more masculine. When visualized with t-SNE,
14

---

Published in Transactions on Machine Learning Research (12/2024)
our Unualoation
SAT Utandooein
Dan Numvo
-10
-12
-00
H9ME
(a) Perplexity = 5
(b) Perplexity = 30
(c) Perplexity 90
Figure 7: This figure shows the results of our t-SNE perplexity experiments. We used an image size of 128px and a blend ratio of 0.5, We varied the perplexity from to 90, In all cases, the generated images fell within or between the input domain images, as expected.
blend-ratio of 0.25 caused the generated images largely overlap with the images from the Male domain. A similar yet reversed pattern was visible with a blend ratio of 0.75, which resulted in more feminine images.
As previously discussed in Section 5 we modified the traditional DCGAN architecture to support any power- of-two image size by scaling the number of layers in the generator and discriminators. To assess the success of this modification, we evaluated MiddleGAN on three image sizes 32px, 64px, and 128px. Figure B shows our results. Our results show that MiddleGAN can produce high quality images across a wide range of image sizes. Additionally, despite having different image resolutions (and thus layers), all three models had similar training losses and all had strong aligument. when visualized with t-SNE. One key observation is that the generated samples fell within the distributions of the input domains at lower resolutions, while the generated samples fell between the distributions of the input domains at higher resolutions. We hypothesize that this is because at lower resolutions, there are not enough pixels to distinguish between masculine and feminine faces.
6.3.1 Blend Ratio Experiments
The inclusion of blend ratio in our model architecture enabled greater flexibility and variety in the range of images that MiddleGAN could produce. In this section we explore the impact that the blend ratio had on generated images as well as the corresponding visualizations. We evaluatod MiddleGAN on three blend ratios 0.25, 0.50, and 0.75. Figure 1 shows our results along with samples of the original input images reference.
When compared to a blend ratio of 0.50, the results shown for the other blend ratios are noticcably different. A blend ratio of 0.25 weighs the loss of discriminator A (which trains on Domain A, female images) at 25% and weighs the loss of discriminator B (which trains on Domain B, male images) at 75%. As such, the resulting images (shown Figure Iap are noticeably more male than images generated with a blend ratio of 0.5, while still being more feminine than the original all-male images. The training loss is also different, discriminator B shows much lower losses than discriminator A. This intuitively makes sense, the loss of discriminator B is more heavily penalized during training, and thus priority for the model to minimize. Lastly, the I-SNE visualization shows shift in distribution, with the generated images falling both within and between the male and female images, as shown in Figure Id. A blend ratio of 0.75 shows similar but inverse changes, with the images, training losses, and t-SNE visualizations all boing biased towards female images (shown in Figures Ic and
6.4 t-SNE & Perplexity Results
Given our reliance on t-SNE as an indicator for the quality of the images generated by MiddleGAN, we wanted to ensure that any results that we observed were not due to chance in terms of hyperparameter selection. Of
15

---

Published in Transactions on Machine Learning Research (12/2024)
t-SNE Visualization
20
t-SWE Dimension 2
-10
Class Names
male
-20
middle
-20
I-SNE Dimension 1
(a) Sample Images
(b) I-SNE Visualization
Figure The preliminary results from our high resolution (256px) implementation MiddleGAN, The generated images have both masculine and feminine qualities when using blend ratio of 0.50. Additionally, the generated images fall within the input domains when visualized with t-SNE.
specific concern to us was the "Perplexity" hyperparameter, which past research has established can have a dramatic effect on the visualizations that t-SNE generates (Wattenberg et al.) 2016 sk] a).
Figure 7 shows the results of three different perplexity experiments we performed, with perplexity values ranging from 5 to 90. We selected this range of values based on the guidance provided in the scikit-learn implementation of t-SNE, which recommends "a value between and 50" [sk] ch. For these experiments, all input images were 128px resolution images, with the generated images created with blend ratio of 0.5. Our results show that perplexity did not have a major impact on the t-SNE visualizations we generated, which is a positive result. We ultimately selected a perplexity value
for all of our other t-SNE visualizations presented in this paper, we found it generated consistently good results. Coincidentally, the default Perplexity value in scikit-learn's t-SNE implementation is 30, which supports our selection [sk]
6.5 High Resolution Results
While our evaluation of MiddleGAN's ability scale to different image sizes presented in Scction 6.3 is thorough, were eager to see if MiddleGAN could produce high quality images beyond 128px resolution. The push to beyond 128px resolution images was prompted by our observation that the original WGAN- GP paper appeared to only test image generation up to 128px resolution (Gulrajani et al.). Given our model's requirement for power-of-two image sizes, the next possible image size for us to generate was 256px resolution. Our initial 256px resolution results are shown in Figure g We believe these results, while preliminary, show positive outcome with both the generated samples and the t-SNE visualization appearing as expected, even with the higher resolution.
6.6 Use Case for the Generated Inter-Domain Images
We present additional experiments using these images as training data for image classification tasks to evaluate their potential utility. Specifically, we applied the generated inter-domain images to a gender classificatiin task on the CelebA dataset.
In the first experiment (Round 1), we trained a basic CNN to classify two genders (male and female) using a training set that included inter-domain images. The model was then tested on standard test containing images of both genders, achieving accuracy of 19.51% In the second experiment (Round 2), we expanded
16

---

Published in Transactions on Machine Learning Research (12/2024)
Without Dotrr- Images
Images With 94.58%
Accuracy
89.51%
Table 3: The classification results on the CelebA dataset's testing set consisting of male and female faces, with and without the inter-domain images. With the data augmentation achieved via having the inter- domain images, we observe an increase in the classification accuracy score of 5.07%. This shows that the inter-domain images, when incorporated into the data augmentation strategy, can increase the performance of the image classification model.
the training set by adding a third "ungoorr-"nsuur'e-rr represent the inter-domain images that blend male and female features. The model was tested on the same two-gender test set from Round 1, but with an additional option: if the classifier identified an image as 'unsure, it could opt not to classify it strictly as male female. This approach led to an improved accuracy of 94.58%. These results indicate that incorporating synthetic inter-domain images increase the flexibility and performance of image classification models by enabling them to beller handle ambiguous cases, as observed in Table 3
6.7 Handwritten Digits Experiments
Our evaluation of MiddleGAN for handwritten digit datasets followed a very similar process to that of the CelebA dataset. We first identified our datasets, trained MiddleGAN to generate images, and then visualized the results using (-SNE. We discuss our process and results below.
Datascts: While our carlier experiments split single datasct into two domains, our handwritten digit experiments leverage two different datasets each one serving as input ton one domain. We selected the MNIST dataset (LeCun et al. 2010), published in 2010, and the DIDA dataset (Kusetogullari et al. 2020bla) a). published in 2020. The MNIST dataset contains over 70,000 black and white cropped and centered images of the digits 0 through 9. The DIDA dataset contains over 250,000 color images of the digits O through 9, sourced from Swedish historical documents from 1800 to 1940 - DIDA is claimed to be the "largest. historical handwritten digit dataset" (Kusetogullari et al. 2020b ab. Unlike the CelebA experiments, in which the domains were both relatively similar (human faces), the DIDA and MNIST datasets are visually distinct and have different levels of complexity (the images in MNIST are at least 3 times less complex then DIDA due to having a single channel of black and white color information).
Image Generation: We trained MiddleGAN using the DIDA and MNIST datasets, specifically images from the class of "7" for both datasets. Samples of both input datasets are shown in Figure g We utilized image size of 32px and a blend ratio of 0.60. Given that handwritten digits are significantly simpler then human faces, we modified many of the hyperparameters shown in Table | to better work for handwritten digits. We selected a blend ratio of 0.60 after observing that blend ratio of 0.50 caused MiddleGAN to only produce MIIST-like images. By adjusting the blend ratio to 0.60 (with DIDA as domain A), we placed more weight on the more complex datasct, which ensured diverse quality of images were produced when MiddleGAN was trained. This observation is important, it means that the blend ratio can be used to compensate for cases when two domains of dataset have radically different complexities.
Feature Extraction & t-SNE: We elected to use the pre-trained GoogleLeNet model provided by PyTorch as the basis of our feature extractor (pyt) b Szegedy et al.) 2014h. We opted for GoogleLeNet over ResNet101 as we felt GoogleL.eNet was better suited for the simpler task of handwritten digit recognition. We fine-tuned the model to provide a binary classification for DIDA vs MNIST digits. From there, we were able to extract feature vectors from the second-to-las layer of the GoogleLeNet model. We extracted feature vectors for the images generated by MiddleGAN, well as both input domains. After performing PCA, we input the feature vectors into t-SNE in order to better understand the relationship between the input images and generated images.
Results:
THe images generated by MiddleGAN during our handwritten digit experiments, as well as associated SNE visualization, are shown in Figure 10 While it is difficult to say if the generated images are truly in
17

---

Published in Transactions on Machine Learning Research (12/2024)
sarole: of magge Yow vense dea.
semples offrnooe from uonelh misio
7
7
7 7
7
7
7
7
7
7
37
7
7
7
7
7
7
7
7
20
7
7
7
%
7
2
7
7
7
7
7
7
7
7
2
49
7
7 7 7
7
7
7
7
2
2
7
7
7
7
7
77
2
7
7
7 7
7
7
7
7
12
22
7
8
7
7
7
7
7 7
7
7
7
7
7
7
(a) Domain A Images (DIDA)
(b) Domain B Images (MNIST)
Figure 9: This figure shows images from both DIDA and MNIST input domains.
ocnernsed images (oped1a0775)
7
7
7
7
7
7
7
3
7
4
7
7
7
1-SNE Visual ization
7
7
7
7
7
7 7 7
7
7
7
7
7
7
7
7 7 7 7
30
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
1
7
7
7
7
7
7
7
7
mist
Dereeated
-30
1.2MC
Dencroan
(a) Generated Images
(b) t-SNE Visualization
Figure 10: This figure shows the generated images for the blended DIDA-MNIST dataset as well as the associated t-SNE visualization.
the "middle" visually from a human perspective, the t-SNE visualization reveals that algorithmically the generated images fall between the DIDA and MNIST datasets. The results of this experiment emphasis the ability of MiddleGAN to blend sharply different domains together in way that will still result in "middle" images when visualized with i-SNE.
7 Discussion & Future Work
In this paper we demonstrated both the thcoretical and empirical results of images generated with Middle- GAN, However, we think there are many opportunities to build upon MiddleGAN. We discuss our existing work and opportunities for future work below.
18

---

Published in Transactions on Machine Learning Research (12/2024)
7.1 MiddleGAN vs. Interpolating Label Embedding for Blending Generation.
One might be prompted to ask the question: Even though the proposed method is feasible, it is possible to simply train a GAN with two domains and interpolate the label embedding for blending generation. What would be the advantage of MiddleGAN? We explain that the advantages of MiddleGAN arc:
MiddleGAN explicitly trains its generator to produce images that blend features from two input do- mains by employing two separate discriminators, ensuring that the generated image is adversarially validated by both domain-specific discriminalors. cont.rast, label interpolation does not enforce inter-domain blending through adversarial training; instead, the generator is tasked with interpolat- ing the learned embeddings, which may not guarantee successful (adversarially validated) blending, as there is explicit supervision over the blend produced by the interpolation.
MiddleGAN's inter-domain images are designed to lie between the two distributions in the feature space, validated by t-SNE visualizations and supported by theoretical results. The architecture is specifically structured to ensure that the generated images fall within this intermediate space, facilitating smooth transition between the domains. Label interpolation, on the other hand, may not guaranlee (hat the generated images occupy this in-belween space in a controlled manner, as interpolation is often heuristic and not explicitly tied to the true data distributions.
While label interpolation is an appealing approach and we agree that it is possible to train a GAN with two domains and interpolate the label embeddings to generate blended images, this method faces a potential issue known as catastrophic forgetting Kirkpatrick et al. (2017) This occurs when the model, trained sequentially on two domains, tends to forget the features learned from the first domain as it adapts to the second. This happens because the weights learned during training on the first domain can be overwritten during subsequent training the second domain. MiddleGAN, in contrast, mitigates this problem by updating the weights of both discriminators and the generator simultaneously during training. This concurrent update ensures that both domains are continuously considered, preventing the overwriting of domain-specific features and thus avoiding catastrophic forgetting. As a result, MiddleGAN is able to generate blended images that more reliably and cohesively capture features from both input domains.
7.2 Comparision with Other Variations of the GAN as Baselines.
Traditionally, considered important evaluate the performance of a new GAN variation by comparing it with other baseline models (existing variations of the GAN). However, we argue that traditional GAN models like DCGAN or StarGAN are not suitable for direct comparison in this case. These models are designed to generate images within single domain or perform domain-to-domain translation, while MiddleGAN is specifically focused on generating inter-domain images that blend features from two distinct input sets.
Since models like DCGAN and StarGAN are not designed to produce inter-domain images, comparing them with MiddleGAN would not offer a meaningful assessment of the core contribution our method the ability create blended images from distinct domains. Therefore, our experiments emphasize demonstrating the unique strengths of MiddleGAN in generating these images and applying them to tasks like classification, which traditional GANs are not equipped to address in the same way.
7.3 Instability on Training.
GANs are indeed known for their instability during training. To mitigate this, we implemented several stabilizing techniques, including the WGAN-GP loss function and adjustments to the training dynamics, such as incrcasing the discriminator-to-ggeaaaaa improvements in training stability in our experiments. However, we acknowledge that further investigation into the stability of MiddleGAN is important. While our current approach has demonstrated stable training across various tasks, including inter-domain image generation, a more thorough exploration of how the model behaves under different hyperparameter settings or with additional discriminators would yield valuable insights.
training ratio. These mcasures have resulted in noticcable
19

---

Published in Transactions on Machine Learning Research (12/2024)
In experiments, we also observed reduction in mode collapse by incorporating adaptive weighted discriminator Zadoroshnyy et al. (2021). which contributed to more consistent convergence across multiple training runs. Initially, without the WGAN-GP loss function, mode collapse occurred frequently during training. Howover, after introducing both the WGAN-GP loss and the adaptive weighted discriminator, mode collapse was significantly reduced, with very few occurrences. These findings show that while progress has been made, there is still potential for further optimization to enhance stability.
7.4 Limitation of CelebA Dataset.
Throughout this work, we have primarily used the CelebA dataset with a split on the "male" attribute to create the "male" and "female" input domains. Although we achieved strong results using these domains, there are some inherent limitations of our approach and we caution against similar blending being blindly applied lo future applications. Firstly, the CelebA dataset comprises images of celebrities, meaning our MiddleGAN model learns to generate faces resembling celebrities rather than everyday people. While we expect that MiddleGAN would perform equally well with photos of non-celebrities, it currently generates new celebrity-like faces. Secondly, and perhaps more importantly, MiddleGAN is learning stereolypica masculine and feminine traits rather than the true essence of being "male" or "female". This could potentially reinforce gender stereotypes if misused in the future, despite the model's ability to demonstrate the blending capabilities of MiddleGAN. We emphasize that MiddleGAN, like any tool, has the potential for artistic and other applications. However, is crucial to remain mindful that certain blending combinations (e.g-- different races) may produce uncomfortable inappropriate results.
7.5 Expanding to N-Domains.
While MiddleGAN currently supports the blending of two-domains, this is not hard limitation. In fact, MiddleGAN could be modified to support N-domains by modifying the training code to support one dis- criminator per input domain. However, while this is theoretically possible, we anticipate larger losses for, and the eventual collapse of, MiddeleGAN's generator if too many different domains were included. We anticipate this happening as the generator would be unable to create a single image that would satisfy all discriminators. Despite this anticipated challenge, we still believe that this area research would be worth future exploration.
8 Conclusion
In this work, we introduced MiddleGAN, a novel variation of the traditional GAN capable of generating images in between two distinct input domains. MiddleGAN leverages two discriminators and one generator create images that appear to be both domains, resulting in blended images. The thcorctical basis for MiddleGAN was covered in detail, in additional the empirical evaluation. We extensively evaluated the capabilities of MiddleGAN Image Sizes (32px, 64px, and 128px). Our evaluation with the CelebA dataset showed the blended nature of the generated images, which contained both masculine and feminine features. In our handwritten digit based experiments, we revealed that with some adjustments of the Blend Ratio hyperparameter, MiddleGAN could handle datascts with radically different complexitics (MNIST being "low" complexity and DIDA being "high" " complexity). We concluded our work by outlining future opportunities MiddleGAN as well as the limitations and caveats of MiddleGAN.
the CelebA dataset across three Blend Ratios (0.25, 0.50, and 0.75) and three
References
VTTA-group/AutoGGA. a. URL httpp:///gthhbb.co//////CCCoo//p 01120:48:332.
original-date: 2019-08-
VITA- TTaaSSAA,, b. http:://gghuubb.co////////oooo 10:48:11:547.
original-date: 2021-02-
20

---

Published in Transactions on Machine Learning Research (12/2024)
Common problems machine learning. URL httpp//////eeeebb...........o//p problems
itertools functions creating iterators for efficient a. httpa//t/////oosspoooooo////////
itertools.htm
googlenet
- torchvision main documentation, b. URL https//ppporrd..or/////s0o/h1h/1
generated/torchvlssin models, googlenet..............o html
rosnet101 torchvision main documentation, ggmertteel/oorrrviiiioo.o...... model:iiiiiiimmoooll.......mm.ee rww.ereeeeneiiieee
URL
http:///ppoordd..rrr//////////12
t-sne: The effect of various perplexity values on the shape, a. UtL/ttps///bblltlloo////bb//r/b
exampless/mamiffllllll... t same__erplexity.hie
sk skeern.decomppsitioo..CC..
b. URL http:///sckkkkeeeaaaaaa//cllllls
Deccoppssttion.PA..hee
Illearm.manifold.cm c. URL http:://wchh...aoo///ooooooollla html
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN. URL http::/arxiv..rg///
1701.07875
Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. 42(4):1 11. ISSN 0730-0301, 1557-7368. doi: 10.1145/3592450. URL https://dl acm. brgg//o//1..114//355533
Eva Cetinic and James She. Understanding and creating art with AI: Review and outlook. 18(2):66:1 66:22. ISSN 1551-6857.
doi: 10.1145/3475799 URL https://dl. acm acm.
Ishan Durugkar, Ian Gemp, and Sridhar Mahadevan. Generative multi-adverrsaril networks.
URL http:
//axxiv. waw.ii.org/abba........ http:/abs/6611..115
Emma Taggart. Digital artist. creates celebrity face mashups that look like new people.
URL
https:
lan Goodfellow, Jean Pouget-Abadis Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron
Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Newral Information Processing Systems, volume 27. Curran Associates, Inc. URL Itttps://proceediiii htw.oocoeiinnn....rrripss.. neurins couurips.ccppppr.... cc/ files// html
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of wasserstein GANs. URL
Corentin Hardy, Erwan Le Merrer, and Bruno Sericola. MD-GAN: Multi-discriminator generative adversarial networks for distributed datasets. In 2019 IEEE International Parallel and Distributed Processing Sympo- sium (IPDPS), pp. 866-877. doi: 10.1109/ M0L100/IIPPSS IPDPS HDDDP..2019.000 201..00955. URL http:://aaax wwwp:/wwwwrr//b///////// 03850
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
URL http:// arxiv. orw.obs//aaaaeee.e 1etp./151....00 03385
Yifan Jiang, Shiyu Chang, and Zhangyang Wang. TransGAN: Two pure transformers can make one strong GAN, and that can scale up. URL http:///axivv http:///aaxxi..o///aa/////// wttp://bbbbb/12..00077/////
James Jordon and Jinsung Yoon. PATE-GAN: GENERATING SYNTHETIC DATA WITH DIFFEREN-
TIAL PRIVACY GUARANTEES.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barwinska, al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521 -3526, 2017.
21

---

Published in Transactions on Machine Learning Research (12/2024)
Huseyin Kusetogullari, Johan Hall Amir Yavariabdi, and Niklas Lavesson. Dida: The largest historical handwritten digit dataset with 250k digits. http:///hhb..mc/ttaaaaaaaee//// 2020a.
Huseyin Kusetogullari, Amir Yavariabdi, Johan Hall, and Niklas Lavesson. Digitnet: A deep handwritten digit detection and recognition method using a new historical handwritten digit dataset. Big Data Research, 2020b.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Avoilable: http://yann..
http://yanm.eeenn.oom wew.nnn.leec...om cow.tt..cc//aaa cwmmerdo/ serddb/mmise 2, 2010.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedtings of International Conference on Computer Vision (ICCV), December 2015.
Jaemin Na, Heechul Jung, Hyung Jin Chang, and Wonjun Hwang. Fixbi: Bridging domain spaces for unsupervised domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1094 1103, 2021.
Tu Nguyen, Trung Le, Hung Vu, and Dinh Phung. Dual discriminator generative ad- versarial nets. In Advances in Newral Information Processing Systems, volume 30. Cur- ran Associales, Inc. URL https://proceeding: neurips cttpssccc/pppee /illes/paprr2211/aahh
4 http:///22...eaaa22aeaarrr.a.a.. Abstract.
Weili Nie and Ankit B. Patel. Towards a better understanding and regularization of GAN training dynamics. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, pp. 281 291. PMLR. URL
https://proceeddings mlr. wtt.:pere///11//iee00000 html
ISSN: 2640-3498.
Frank Nielsen. On generalization of the jensennshannene shannon divergence and the jensen shannon centroid.
Entropy, 22(2))221 2020.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolu- tional generative adversarial networks. URL htt.:ppp/aarii...r//////11.....1
Mohammad Mahfujur Rahman, Clinton Fookes, Mahsa Baktashmotlagh, and Sridha Sridharan. Multi- component image translation for deep domain generalization. In 2019 IEEE Winter Confer- ence on Applications of Computer Vision (WACV), pp. 579-588. doi: 10.1109/WACC) 2019.00009.00067. URL https::/isseepplort ottp:///......aaaaao/ooooooo//aa ottp:////e..ao/oooooooooooooooa.
wag.ahacchhiiiiittaSSSSsiiiiiate ISSN: 1550-
5790.
Swami Sankar, Yogesh Balaji, Arpit Jain, Ser-Nam Lim, and Rama Chellappa. Unsupervised domain
adaptation for semantic segmentation with GANs.
Ahmet Sangünn. Re-implermnntation of TransGAN; Two transformers can make one strong GAN. URL
Jascha Sohl-Dickstein, Fric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru
Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions, 2014. URL https:
//arxiv.org/ httgg/bbs14409 4842
PyTorch Tutorials. DCGAN tutorial. URL http:///goeeeaa.gghhhhhhanaagggg ffcee faces butorial.html
Haoxiang Wang, Bo Li, and Han Zhao. Understanding gradual domain adaptation: Improved analysis, optimal path and beyond. In International Conference on Machine Learning, pp. 2778422881., PMLR, 2022.
22

---

Published in Transactions on Machine Learning Research (12/2024)
Martin Wattenberg, Fernanda Viégas, and Ian Johnson. How to use t-sne effectively, Distill, 2016. doi: 10.23915/dissiil.0000/ URL
Huikai Wu, Shuai Zheng, Junge Zhang, and Kaiqi Huang. GP-GAN: Towards realistic high-resolution image
blending. URL http://awww..//r/////////////010
Depeng Xu, Shuhan Yuan, Lu Zhang, and Xintao Wu. FairGAN: Pairness-aware generative adversarial
networks. URL http://arxiv. org/abs/1805. 11202
Vasily Zadoroshnyy, Qiang Cheng, and Qiang Ye. Adaptive weighted discriminator for training generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recoguilion, pp. 4781-4790) 2021.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial worworks. URL http///axvvvv...raaa/ 10593
23


Reviews:
[{"review_id": "qtKep9VJNc", "reviewer": "Reviewer_4MKx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The topic of blending inter-domain images from two distinct input sets is engaging.\n\n- The writing style is clear and makes the paper easy to understand.\n\nWeaknesses:\n\n- The proposed method, while simple, may not provide significant advancements compared to typical GAN approaches. It just introduces another discriminator in comparison with typical GAN.\n\n- As the training of GAN is often unstable, how the proposed method containing two discriminators suffers from the instability. Although the authors attempt to address the issue by modifying the network structure, the instability should be further studied.\n\n- The authors use a blend ratio to control the strength of the two domains. However, the reviewer thinks the strength is not only related to the weights but also the distributions and the used data (like the number of training samples in each domain), which are not analyzed in the paper. In addition, how to select appropriate blend ratio should be studied. It is troublesome to try different values.\n\n- Although the reviewer thinks this task is interesting, the cases in the experiments do not convey the values. More practical cases should be selected for a more impactful study.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a method called MiddleGAN that aims to blend inter-domain images from two distinct input sets. In contrast to the typical GAN, MiddleGAN introduces a second discriminator to force the generator to create images that fool both discriminators, capturing the qualities of both input sets. The method is validated through experiments on the CelebA and Handwritten Digits datasets (DIDA and MNIST), with visualization results (t-sne) provided to demonstrate the learned distributions.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- More experimental analysis.\n\n- Improve the contributions.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "G1l7qXbn6z", "reviewer": "Reviewer_5mDC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The generation of inter-domain images remains a relatively unexplored area. The concept of using an additional discriminator to guide the generator in synthesizing images that lie between two domains makes sense to me.\n\n2. The qualitative example and tSNE embedding show that the model successfully generates inter-domain images, and the theory analysis shows that the optima of the generator is \n\nWeakness: \n1. The realism and diversity of the synthesized images are not quantitatively evaluated. For example, the authors could calculate the FID or KID scores on the Celeba dataset to evaluate whether the synthetic images still resemble human features. The reason I'm asking for it is that I noticed on Fig. 10 (a), the edges of some images contain artifacts.\n\n2. The authors could conduct some experiments to demonstrate the application of synthetic inter-domain images. For example, the authors may use the generated images for data augmentation for supervised classifier, it may enhance the model robustness.\n\n3. It would be good if the authours could add some baseline methods for comparison.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose a method for generating images that blend styles from two different domains. Specifically, a second discriminator is introduced to encourage the generator to fool discriminators from two different domains. In addition, the authors conducted theory analysis to prove that the proposed method is essentially looking for the JSD controid of the source domain distribution and the target domain distribution. The authors performed experiments on multiple datasets, including CelebA, DIDA and MNIST datasets. The results show that the proposed method can successfully generate inter-domain images.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Quantitative evaluation of generated images. For example, the authors could use FID and KID scores to evaluate whether the synthetic images still resemble human features.\n\n2. The authors could perform experiments to showcase the application of synthetic inter-domain images. For instance, they could apply the generated images for data augmentation in a supervised classifier, which may improve the model's robustness.", "broader_impact_concerns": "I didn't find ethical concerns for this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pEQcwqr0eo", "reviewer": "Reviewer_yZV8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. The idea is interesting that the end2end training framework enables the generation of unseen distribution , which is the mixing between the styles from two domains.\n2. The insight from the theory can further support the formulation of the proposed method.\n\nCons:\n\n1. Even though the proposed method is feasible, we can just train a GAN with two domains and interpolate the label embedding for blending generation. What would be the advantage of the proposed method?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new framework for blending generation which will generate inter-domain data. And the authors identify the feasibility of the method through theoretical analysis and experiments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please answer the question above.", "broader_impact_concerns": "There is no obvious ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new method to allow GANs to generate images between different distribution, which is called domain interpolation in the paper. To this end, this paper introduces a second discriminator and also uses a blend ratio hyperparameter to control the weighting of input sets. The main concerns from reviewers are 1) the advantage compared to exisiting GANs may not be significant, 2) the incorporation of an additional discriminator might cause instability in training. The authors have addressed these concerns in the rebuttal. While one reviewer still has remaining concerns on the potential application of the proposed MiddleGAN. However, I believe this paper already offers sufficient contributions, and its potential applications are likely to become clearer in the near future. Therefore, I recommend accepting this paper.
Decision: Accept (poster)