PaDPaF: Partial Disentanglement with Partially-Federated GANs
Authors: Abdulla Jasem Almansoori
Samuel Horváth
Martin Takáč
Venue: Accepted by TMLR ()

Abstract: Federated learning has become a popular machine learning paradigm with many potential real-life applications, including recommendation systems, the Internet of Things (IoT), healthcare, and self-driving cars. Though most current applications focus on classification-based tasks, learning personalized generative models remains largely unexplored, and their benefits in the heterogeneous setting still need to be better understood. This work proposes a novel architecture combining global client-agnostic and local client-specific generative models. We show that using standard techniques for training federated models, our proposed model achieves privacy and personalization by implicitly disentangling the globally-consistent representation (i.e. content) from the client-dependent variations (i.e. style). Using such decomposition, personalized models can generate locally unseen labels while preserving the given style of the client and can predict the labels for all clients with high accuracy by training a simple linear classifier on the global content features. Furthermore, disentanglement enables other essential applications, such as data anonymization, by sharing only the content. Extensive experimental evaluation corroborates our findings, and we also discuss a theoretical motivation for the proposed approach.

Published in Transactions on Machine Learning Research (04/2024)
PaDPaF: Partial Disentanglement with Partially-Federatedd GANs
Abdulla Jasem Almansoori
Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, UAE
Samuel Horváth
semmelLhorrtth:imbbaaau..cc
Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, UAE
Martin Takáč
martin, www.kakkkwbbbaa......o
Mohomed bin Zayed University of Artificial Intelligence Abu Dhabi, UAE
Reviewed on OpenReview: /ttps://openreer opw.reieee...v.e... net/forun' id- usez76EAV6
Abstract
Federated learning has become a popular machine learning paradigm with many potential real-life applications, including recommendation systems, the Internet of Things (IoT), healthcare, and self-driving cars. Though most current applications focus on classification- based tasks, learning personalized generative models remains largely unexplored, and their benefits the heterogeneous setting still need to be better understood. This work proposes a novel architecture combining global client-agnostic and local elient-specific generative models. We show that using standard techniques for training federated models, our proposed model achieves privacy and personalization by implicitly disentangling the globally consistent representation (i.c. content) from the client-dependent variations (i.c. style). Using such decomposition, personalized models can generate locally unseen labels while preserving the given style of the client and can predict the labels for all clients with high accuracy by training a simple linear classifier on the global content features. Furthermore, disentanglement enables other essential applications, such as data anonymization, by sharing only the content. Extensive experimental evaluation ion corroborates our findings, and we also discuss a theoretical motivation for the proposed approach.
1 Introduction
Federated learning (FL) Konečný et al., 2016a b) a recently proposed machine learning setting where multiple clients collaboratively train a model while keeping the training data decentralized, i.e. local data are never transferred. FL has potential for the future of machine learning systems as many machine learning problems cannot be efficiently solved by a single machine for various reasons, including scarcity of data, computation, and memory, well adaptivity across domains.
Most of the focus of federated learning has been on classification models, mainly for decision-based applications. These applications include a wide variely of problems such as next-word prediction, out-of-vocabulary word and emoji suggestion, risk detection in finance, and medical image analysis (Wang et al. 2021 Kairouz et al, 2019). However, based on recent works on representation learning and causality (Schōlkopf al. 2021 Wang & Jordan 2021 Mouli & Ribeiro 2022), we believe that in order to understand the causes, explain the process of making decision, and generalize it to unseen domains of data, is crucial, if not necessary, to

---

Published in Transactions on Machine Learning Research (04/2024)
learn generative model of the data. A representation of the data (e.g. its description features) can allow us to make decisions directly from it without seeing the data itself. For example, a description of what a digit. looks like is sufficient to infer a previously unseen label. Therefore, by collaboratively learning a generative model and communicating with other clients having the same representations, the clients can potentially generalize to the unseen domains of the other clients. In general, the potential of a group of specialized machines that can communicate efficiently could be superior to that of a single monolith machine that can handle every task. These insights motivated us to propose the ideas presented in our work.
We take step towards this objective and introduce core idea, which can be described follows: to learn generative models of heterogeneous data sources collaboratively and to learn a globally consistent representation of the data that generalizes to other domains. The representation should contain sufficient information to classify the data content (e.g. label) with reasonable accuracy, and the generative model should be able to generate data containing this content under different domains (e.g. styles). Thus, we make no assumptions about the availability of labels per client and the interscction of the clients' data distributions.
A direct application of our model is to disentangle the latent factors that correspond to the content and learn simple classifier on top of it. This approach tackles the problem of domain adaptation (Zhang et al. 2013), where, in our case, the domain is the client's 's local distribution. In this scenario, our model can learn a client-agnostic classifier based on the global representation, which should generalize across all clients. We call the client-agnostic part of the representation the content, and the private part- the style, so we partially disentangle the latent factors into content and style factors (Kong et al., 2022). Furthermore, the generator's data samples from the client's distribution can be used for data augmentation or as synthetic data for privacy. We also actively remove private information any client-identifying variations from the client's data by introducing a "reference" client that trains only standardized, publicly available data with no privacy concerns. This way, we can anonymize private data by extracting its content and generating a sample with the same content but with the reference client's style.
The generative model we use in this work is Generative Adversarial Networks (GANs) Gooddellow et al., 2020). In particular, we use the style mapping idea from StyleGAN (Karras et al. 2019). In the case of supervised data, we condition the GAN using a projection discriminator (Miyato Koyama 2018) with spectral normalization MMyyato et al.) 2018) and a generator with conditional batch normalization (De Vries) et al. 2017). We believe that designing more sophisticated archilectures can improve our results and make them applicable to a broader variety of problems.
Contributions. Below, we summarize our contributions:
We propose our framework for Partial Disentanglement with Partially-Federatte Learning Using GANs. Only specifie parts of the model are federated so that the representations learned by the GAN's discriminator can be partially disentangled into global and local factors, where we mainly concerned with the global factors, which can be used for inference. We further enforce this disentanglement through "contrastive" regularization term based on the recently proposed Barlow Twins (Zbontar et al. 2021).
Our model can learn a globally consistent representation of the content of the data (e.g. label) such that it
classifies the labels with high accuracy. In the case of supervised data, our model can generalize to locally unseen labels in terms of classification and generation.
Through extensive experimentation, we validate our claims and show how popular techniques from federated learning, GANs, and representation learning can seamlessly blend to create a robust model for both classifications and generation of data in real-world scenarios.
We make the code publicly available for reproducibility at hattgs//ggttau..ccom/allggg/////
Organization. Our manuscript is organized as follows. Section 2 talks about related work and the novelty of our approach. Section 3 describes notations and preliminary knowledge about the frameworks used in our model. Section ! contains the main part of our paper and describes the model design and training algorithm in detail. Section 5 shows a detailed evaluation of the model's generative capabilities across different domains, robustness under limited label availability, and generalization capabilities based on its learned representation. Section 0 concludes the manuscript by providing exciting future directions and summarizing the paper.
2

---

Published in Transactions on Machine Learning Research (04/2024)
2
Related Work
Training GANs in the FL setting is not new. For example, Fan & Liu 2020 either the generator or the discriminator, but their training algorithm has no client-specific components or disentanglement. Similarly, FedGAN (Rasouli et al., 2020) also proposes a straightforward federated algorithm for training GANs and proves its convergence for distributed non-iid data sources. Another work exploring distributed GANs with non-fid data sources is (Yonetani et al. 2019), which interesting because the weakest discriminator can help stabilize training. Still, general, their framework does not align with
considers the options of averaging
ours.
The idea of splitting the model into two more sub-modules where one focuses personalization or local aggregation has been explored before under the name of Split Learning. Very few of these works consider a generative framework, which is the core consideration of our work. For example, FedPer (Arivazhagan el al. 2019) and ModFL (Liang et al. 2022 suggest splitting the model depth-wise to have a shared module at the beginning, which is trained with FL, and then a personalization module is fine-tuned by the client. Inversely, (Vepakomma et al. 2018) personalize early layers provide better local data privacy and personalization. A more recent work (Pillutla et al. 2022) proposes the same idea of partial personalization based on domain expertise (e.g. in case, setting style-related modules for personalization). HeteroFL (Dioo et al. 2020 is another framework for addressing heterogencous clients by assigning different levels of "locality" where the next level aggregates all parameters, excluding ones from the previous levels so that higher levels are subsets contained in the earlier levels. Further, (Horváth et al. 2021) trains orderly smaller subsets of weights for devices with fewer resources. This is similar in spirit, as we aim to decompose the parameters based on how they are federated.
Splitting parameters is one way of personalizing FL methods. Another well-known approach is Ditto Li et al. 2021ah, which trains a local model with proximal term that regularizes its distance to global model trained with FedAvg (without the regularizer). Similarly, FedProx (Li et al. 2018) adds the proximal regularization in the global objective without maintaining a local state. Other approaches work by clustering similar clients (Ghosh et al. 2020 Werner et al. 2023) or assuming a mixture distributions (Marfoq et al, Wu et al. 2023), which can be relevant to our Tramework when many clients share the same style.
2021
Regarding the privacy of federated GANs, (Augenstein al. 2020 considers a setup where the generator can be on the server as it only needs the gradient from the discriminator the generated data. However, they are mainly concerned with resolving the issue of debugging ML. models on devices with non-inspectable data using GANs, which is a different goal from ours. Though they mention that the generator can be deployed entirely on the server, the discriminator still needs direct access to data.
Combining GANs with contrastive learning has also been explored before. ContraGAN (Kang & Park 2020 is a conditional GAN model that uses a conditional contrastive loss. Our model is not necessarily conditional, and we use & contrastive loss a regularizer and train GANs regularly. Another work (Yu al. 2021 trains GANs with attention and contrastive objective instead of the regular GAN objective, which again different from our implementation. Perhaps the most relevant is ContraD (Jeong & Shin 2021), contrastive discriminator that learns representation from which parts are fed to different projections for calculating a GAN objective and a contrastive objective. Our work differs in many details, where the main similarity to prior work is the usage of contrastive learning on a discriminator representation.
3 Preliminaries
This section describes the frameworks we employ to design and train our model, namely federated learning and generative models. We will also explain the notations we use detail remove ambiguity. Generally, we follow prevalent machine learning notation and standard notations used in federated learning (Wagg et al. 2021 and GANs (Miyato& & Koyama 2018).
3

---

Published in Transactions on Machine Learning Research (04/2024)
pi(x")
G!
Pi
(x)
q: (x)
Pi(x 22, x²)
Discriminative
(x)
pi(x))
G
pi(x|2²,,'))
Contrastive
g;(y) pi(r|2²)
Figure 1: The architecture of our model. Federated modules are aggregated in each communication round (shown green, solid blocks), whereas private modules are not (shown in rod, dashed blocks). The discriminative loss just the GAN loss as described Sec. 4.1 whereas the contrastive regularization term is explained in more detail Sec. 4.3 We describe the model's components in full detail in the A Appendix.
3.1 Federated Learning
Federated learning is a framework for training massively distributed models, where a server typically orchestrates the training that happens locally on the models deployed on user devices (i.e. clients) (Kairouz et al. 2019h. The most widely used algorithm for training models in such a setting is FedAvg (McMahan
et al. 2016), which operates by rumning stochastic gradient descent the client's models for few steps and then aggregating the updates on the server. Another popular algorithm that accounts for data heterogeneity is FedProx (Li et al.) 2018), which adds a proximal term to the objective that regularizes the local model to be close to the global model instead of aggregation.
The goal federated learning is to optimize the following objective
F(0) IE [F(O)),, E((0) E [f((0:E], C~D,
(1)
where 0 E Idd is the parameter, is the client distribution, i the index of the sampled client, and & is a random variable of the local distribution D; of client i. Due the nature of our framework, we also denote 0 the parameters at client i. We reserve the client index 0 for the server. For example, the client distribution P could be proportional to the size of the local dataset. The local distribution D, could be minibatch distribution on the local dataset (();,,))) ) so that & D, is a subset of indices of minibatch.
Note that the notation for the distributions in this work will be based on the generative model literature to avoid confusion. Namely, (x(xy)) = q(x,y) y/i) the target (data) distribution for client and (x 0) the generative model for client i, where we often hide the dependence on for clarity.
Private Modules. Our model uses private modules parameters, as in (Bui al. 2019). Namely, for each client i, we split the parameters of its model 0; into private part g. and federated part 6fed typically in non-trivial, structured manner that is specific to the model design. The only difference between these two parts is that the federated parameters are aggregated, whereas the private parameters are kept as is.
3.2 Generative Adversarial Networks
Generative Adversarial Networks (GANs) (Goodfellow et al. 2020) are powerful generative models that are well-known for their high-fidelity image generation capabilities [Karras et al. 2019) among other applications. The framework consists of two competing models playing a minimax game: a generator and a discriminalor. The generator's s main objective to generate samples that cannot be discriminated from real samples.
X' and the discriminator be D X - R, for some data space X' and latent space 2. Further, let p(z) be the latent prior
To put it formally, let the true distribution over the data be qxx), the generative model be G
4

---

Published in Transactions on Machine Learning Research (04/2024)
distribution, which is typically standard normal MVO, Thus, the GAN objective is ming maxp V) where
V(D,))= = E ( + E log(1 - D(G(z))). p(z)
(2)
o(x)
Conditional GANs. II labels y are available, the data space becomes X' x } where ) can be {0, 10 for set of dy classes, for example. We can define generative distribution over data G(z) = p(x, ylz)p(z)) where we can factor ylz) = p(xly, z)p(y/z), for example. So, we can rewrite the GAN objoctive with an implicit generator as follows
V(D,G) E E log(D(x,y)) + p(yp((l()
log(1 y)x.
The method we adopt for conditioning is the projection discriminator Miyato & Koyama 2018 First, let us examine the unconditional case in more detail. We can write D(x) = (A lb (x( a decomposition of the discriminator, where A is the activation (c.g. AA is the sigmoid function in (3), ) 1p R + is a linear layer, and ((x): + returns the features of x. Note that. = sigmoid implies D(x) = p(x).
Then, the basic idea of adding conditional variable y is to model p(y (x) as log-lineat model in ((x)
logply y|x) == (vz) log (x)))),
(4)
where Z(o(x)) := Eye exp (() Tox)) is the partition function and f is an embedding of y w.r.t. p. The authors observed that the optimal D has the form D = A(loog q y)/ ///log/'''' (x, (Goodfellow et al. 2020 so given and an embedding V, they reparameterized D as
Dxx,y) A (yTVo(()) +(d())))
(5)
The choice A sigmoid is used in the vanilla GAN objective, and A = id is used for the Wasserstein metric Guulaaaai et al. 2017). We use the latter in our implementation.
3.3 Supervied Learning
Self-supervised lcarning (SSL) is a modern paradigm in which the supervision is implicit, either by lcarning a generative model of the data itself or by leveraging some invariance in the data (Liu et al. 2021). One well-known method uses a contrastive objective (Le-Khac et al. 2020). which pulls similar data together and pushes different data away.
Let us describe a general contrastive loss, which includes InfoNCE (Oord et al. 2018 and NT-Xent (Chen) et al. 2020). Let & be the feature of x, and let and x be two similar data. Then, given similarity metric "sim" e.g. the inner product, and a temperature v, we have
explsim(d(x) 117)) Ccontrast (x:x*) log F_- (xx(sim(@(x)).l 1//7)
(6)
The need for mining x has been mitigated with recent advances such as moving-average encoders (He et al. 2020) or stop gradient operators (Chen & He 2021). One technique of particular interest. is Barlow Twins (Zbontar et al.) 2021), which optimizes the empirical cross-correlation of two similar features to be as close to identity as T The empirical cross-correlation between two batches of (normalized) projected features A g(o(x)) and B g(o(x+)) for some projection g is defined as
E, Bo. C(A,B)is = CAW V, (B,,)
Now, the objective of Barlow Twins is LBT(O(x), = E, XBT Ei
(8)
where we omit. the dependence of C on g(o(x)) and g(((x+)) for clarity. The hyper-parameter AET controls the sensitivity of the off- -ff-diaan "redundancy" terms and is often much smaller than 1.
Which maximizes correlation and minimizes redundancy, thus the name "Barlow Twins"
Barlow
2001
5

---

Published in Transactions on Machine Learning Research (04/2024)
4 Partial Disentanglement with Partially-Federated GANs
In this section, we present framework for Partial Disentanglement with Pertially-Federated (PaDPaF) GANs, which aims to recover the client invariant (content) representation by disentangling it from the client- specific (style) variation. This is done by a contrastive regularization term that encourages the discriminator to learn a representation invariant to variations in some latent subspace, ie., given a content latent, the representation should be invariant to variations in style latent.
4.1 Federated Conditional GANs
Starting from the GAN framework in Sec. 3.2 we aim to optimize (3) in the federated setting with respect to the parameters of D; and G; per client i. The discriminator architecture is based on (He et al. 2016)
with the addition of spectral normalization (Miyato et al. 2018). In addition to the global discriminator, we also introduce a smaller, local version of the global discriminator for each client. The global generator is a style-based ResNet generator following Gulrajani et al. (2017) with the addition of a local style vectorizer for each client (Karras et al. 2019). The generator is conditioned on the output of the style vectorizer through conditional batch vormalization layers (De Vries el al. 2017 that are kept local too (Li et al. 2021b
This split. between local and global parameters is done in a way such that the domain-specific parameters are private and solcly optimized by the clients' optimizers, whereas the "content" paramcters are federated and optimized collaboratively. Formally, for each client i, we introduce Df y) X x y' IR the "content"" discriminator, and (x) : X + R is the "style" discriminator. Similarly, let the content (global) generator be (2) y) 2 x S x y + X' and the the style vectorizer be G 2 St that s; (z)). For simplicity, assume that D and Dj have the same architectures, but this is not necessary. Let the content featurizer be of = - Rdd, and similarly the style featurizer of : X + Rdd, For simplicity, we can assume that d, = do- Note that of is a part of DE, and of is a part of D as explained in Sec. 3.2 We make the dependence of these modules on their parameters implicit unless stated otherwise. Now, we rewrite the
client objective to follow the convention in
E((0:E)= = E (xx,(D((x,)) + log( X.YEE
+
E (-
[log(1 - (x 3)) + log(1 - Dx)))),
(9)
where & y). Here, we make the dependence on 0 explicit for the generative model p; because Px( F; = (2, (;(2)))) where 2", 2t ~ ~N(0 7) and y ~ Unil()).
4.2 Partially-Federated Conditional GANs
When the client's data distributions not (i-ih is the case in practice the client's domain can act as confounding variable. In our case, we assume that each client has its own domain and unique variations, as in a cross-silo setting. In practice, especially when we have millions of clients, we can cluster the clients into similar domains and treat them as silos. The implementation of this idea will be left for future work.
Before we procced, we make simplifying assumption about the content of y.
Assumption 1 (Independence of content and style) Given x ~ q(x), the label y is independent from client i, i.e. we gain no information about y|x from knowing i. Namely,
q. ( = g(y(x).
(10)
It is worth mentioning that this assumption might not be entirely true in some practical situations, particularly when "personalization" is needed. Indeed, label prediction personalization helps exactly when our assumption does not hold. For example, in the case of MNIST, where client A writes the digits 1 and 7 very similarly to each other, and client B writes 1 similarly to client but writes 7 with a crossing slash such that is perfectly distinguishable from 1. Then, sample of the digit without a slash would be hard to distinguish from 1 for client A but easily recognizable as 1 for client B. Hence, given x, there could be some information in i that can help us predict y, which is what personalization makes of. However, we make the simplifying
6

---

Published in Transactions on Machine Learning Research (04/2024)
z'
y
z
Figure 2: A causal model with the generative latent variables z" and z² of x. The dotted arrows show the underlying cansal model without latents. Both latents depend on the client i, and we assume that the content. latent z generates the label y as well. The blue arrows drop when we do(z"), and the red arrows drop when do(z"). The dashed arrows illustrate the reduced inffuence of i over the mechanism i + z we federated averaging. x² and its mechanism i + x", on the other hand, remain specific to the client in our case.
assumption that the global representation of the content of x should be, in theory, independent from the local variations introduced by the client.
Given this assumption, we would like to construct our base generative model such that p;(y)x) =- p(y|x,)) i) is invariant to and p((x)= == px generates data from domain i. Let p(i) be the probability of sampling client i, which we assume to be proportional to the number of data in client i, and p(y) the prior of the labels, which we also assume to be proportional to the global occurrences of y. Note here that the label distribution for a given could be disjoint or has a small overlap with different i's. This is particularly true in practice, as clients might not have access to all possible labels. More importantly, it implies that x can depend non-trivially on as well, e.g. through data augmentations (more details on the construction in the Appendix). However, we want our model to be agnostic to the client's influence on y/x for a globally consistent inference while being specific to its influence on for data generation personalization.
We enforce the independence of y from i in (y(x)) via our model construction and split of private and federated parameters. This split influences the learned generative mechanism z²,z² + The generative mechanism from z -> s is private and dependent on i, while the generative mechanism from z + x is also dependent on i but becomes independent as we run federated averaging. The causal model we assume is shown in Fig. ?
Generative vs. discriminative. We can write the generative probability as
z² (z)
(11)
which describes the generation process starting from the right- term. We can also factor the probability distribution in a discriminative form as
g(z" x, y, ()g(z")) |x, (x) (1))
(12)
which also describes the discriminative process from right to left. Note that we have access to the sampling distribution q()), local datasets y(x,y)i), and latent distributions (i2% and q(z"))).. It remains to estimate the generators q(ylz") and qx|2z", 2²) and the encoders encodens/discrimmnate /disrrmminatoosss q(2") xx, y, i) and q(z*|x, a) with the model 0). We can generate p(x|2", z²) in a straightforward manner with GAN training and federated averaging. We can also generate augment d with a variable ý ~ Unif())), and then set y y. Recall (4) and note that we maximize p(yix) as log-linear model in the global features (x). We now make another simplifying assumption.
(z) by sampling labels uniformly. This is possible because we
Assumption 2 (Feature to latent map) There exist functions 3 Rdde y + 2 and 3" Rd. 2 such that
x)) ss 22, and (x) 3
(13)
The functions 3r and 3 can be thought of as an approximate inverse of G and G", respectively. 3 and 3* are invertible, then we can transform the random variables exactly. Otherwise, we can learn an estimate of xx,")x,,,)) and pxz*|x, i) by knowing (x) and (x(x). In Appendix C.2.3 we show how we can estimate 3r and 3 with simple models in an unsupervised manner. One benefit of this is transferring the content.
7

---

Published in Transactions on Machine Learning Research (04/2024)
of an image from client's domain to another, which includes anonymization special case. We look at the problem of inferring 2", x given y, from client i as a problem of "inverting the data generating process" (Zimmermann et al. 2021). We will next show how we can encourage disentanglement between z and z" using a contrastive regularizer on the discriminators.
Before we move on, note that due federated averaging, we have 0 parameter of G. in client and the number of local steps. This applies all aggregated parts of the model. Thus, assuming that global optimal solution 02 exists, we have p(z])x, xx i; 0 ' p(z))x; 0-)), and the same goes for all federated parameters.
Epcc where th is the
4.3 Contrastive Regularization
The main molivation for partitioning the model into lederated and private components is that the lederated components should be biased towards a solution that is invariant to the client's domain i. This is a desirable property to have for predicting given x. However, from & generative perspective, we still want to learn x given and y, so we are interested in learning a personalized generative model that can model the difference in changing only the client i while keeping x and y fixed, and similarly changing only æ while keeping and y fixed. This type of intervention can help understand how to represent these variations better.
Suppose that do not know y. We want to learn how i influences the content in x. The variable i might have variations that correlate with the content, but the variations that persist across multiple clients should be the ones that can potentially identify the content of the data. From another point of view, data with similar content might still have a lot of variations within one client. Still, if these variations are uninformative according to the other clients, then we should not regard them as content-identifyy we propose maximize the correlation between the content representation of two generated images given the same content latent z². This is also applied analogously to style. To achieve that, we use the Barlow Twins objective as regularizer and the gemerator as model for simulating these interventions. Indeed, such interventional data are sometimes impossible to sample demand from nature, simulating an intervention on a good enough generative model can help as a surrogate to reality. For example, we can ask what the digit 5 would look like, what could be considered a chair, or what could be classified as disease or a tumor according to elient A vs. B.
variations. Therefore,
Let g be projector from the feature space to some metric embedding space. With some abuse of notation, let # s) denote the component we are interested in. Given a generative sample x ~ pxx|2", z", ,²) i) let denote the embedding of of (x) $ff z²) 9i
( (x)) where we compute x sg (G G ()) Note the usage of the stop gradient operator "sg" to emphasize that x is treated as a sample from P, so we do not pass gradients through it. Given this notation, the Barlow Twins regularizer we propose can be written
as follows
T (0;022zz")
[ET (2) (0l"=22))) p(2
E
22) (22,22)) LET (z) x²), (2 x²)]]
(14)
The intuition here is that the correlation between the content features of pair of samples generated by fixing the content latent and changing other latent should be close, and similarly for the style. Assuming optimal generator, we can simulate real-world interventions and self-supervise the model via this contrastive regularization. Note that the term "contrastive" is not concerned with positive and negative samples as is common in the self-supervised learning literature. Instead, we contrast a pair of samples that are the same except for some latent variable, it could be understood being "latent-contraastvve."
If we let. # (x,, x2) = LAT (Dr (of (of(x)) :9 (0. (xa))) then we can rewrite the objective in terms of (x1)
interventions on the generator's latent variables
T2 z
Cir((x):
+
E
(xI,
2"
X [x( (do(x)))
2 (x
X1 Pi (x] (x|x""ddoxx"" ddo(x"))
Pe(x")
8

---

Published in Transactions on Machine Learning Research (04/2024)
Figure 3: Our partially-federated linear generator can learn the right data-generating process for this dataset (top right). Federated averaging on linear model will yield the ordinary least squares solution at best (bottom left), whereas linear model with a private bias has better generalization per client (bottom right). The top left plot shows the full dataset. (color client)
Finally, for some hyperparameter X, we can rewrite the objective
with our regularizer
F((O;A) = E [f((0xx.))) (
(15)
x.y
See Fig. 2 for better understanding of the interventions.
Implementation details. The algorithm for training our model straightforward implementation of GAN training in a federated learning setting. We FedAvg (McMahan et al.) 2016) algorithm backbone. The main novelty stems from combining GAN architecture that can be decomposed into federated and private components, as depicted in Fig. ! with a latent contraicoeee regularizer (14 for the discriminator in the GAN objective. The training algorithm and othe procedures are described in more detail in the Appendix.
5 Experiments
In this section, we show some experiments to show the capabilities of the PaDPaF model. First, run a simplified version of the PaDPaF model on a simple linear regression problem with data generated following Simpson's Paradox as shown in Fig. Next, we run the main experiment on MNIST (LeCun et al. 1998) and CIFAR-10 Krizhevsky 2009). We compare our method with Ditto (Li et al. 2021a) as well as Ditto with FedProx (Li et al.) 2018). We further demonstrate that our method also works with variational auto-encoders (VAEs) (Kingma & Welling 2013). Finally, we show our model's performance on CelebA (Liu et al.) 2015). mainly to show its abilities in generating and varying locally available attributes (i.c. styles) on data sharing the same content. More experimental details can be found in the supplementary material. We generally use Adam (Kingma & Ba 2014) for both the client's and the server's optimizer.
In our experiments, we show how our generator disentangles content from style visually. We also show how the generator can generate locally unseen labels or attributes, i.e., generate samples from the client's distríbution given labels or attributes that the client has never seen. Next, we show how the features learned (without supervision) by the content discriminator contain sufficient information such that simple classifier the features can yield high accuracy on downstream tasks.
Linear Regression. We generate federated dataset with Simpson's Paradox by fixing a weight and changing the bias b across clients i, which would be y = wr + b, Further, for each client, we sample æ from intervals such that the opposite trend is observed when looking at data from all clients (see Fig. 3
Training a naive federated linear classifier leads to seemingly acceptable performance but learns the opposite true trend, as the negative trend is only due to sampling bias r/i. By federating the weights and keeping the biases private, we allow each client to learn shared trend and locally learn their own biases. Then, given that each client has discriminator that can tell with high accuracy whether some data is sampled from their distributions or not, we can achieve significantly smaller generalization error by routing new data to the most confident client and laking its prediction. Alternatively, we can weight the predictions of the clients based on their discriminators' probabilities (i.e. compute p(ý(z)) == Epxxxs))(((xx x, i) vs. p()(x)) == p(y|x = x. = f'), where i' == arg max, p(x[i)). Given good enough discriminators and expressive enough generators, we can generalize better while giving the clients the flexibility to handle the private parameters completely their to their benefit.
9

---

Published in Transactions on Machine Learning Research (04/2024)
1.50
3 04
2014
02
280 9.5
Figure 4: Left two plots: Accuracy and loss of linear classifier on the content and style features of unsupervised GAN training MNIST (i.c. labels y are not used in training). Right two plots: The progress of the accuracy and loss achieved by linear classifier in terms of communication rounds (better seen with colors).
Accuracy
Cross Entropy Loss
content
2.0
style
1.8
Acuracy
1.6
1.4
content style
1.2
10
30
20
Epochs
Epochs
Figure Accuracy and loss of linear classifier on the content and style features of unsupervised GAN training on CIFAR-10 (showing performance the style features of one client only)
The model we consider for this problem is a simple linear model for the generator with federated weight and private bias, and a small MLP for the discriminators (as it is impossible to linearly separate the clients in the middle from the rest). For all clients i, we generate a point x = wz" + b;, where 2 ~ (O,, 1) E IR and x, w. b, E IR², where we choose w to be federated and b, to be private. However, we can also consider an agnostic additive parameterization between federated and private parameters as x wzz + wiz" + b + bj, where x (0, 1), w and b0 are federated, and w and be are private. We show the performance of the latter model in Fig.
MNIST. In our MNIST experiments, we generate a federated dataset by partitioning MNIST equally on all clients and then adding different data augmentation for each client. This way, we can test whether our model can generate global instances unseen to some clients based on their data augmentation. The data augmentations in the figures are as follows (in order): 1) zoom in, 2) zoom out, 3) color inversion, 4) Gaussian blur, 5) horizontal flip, 6) vertical flip, 7) rotation of at most 40 degrees, and 8) brightness jitter.
See Fig. 6 for samples from conditional GAN (i.c. labels y are used in training). One interesting observation about the samples is that, since the number 7 was only seen in the horizontally flipped client, it retains this flipped direction across other clients as well. This is because learning flipping as a style is not incorporated in the design of our generator, and the same goes for rotations (see Fig. G lower left). However, the content was generally preserved throughout other clients. We argue that better architectures can mitigate this issue. Performance on other more difficult augmentations can be seen the Appendix.
From Fig. we see the content features contain sufficient information such that a linear classifier can achieve 98.1% accuracy MNIST without supervision. It is also clear that the style features do not contain much information about the content. This empirically confirms that our method can capture the most from 2 and disentangle it from zz.
In comparison with the baselines Ditto and FedPoxFe it is clear by visual inspection from Fig. that our method better and faster. We corroborate this by reporting the FID scores as well /Seizeer (2020) for each method in Tab. D Furthermore, the results from the VAE experiment in Fig. X show that our idea can extend to other generative models well, which is direction for future work.
10

---

Published in Transactions on Machine Learning Research (04/2024)
Alg. / id.
#1
#2
#F3
#4
#5
#6
#7
#8
Average
PaDPaF
11.47
4.10
11.12
7.77
8.38
6.84
7.94
8.62
8.28
Ditto
29.70
86.13
240.70
39.75
22.03
21.34
21.43
22.61
60.46
+
90.87
110.90
170.13
49.71
63.02
54.01
58.55
53.545
53.55
Table 1: FID for each client's GAN model and personalized training algorithm.
0
1
2.3
0 8 9
0
2
3
T
8 8
9
0
5
6
5 r
8
9 9 4 9 9 9 4
0
5
9 8 9 8 89 r 8 H M 8 Y
8
2
3
2
3
6
0
2
4
0 1
F
F
O
3
6
8
O
1
P
3
4
6
6
8 8 8
0
/
2
1 r
O
2
5
3
/
2
3
4
5
6
0
1
3
J
X
X
O
3
3
5
6 6 0
Y /
R / 8
9
0
1
2
3
5
6
0
3
R
B
r
O
2
4
6
0
1
3
6
8
0
2
O
5
t
H
O
9
3
O
2
3
5
6
8
0
3
T
O
2
5
|
3
6
9
Figure 6: Samples from conditional GAN on MNIST. Top to bottom, z is changed while z' and are
fixed. Left to right, z is fixed while z and y are changed. z and z² are the same across clients.
CIFAR-10. We prepare the clients' data in the same way as the MNIST experiments but with slightly different data augmentations and 10 clients instead. The data augmentations are as follows (in order): 1) zoom in, 2) color inversion, 3) Gaussian blur, 4) grayscale, 5) vertical flip, 6) brightness jitter, 7) random perspective (affine), and 8) solarize, 9) custom color transform 1, and 10) custom color transform The model demonstrates the same benefits from the MNIST experiments, as can be seen from Fig. The content features learned correspond to the content more than the style features. We show some samples from the model in Appendix CK well.
The federated dataset is created by a partition based on attributes. Given 40 attributes and 10 clients, each client is given 4 unique attributes and only has access to data having any of these 4 attributes. The content part should then capture the general facial structures, and the style should capture the other local variations within the altributes. Though not state-of-the-art terms of image fidelity, our results confirm the empirical feasibility of our model. The generated data for the client has variety that is specific to the attributes available to the clicnt.
CelebA.
6 Discussion
Our framework leaves some room for improvements and further work to be done in multiple directions. Here, discuss some of those directions and highlight interesting aspects we can exploit.
Client shift. Our architecture adapts to different clients under covariate shift (x(x) and prior shift (:y)). Can we remove assumption Assumption D and extend our framework to the case of conditional shift ()(y)))? This would help personalize the model's prediction based on the client.
Model improvement. How can we design generator that can effectively incorporate the clients' variations? Is the choice of private parameters mostly problem-specific? What about different modalitics of data, such as language and time series?
Generator on the server. Our model is trained with FedAvg and can benefit from the same privacy guarantecs. Privacy in our case could be better because the foderated part of the generator can stay completely in the server and the discriminator can share only the federaled part of its parameters. Can we train the federated discriminator completely on the server as well to ensure complete privacy? The server would then need data samples or least discriminative features of the data from the clients, which might reveal private information about the client. Still, it would be interesting to explore the benefits of having the generator be completely the server Augenstein et al. (2020)
11

---

Published in Transactions on Machine Learning Research (04/2024)
9
3
9
7
3
9
8
O
6
3
4
3
3
{
99
9
4
/
O
5
5
3
S
4
&
U
o
8
4
Q
8
7
7
9
/
2
1
4
?
//
9!
4
7
?
ah
7
6
1
6
4
3
8
5 3
7
J
6
9
&
0
9
7
9
A
2
2 J
J
3
2
8
1
8 ?
O à 8 3
D
2.
8
4
8
>
d
R
4
A
Q
8
X
2 S
J
2
\
T
6
S
8 J
R
3
R
2
P
P
Y
Q 7
8
9 3 2
7 6
0
O
1
3
4 8 6
4 4
S
4
7
T
&
1
6 3 5 2. 0 1
7
9
6
3
c 8 0
Ω 9 7 4 7
9
0
2
(a) PaDPaF
(b) Ditto
(c) Ditto + FedProx
Figure 7: Samples from clients trained for 120 rounds with different personalization methods.
f 4
5
2 4 5 4
5 7
3
J
3
9
6 C Σ
7 7 G
2
9
9 6 9 3
2
I S
7 9
2 0 8
7 4
1 5
2
a
C
X
P
C
J R
5 4
9
A
3
8
7
0 >
f F
t
&
6
2
2
0
7
8
5
9
7 3
0 9
0
$
3
8
p
S
9
3
7
9
2
J
4
8
9
Figure 8: VAF samples after 100 rounds of training with PaDPaF.
Conclusion. We introduced a framework combining federated learning, generative adversarial learning, and (causal) representation learning together. We do so by leveraging the federated setting and learning a client-inarriant representation based on a causal model of the data-generating process, which model GAN. We disentangle the content and style latents using contrastive regularizer on the discriminators. Experiments validate our framework's benefits and show that model is effective personalized generation and client-invariant classification, and benefits further from supervised data.
References
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers, 2019. URL httpss//arxii.... org/ hbgg//bs/// /abss//991..... 00818
Scan Augenstein, H. Brendan McMahan, Danicl Ramage, Swaroop Ramaswamy, Peter Kairouz, Mingqing Chen, Rajiv Mathews, and Blaise Agüera Arcas. Generative models for effective ML on private, decentralized datasets. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL httpp:://ppaarrriii...eefffoooot SJgaRA4FPH
Horace Barlow. Redundancy reduction revisited. Nelwork: compulation in neural systems, 12(3):241, 2001.
Duc Bui, Kshitiz Malik, Jack Goetz, Honglei Liu, Seungwhan Moon, Anuj Kumar, and Kang Shin. Federated user representation learning. arXie preprint orXiv:1909, 19w..19999.......1 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations, In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the TRFF/CVF Conference Computer Vision and Pattern Recogrition, pp. 13750-157555 2021.
12

---

Published in Transactions on Machine Learning Research (04/2024)
Harm De Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C Courville. Modulating early visual processing by language. Advonces in Neural Information Processing Systems, 30, 2017.
Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communcction efficient federated learning for heterogeneous clients. arXiv preprinl ar arv:20000. 01264, 2020.
Chenyou Fan and Ping Liu. Federated generative adversarial learning, 2020. URL https://arxii....t. orggabbboorg/b////////oo/ abs/
Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Newral Information Processing Systems, volume 33, pp. 19586-19597. Cur- ran Associates, Inc., 2020. URL 63223000771105556007777777771212
fttess/ppper/222//1112
pdf
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11): 139-144, 2020.
Ishaan Gulrajami, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pottern recognilion, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pp. 9729 9738, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium, 2018.
Samuel Horváth, Stefanos Laskaridis, Mario Almeida, Maas Lcontiadis, Stylianos Venieris, and Nicholas Lane. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vanghan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 12876 12889. Curran Associates, Inc., 2021. URL hhtps: : .dd
Jongheon Jeong and Jinwoo Shin. Training gans with stronger augmentations via contrastive discriminator. arXiv preprint XrXt::2103.0974 09742, 2021.
Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cumminns, Rafsel G. L. D'Oliveira, Hubert Eichner, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaid Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konečný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Ozgür, Rasmus Pagh, Mariana Raykova, Hang Qi, Daniel Ramage, Ramesh Raskar, Dawn Song, Weikang Song, Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramer, Praneeth Vepakomma, Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang, Felix X. Yu, Han Yu, and Sen Zhao. Advances and open problems in federated learning, 2019. URL https: www.ooog/////............o..aeae abs abs/aaas/1111 1997.04977
Minguk Kang and Jaesik Park. Contragan: Contrastive learning for conditional image generation. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (ods.), Advances in Neural Infor- Curran Associates, Inc., 2020. URL https;
mation Processing Systems, volume 33, pp.
13

---

Published in Transactions on Machine Learning Research (04/2024)
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CV /CVF conference on computer vision and pallern recognilion, pp- 4401 4410, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
1xX.v:1422.60 2..0,, 2014.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint drXiv:3112.6114 2013.
Jakub Konečný, H. Brendan McMahan, Daniel Ramage, and Peter Richtárik. Federated optimization: Distributed machine learning for on-device intelligence, 2016a. /tt/stps:://xxxiv.rrr wtt.:/rr/aabs/11...2222.22
Jakub Konečný, H. Brendan McMahan, Felix X. Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving commmication efficiency, 2016b. URL https :
//axxiv.org/ab///ee......o 05492.
Lingjing Kong, Shaoan Xie, Weiran Yao, Yujia Zheng, Guangyi Chen, Petar Stojanov, Victor Akinwande, and Kun Zhang, Partial disentanglement for domain adaptation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 11455-11472. PMLR, 17 23 Jul 2022. URL https://prroceedings mlr. httpe/diiinnnmm....ee/eeeeeeeeee ht/p/vv62////// 1622/kog2222 html
Alex Krizhevsky. Learning multiple layers features from tiny images. pp. 32-33, 2009. URL https //www. cs. toronto..o....ooo wttt:///trrtt/eeeernngeeeetrerre
Phue H. Le-Khac, Graham Healy, and Alan F. Smeaton. Contrastive representation learning: A framework and review. IEEE Access, 8:193907-193934, 2020. doi: /ttp:/AACCSSSS22003333333
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2224,, 1998.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks, 2018. URL https://arxxv wttp://oggg/bbba//.....m
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pp. 6357-6368. PMLR, 2021a.
Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbu: Federated learning on non-iid features via local batch normalization, 2021b.
Kuo-Yun Liang, Abhishek Srinivasan, and Juan Carlos Andresen. Modular federated learning. In 2022 International Joint Conference on Newral Networks (IJCNN). IEEE, jul 2022. doi: 10.1109/jicm550644 2022. 9892377
2022.9892377 URL https://doi.orr//1 1109/
cnn55064. 1
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self-supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data Engineering, 2021.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Othmane Marfoq, Giovanni Neglia, Aurélien Bellet, Lactitia Kameni, and Richard Vidal. Federated multi- task learning under a mixture of distributions. Advances in Neural Information Processing Systems, 34: 15434-15447, 2021.
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaisc Agüera y Arcas. Communication-fffieen learning of deep networks from decentralized data. 2016. doi: 10.48550/ARD ARXIV. 1602.05629. URL http:///aadi....ga/////1..00
14

---

Published in Transactions on Machine Learning Research (04/2024)
Takeru Miyato and Masanori Koyama. cgans with projection discriminator. In 6th International Conference on Learning Represenlations, ICLR 2018, Vancouser, BC, Camada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL http::://peeeeeiee..eeee. wtt.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.nel, 2018. URL https://openrevieer net mett//orrumidddiQqgzZ2
S Chandra Mouli and Bruno Ribeiro. Asymmetry learning for counterfactuall---nvriiil classification in ood tasks. In International Conference Learning Representations, 2022.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. Xrviv preprint or
2018.
Krishna Pillutla, Kshitiz Malik, Abdel-Rahman Mohamed, Mike Rabbat, Maziar Sanjabi, and Lin Xiao. Federated learning with partial model personalization. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 17716-17758. PMLR, html
17-23 Jul 2022. URL /attps://pree httts::/pproceedings..t mir. press/v1622
Mohammad Rasouli, Tao Sun, and Ram Rajagopal. Fedgan: Federated generative adversarial networks for distributed data, 2020. URL htttss///axiiii wwwp:/iii.rr///bbaaaa wttp:////a/....o../.../ 0722
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal represenlation learning. Proceedings of the IEEE, 109(5):612 634, 2021.
Maximilian Seitzer. pytorch-fid: FID Score for PyTorch. htttps///gtabbooc///setttrtrrrrr August 2020. Version 0.3.0.
Praneeth Vepakomma, Otkrist Gupla, Tristan Swedish, and Ramesh Raskar. Split. learning for health: Distributed deep learning without sharing raw patient data, arXiv preprint arXiv: 000664, 2018.
Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H. Brendan McMahan, Blaise Aguera y Arcas, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Decpesh Data, Suhas Diggavi, Hubert Eichner, Advait Gadhikar, Zachary Garrett, Antonious M. Girgis, Filip Hanzely, Andrew Hard, Chaoyang He, Samuel Horvath, Zhouyuan Huo, Alex Ingerman, Martin Jaggi, Tara Javidi, Peter Kairouz, Satyen Kale, Sai Pranceth Karimireddy, Jakub Koneeny, Sanmi Koyejo, Tian Li, Luyang Liu, Mchryar Mohri, Hang Qi, Sashank J. Reddi, Peter Richtarik, Karan Singhal, Virginia Smith, Mahdi Soltanolkotabi, Weikang Song, Ananda. Theertha Suresh, Sebastian U. Stich, Ameet Talwalkar, Hongyi Wang, Blake Woodworth, Shanshan Wu, Felix X. Yu, Honglin Yuan, Manzil Zaheer, Mi Zhang, Tong Zhang, Chunxiang Zheng, Chen Zhu, and Wennan Zhu. A field guide to federated optimization, 2021. URL https://arxiv.orrr wttp:///b//....00011111111
Yixin Wang and Michael I Jordan. Desiderata for representation learning: causal perspoctive. arXiv preprinl Xrx:::2109. 03795, 2021.
Mariel Werner, Lie Hc, Michael Jordan, Martin Jaggi, and Sai Pranecth Karimireddy. Provably personalizod and robust federated learning, 2023.
Yue Wu, Shuaicheng Zhang, Wenchao Yu, Yanchi Liu, Quanquan Gu, Dawei Zhou, Haileng Chen, and Wei Cheng. Personalized federated learning under mixture of distributions. arXiv preprint arXiv:2305.0100 2023.
Ryo Yonetani, Tomohiro Takahashi, Atsushi Hashimoto, and Yoshitaka Ushiku. Decentralized learning of generative adversarial networks from non-iid data, 2019. URL
Ning Yu, Guilin Liu, Aysogul Dundar, Andrew Tao, Bryan Catanzaro, Larry Davis, and Mario Fritz. Dual contrastive loss and attention for gans. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6731-6742, 2021.
15

---

Published in Transactions on Machine Learning Research (04/2024)
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Leurning, pp. 12310- PML.R, 2021.
Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. In International conference machine learning, pp. $19-827. PMLR, 2013.
Roland S. Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. 139:12979 12990, 2021. URL http://pproceedingg..t prees/vv39//immermaaa html
A
Algorithm
In this section, we write the training algorithm in detail for completeness. The training is straightforward application of FedAvg on GANs without its private parameters.
Here, we let OpT(O, f())) be a first-order optimizer, such as SGD or Adam, that takes the current parameters 0, and the gradient (0)) and returns the updated parameters given step size n). We also define subroutine GANOPT that runs single descent or ascent step on the GAN's objective. The term traing is a boolean variable that specifies whether the generator or the discriminator should be trained. In the GAN literature, the discriminator is often trained 3 to 5 times much as the generator. In our experiments, we let TD = 3, so D is trained 3 times before G is trained, but we also update 8 with double the learning rate, following the Two Time-scale Update Rule (Heusel et al. 2018). As for the client's 's loop, we can simply train it on the dataset for one epoch before breaking the loop so that fmax = [X,] and n. = 0, but we add a loopless option as well by lctting max = oo and π, x Xx]] for the same training behavior on average.
Note that could have removed line 12 of the algorithm and let line 11 be Afed (fed 0fed (1), but chose the current procedure to emphasize that the server's update the private parameters is always set to 0.
B Model Architecture
Our model's architecture is composed of regular modules that are widely used in the GAN literature. The generator is a style-based generator with ResNet- archilecture (Gulrajani et al. 2017). where we use a conditional batch normalization layer (De Vries et al., 2017) to add the style, which is generated via a style vectorizer (Karras et al. 2019). The discriminator's architecture is ResNet-basee as well (Hle et al. 2016) with the addition of spectral normalization (Miyato et al. 2018). In the case of conditioning the construction of the projection discriminator as in (Miyato & Koyama 2018 so we add an embedding layer to the discriminator, and we adjust the conditional batch normalization layers in the generator by separating the batch into two groups channel-wise, and condition each group separately. Finally, we add a projector to the discriminator for contrastive regularization, which is a single hidden layer ReLU network. Recall that we use two discriminators, one is private, and the other is federated.
labels, we follow
As for the model architecture for the linear regression dataset, we simplify the discriminator to 2-layer neural net with 8 hidden dimensions. The generator is linear layer and the style vectorizer is also linear layer with latent vector of dimension 1. The projectors are lincar layers as well.
The variables that control the size of our networks are the image size, the discriminator's feature dimension, and the latent variables' dimension. We make the dimension of the content and style latent variables equal for simplicity. For MNIST, we choose the feature dimension to be 64 and the latent dimension to be 128. For CelebA, we choose the feature dimension to be 128 and the latent dimension be 256. Kindly refer to the code for more specific details.
16

---

Published in Transactions on Machine Learning Research (04/2024)
Algorithm Contrastive GANs with Partial FedAvg
1: Input:
Clients i E [M], datasets X; x; {(x),,,))) (5)) [x4] target distribution g((10)(() over X,, GAN models pi(;;s), 0s), GAN objectives fi, server and clients first-order optimizers OPTo and OPT, with step sizes 7o and TJ, client's loop stopping probability my x [X, and Tp iterations for training D vs. G. Output: g/ad and 6pvt, Vi.
Set 0, == On Vi [M]
for 1, Tmax do
Sample subset of clients I ~ ((i)
6:
for E in parallel do
7:
for t= 1, ... maxx do
traing < 1l mod TD + 11 (1 < GANOPT (i, 0.(4),traing traing; (6)
10:
if (ernoullimm)) then
11:
A; 5- (1) (t)
12:
A.M + 0
13:
break loop end if end for end for
14:
15:
16:
17: 0o(T + +1) +) OPTo ( EiEr As; (o(T). m))
gfod - ofod for all clients E [M]
18:
19: end for
20: procedure GANOPT(i, 0,, traing; fi, q:)
21: if traing then
22: return OPT; fi 0, m3)
23: else
24: Sample & ~~ a(-li)
25: return OPT: - Vop (1, {); 0, 2m:)
26: end if
27: end procedure
17

---

Published in Transactions on Machine Learning Research (04/2024)
Federated weights and Blases
Federated weights and Private Biases
-2.0
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
-2.0
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
Figure 9: A single linear regression model on Simpson's Paradox dataset federated linear regression models with private biases.
C Experiments
In this section, we describe the experiments we rum in more detail and show results extra results supporting
our framework.
For all our experiments, the client models are handled by "workers". The workers only train their models on specific subset of clients that does not change. We often assign worker for each client, but due to constraints on computational resources, we might create a smaller number of workers and assign a specific subset of clients for each worker. We run all experiments on a single NVIDIA A100 SXM GPU 40GB.
Our choice for all the optimizers, including the server's optimizer, is Adam (Kingma Ba 2014) with 81 = 0.5 and B2 0.9. We choose Adam due its robustness and speed for training GANs. We found that choosing learning rates 0.01 and 0.001 for the server optimizer and the client optimizer, respectively, is good starling point. We also use clients, with decay rate of 0.99 for MNIST (0.98 for CelebA) per communication round. This can help stabilize the GAN's output. For Ditto and FedProx, we choose the prox parameter to be equal to 1.0.
aoonential-decaa learning rate schedule for both the server and the
C.1 Linear Regression on Simpson's ss Paradox
We generate a datasel that remonst.rtes Simpson's Paradox for M clients as follows. We fix a weight, say, w. := 1. Then for each client i {1,... .M), we sample x uniformly from [M i, i 1] and a bias b from [L(i- 1), Li] for some constant L. Finally, for each client i, we generate n, points y ~ wx + and then normalize and y by subtracting the mean and dividing by the standard deviation. In experiments, we choose M : 8, T. == 4, and == 50 for all i. For training PaDPaF model on this dataset, we noticed that D should have slightly larger feature dimension, like 16, whereas D can be smaller, like 2.
If we train a naive linear regression model on this dataset for 50 epochs with a batch size of 10, we would get mean-squared error of about 0.182, whereas similarly training a linear regression model for each client, and then freezing the models and training again a linear routing model would yield an error about 0.004, which is two orders of magnitude smaller (see Fig. In Fig. we see that can still find the right trend for each client with more cluttered data.
Code for reproducing plots and similar errors for this experiment are provided in the linked repository in the main text.
C.2 MNIST
The MNIST train datasct partitioned into subsets assigned clients, and cach client handled by a unique worker. The data augmentations used for each partition are clear from the images. We write the Torch code for each data augmentation:
18

---

Published in Transactions on Machine Learning Research (04/2024)
8
2
7
5
5
6
0
3
4 A
L
6 8 2 8 9 & 9 L R
9 9 9
7
9
7 7 7 7 9 7 7
2
4
2
9
8
7 7.
9
1 I
5
7
3
6
8
O
4
4
3
1
9
9
4
3
O
4 3 R
7
5
5
7
5
5
6
&
5
7
5
6
&
C
7 7
O L O L C L L
4
4
7
2
9
2
2 8
2
1
9
7
7
9
8
C
8 9
7
9 9
y
8
9
7
9
A
2
/
8
7
2
/
3
3
6 4
8
C
7
4
9
3
8
/
9
4
8
3
8
I
9
b
6
1
3
C
8
C
4
8
9
6
5 3
8
R
d
P
5
J
2
O A
P
R
P
P P
P P P P
I P P I P P 7 7
O O A
A
R
L
P
9
7
2
4
R
A
2
8 3 Q
O
p
R < R 8 4 8 9
P
L
9
R
A
8
1
2
6
+
P
$
2
2
O
P
P
O
L
P
9
R
2
3
P
P 8
2
I 2
Q
6
3
P 4 O L
O
P 9 9 9 9
P 9 9
A
O
A
R
L
9
R
6
0
R
9
5 9
/
b
o & 8 8
3
R
7
5
6
0 9 6
2
O 1
4 4 9
I I
O
4
8
9
9
9 A
7
1
4
A
1
7
8
O
4
9
7
4
7
/ A
2
J &
3
6
8
/
2
7
3
O L
8
O
Z
4
9
4
3
8
3
L
4
8
8
4
9
(a) x and z' are the same across clients, but different
(b) Top to bottom, z' is changed while z and y are fixed. Left to right, x" is fixed while z and changed.
within clients.
Figure 10: Samples from GAN without conditioning. 2 and 2 are consistent across clients.
0
1
5
4
7
0
23
4
5
6
8
0
23
5
6
O
2
6
8 8 6
0
7 8 9
6
S
0
2
1
89
O
2
3
4
0
0
23
4
156
8
4
2
3
4
5
8 8
O
2
3
+
6
8 9
1
9
5
6
7
8
9
0
3
5
9
J
4
0
2
3
4
6
O
2
3
4
5
6
8
9
1 O
0
2 2 2 1 1
3
5
7
8
9
0
1
2
3
4
5
9
5
6
O
2
3
4
6
8
O
2
S
9
8
3
W
6
9
0
2
3
5
9
)
2
4
6
P
9
O
2
3
7
6
/ R
9
0
3
5
9
D
/
2
3
5
7
O
S
8 1 8.9 7
C
Y
R
O
3
5
3 9
0
1
5 J
3
&
7
R
0
3
5 4 3 4
R
S 5
3 +
-
8 3 8
9
O
S
3
9
0
3
r
7
0
R
7 8 R P 8 J
O
R
T
6
3
5
3
6
0
J
3
K
T
R
O
8
à
9
O
5
a 3
\
V
t
R
O
S
3
S
R
O
1
3
R
0
3
3
6
9
O
2
5
6
8
9
0
/
7
3
4
5
6
9
0
/
2
3
4
5
[
0
2
4
O
2
6
8
9
0
3
4
5
6
9
3
5
6
(
9
0
9
3
4
6
0
9
O
2
3
5 6
8
4
0
7
3
4
5
6
9
0
/ 2
3
4
3
6
)
9
3
4
0
2
6
8
9
0
3
5
9
0
3
6
(a) Top to bottom, x" changed while x" and y are fixed.
(b) Top to bottom, x" is changed while z" and y are fixed. Left to right, zf is fixed while x" and are changed. Left to right, x" is fixed while xf and y are changed.
Figure 11: Samples from conditional GAN, 2 and z are consistent across clients.
19

---

Published in Transactions on Machine Learning Research (04/2024)
1. Zoom in: CenterCrop (22).
Zoom out: Pad(14).
3. Color inversion: RandomInvert (p=1. .0).
Blur: GaussianBlur (5, sigma=(0.1, 2.0)).
Horizontal flip: Randomlortzzonntalllppppfp....
6. Vertical flip: Randomm/errtical/111pps1....
Rotation:
RandomBorationn44)..
8. Brightness:
ColorJitterr
In the case of conditioning on y, we further restrict the datasets and drop 50% of the labels from each partitioned dataset. This implies that data will be lost, and each client will see only 50% of the labels. This is intentional we want to restrict the dataset's size further and test our model's robustness unseen labels.
The results for training our model on this federated dataset are shown in Fig. 10 for the unconditional case (i.e. y is unavailable), and Fig. [ for the conditional case with 50% labels seen per client. The results were generated after training the model for 300 communication rounds. For MNIST, we train the models for a half epoch in each round further restrict the local convergence for each client.
C.2.1 Adaptation to New Clients (i.e. Data Augmentation)
After training our conditional model on the previous data augmentations, we re-train it for 25 communication rounds on the following data augmentations and show the results:
1. Affine:
RandomAffine (dogrees (30, 70), (0.1,, 0.3), scale= (0.5, 0.75)).
Solarize: Bandomm@olarizeehhheeetooo...
Erase:
transforms. Ran.eeaeedddddnnnnnP111... scale= 02.02, 0.1)) (after ToTensor ()).
This is to show that conditional model can quickly adapt to new styles without severely affecting its content generation capabilities and its generalization to unseen labels. We can see in Fig, [3] that solarization is learned as a style, while erosion is learned as content. The generalization to unseen digits is maintained but less so in the challenging affine augmentation.
C.2.2 Can We Predict the Client from the Image?
We ask a question analogous to the one we are concerned with in representation learning. Instead of predicting the label from the image, we want to predict the augmentation, or the client, that generated the image. We show that we can predict the client with good accuracy (93.3%). See Fig.
The style prediction done by linear transformations on the content representation and the style representa- tions. For the style representations, we lincarly map each representation from each client to a scalar u, so that is vector of all the scalars. We then pass through an extra linear transformation to get vector of logits for p(ilx). Note that this is still a linear transformation of the style representations, but we do it this way to reduce dimensionality.
Note that the content representation can still predict the style with good accuracy. Indeed, we have seen that our generator architecture finds difficulties in assigning rotations to style variations, which due to the style generator construction as a conditional batch normalization layer, which shows limited capabilities in capturing rotation-like variations. Still, the prediction accuracy from the style representations is better overall, which shows that our model does disentangle, to some extent, the style from the content.
20

---

Published in Transactions on Machine Learning Research (04/2024)
3
6
5
8
5 5
3
8
3
3
3
5
8
3 2 B 2
3
3
5
4
6
10 3
3
3
6 8 6 8 B 6 2 8 a ,
2
4
2
3
R 6
5
6
5
3 6 8 6
5 6 5 6
5
8
8
3 2
8
5 2
9
3 6
5
8
4
8
8
3
2
5
8
8
9
8 8/ 2
6 3
5
6
4
8
8 3
8 5 3
w
5
5
8
8 3 8
B
5
4
6 0
6
8
3
3 8 8 5 8 3
s
6
9
6
5 5
8
8
D
8
8
5
3
6
6
8
3
8 2
Figure 12: Results after re-training for communication round on the three new data augmentations. Rows 1-2 are real samples and rows 3-8 are generated. Rows 3-6 share the same 2", and rows 7-10 as well. Rows 3-4 and 7-8 share the same z", similarly for rows 5-6 and 9-10.
9
6
q
V
9
0
9
D
O
2
3
6
6
9
2
3
4
5
6
8
9
O
2
4
O
/
3
3
9
3
1
5
6
F
8
9
4
3
4
5
6
B
/
2
3
5
8
2
3
4
6
6
8
3
3
H
5
B
2
3
6
8
9
3
#
5
C
5
8
9
/
2
5
4
D
/
2
3
(
6
8
9
0
2
3
4
5
6
5 f
0 8
7
2
3
y
-
6
8
9
0
/
2
3
4
6
8
9
3
2
R
3
V
5
6
8
8
2
3
4
3
P
0
/
2
3
8
9
8
4
2
3
6
9
3
/
3
5
6
0
/
2
3
8
9
0
)
2
3
1/
5
G
8
8
O
/
e
S
5
8
3
(a) Changing z per row, fixing n.. (b) Changing 2 per row, fixing z",
(c) Changing both z and z
Figure 13: Results on unseen clients after 25 communication rounds.
Accuracy
Cross Entropy Loss
0.9
1.2
content
Accuracy
0.8
1.0-
0.8
0.6
0.6
content style
0.4
0.2
10
20
30
40
50
10
20
40
50
Coochs
Epochs
Figure 14: Prediction of style. We see that the content features have good accuracy (82.6%), but the style features can classify the source client with much better accuracy (93.3%).
21

---

Published in Transactions on Machine Learning Research (04/2024)
0
4
7
5
9
0
5
0
4
7
5
9
0
5
0
4
7
5
9
1
0
5
7
+
7
9
6
5
8
4
7
4
9
6
3
8
/
4
7
9
9
6
5
8
7
7
1
7
2
6
9
8
2
7
1
7
2
9 9 9
6
6
2
7
7
7
2
6
9
3
6
5
2
$
3
9
0
6
3
2
8
9
3
0 5 9 9
6
5
2
8
9
3
O 5 8
9
5
6
<
4
4
9
5 8 8 9
5
6
(
4
5
6
6
4
4
8
4
3
4
(
/
r 6 8
4
3
r
(
/
8
3
4
6
E
8
5 6
3
5
9
0
6
3
3
5
9
0
6
3
6
3
5
8
0
6
3
8
4
3
9
2
>
/
5 3
9 2
}
3
1
5
3 8
2
8
>
3
(a) Source (generated) images.
(b) Reconstructed images.
(c) Re-styled, transferred content.
Figure 15: The images show results from the regular, unconditional GANs, whereas the bottom shows results for conditional GANs with 50% labels scen per client. Observe how the content stays preserved after transfer. Here, we show example of transferring the content from client 8 (brightness) to client 3 (color inversion).
C.2.3 Re-Styling via Content Transfer
We describe simple method for transferring the content of an image to another style, given the style vectorizers G.. In other words, can transfer the content of an image taken from one client to another without sharing the image but by finding the approximate latent variables that generate the image and transferring the content latent variable. We shall see that this approach can indeed transfer the content well enough, where the content is up to the generator's expressiveness and its separation of style. For example, we show that our generator finds difficulties in distinguishing some rotations as non-content variations. This is because all other clients have slightly rotated digits as well.
To find the approximate latent variables from the image without any significant. overhead, we use the discriminator's representations prediet the latent variables. We show that this approach can predict latents that reconstruct the image very well, and thus we use the same content latent on the new client.
Algorithm Re-style via content transfer
Input: Trained models, fmax number of epochs, x. (and ya if any) sample from source client a, and a target client b.
Output: Representatiion-tollate transforms 3 and 3 for each Dg and Dj, resp., Vi [M], and re-styled
3: for t= 1. fmax do
4:
Sample client i ~ (i), and sample label y ~ (y(), if any.
5: Sample latents z ~ (z9) and zz ~ qz(z)).
6: Sample image x ~~ pr (x (2 z", (i.e. set x Gz "; Transform latents म (of(()) and 2
(x)(). Sample reconstructed image & (x(x|2 2" y).
9: Minimize (Er()(()), (x) (8) w.r.l. 3 and 3 (see
8
10: end for
11: Transfer a - b: Sample xs, ~ (x zn, Ya), where zi ~ pp((z).
First, we show the algorithm for reconstructing and re-styling an image from some client. The main idea is to use LeT from g to maintain an identity correlation between the source content representation and the target (i.e. reconstructed) content representation. We noticed that we do not need to do the same for the style representation. Not enforcing an identity correlation between styles can even improve the results terms of style consistency. We do not find a good explanation for this. We believe that applying Barlow Twins loss in our setting warrants further empirical and theoretical investigation.
22

---

Published in Transactions on Machine Learning Research (04/2024)
8
5
6
9
5
4
6
2
0
8
3
4
9
0
7
8
/
6
4
0
8
0
5
4
/
6
4
3
0
3
8
8
5 6
2
0
5
5
7 8
5
9
4
8
3
7
5
9 9
3
3
1
6
5
8
4
2
3
4 0 3 9 9
9
9
9
8
8
7 /
5 4
9
4
2
0 4
8
3 4
4
8 3
:
e
8
6 a
5
6
8
5
0
6
5 5
9
4
8
3
3
6
6
58
4
23
4
6
1 8
4
q
q
5
9
5
4
0
8
7 5
9
5
4 0
2
8
3
/
8 0 5 R O 1 R
0
2
C
3
4
0
8
6
0
8
0
5
8
/
6
4
0
8
0
5
8 /
4
3 &
4
6
4
0
9 6 6
9
5
5
5 3
5
8
5
5
5
7 3
5
4
3
3
9
3
D
5
3
6
5
3
9
3
6
6
5
3 3 8 7 3
3
8 V X
2
4
6
3 V R
8
P
8 8 7 3
O 5
R
P / S
0 3 2 8
O A
1 R
S
R
3
r
0 3 1
1
3
0 &
0 7
2
R
0
0 7 3 0
H F 3
O $
X
C
U 5
R
R
3
P
2
Q
à
A
3
7
E
d
5
4
3
X
d
5
4 A
A 0
S
R
9
5
9
5
0
8
7
5
5
J
O
3
4
9
T 8
4
9
0
r
/
6
0
8
0
8
/
b
4
0
0
4 / 6
4
3
0
3
4
6
4
0
3
5
0
<
5
5
8
3
2
0
5
5
5
9 4 /
3
[
9
3
l
3
J
9 9
5
3
4
6
5
3
3
4
2
4
7
9
9
8
9
4
9
8
(a) Unconditional.
(b) Conditional, 50% labels per client.
Figure 16: Transferring an image from MNIST's test set to all clients. Test image on top transferred image to each client at the bottom.
The effectiveness of this procedure can be seen in Figs. 15 and 16 where we transfer the content of some image to a specific client. In Fig. 15 we choose client as the source and client 3 as the target to demonstrate that the roconstruction preserves both the content and style and that the transfer preserves the content. In Fig. 16 notice that, since the MNIST's test set is unscen during training, the unconditional GAN might map scemingly easy digits to other similar-looking ones. Simply training unconditional GAN to mitigate this effect. Note that this effect is not seen in the conditional GAN, given that the clients see only 50% of the labels during training. It is worth mentioning that, as long as the content representation encoder σ6 is good enough for images from some unseen domains, like the Lest set of MNIST, then we can still transfer its content to a known client i with a trained style vectorizer G
the test set should allow the
C.2.4 Experiments on VAEs
We consider applying our method on VAE (B-VAE to be specific), which is a well-known generative model that optimizes the evidence lower bound (ELBO), which general objective that many algorithms can traced back to. Let z = (z²,z"). and with some abuse of notation, let 0D, g be the parameters of the encoder and decoder, respectively. Then, the V-E objective can be written as:
1(0". -Eapp (z/x) log Pae (xlz) + 3.r, (2)x)])ppo(z))
(16)
Perhaps the main difference here is that VAE's encoder outputs the latent variable's statistics, from which the latent can be sampled with the reparameterization trick.
The results can be seen in Fig, S One should be careful with applying the SSL regularizer with VAEs so that it does not overcome the training of the divergence term. Tuning VAEs is thus not very straightforward. Using lower values of 8 and X < B seems be desirable. We have not run extensive experiments on VAEs obtain performance par with GAN experiments, but this setup should be a proof of concept that
23

---

Published in Transactions on Machine Learning Research (04/2024)
framework is possible to extend to other generative models and that its benefits demonstrated, particularly for client 3 (color inversion). This is emphasized when taking into consideration the comparisons with Ditto in Fig. 7
C.3 CIFAR-10
Training personalized GAN model MNIST in the federated learning setting is in itself tricky, but we show that it is also possible on more realistic data, such on the CIPAR-10 dalaset. Similar results can be obtained on CIFAR-100, but we chose to show results only on CIFAR-10 for simplicity. The results for this experiment can be seen in Fig. 5 and Fig.
For this experiment, we introduce 10 clients with different data augmentations and we train them using Adam with local learning rate of 0.0003 and a global learning rate of 0.003. Preciscly speaking, the data augmentations
1. Zoom in: CenterCrop(22) (22).
Color inversion: LandomInvert(p=1....
3. Blur: GaussianBlur(C (5, sigma- (0. (0.1, 2.0)).
4. Grayscale: Grayscale
Vertical flip: RandmwVarttical1111((11111)
6. Brightness: colortiitter((riighhtuusss:...
7. Perspective: R Bandoopperpeeciiicccisstttooonn 0ftp://isstttttttoocalaaa0.... p=1.0).
Solarize: (antam3blarriaahhhhhehhoo..1....
Color Transform 1:
(a) Normalize (mean= [0.5], std=[0.5]]
(b) transforms. Lambda (lambda (x + 0.5 * torch.rand ra,ch,,, (3,1,1))
torch. (3, 11lpp.,111..
10. Color Transform 2:
(a) Normalize (mean= (0.ean(00)) std= [o.=)..5)..
(b) transforms Lambda (lambda (x 0.5 * forch.rand (3, (3,1,1))
torch. rand 1),1
(c) termalizemmaarr [], std=(o.5)).
(d) transforms.c Lambda (lambda x. c.clip(-1,))).
C.4 CelebA
We our model on ColobA, which more challenging datasct. The datasct contains pictures of oelebrities, where each image assigned a subset of attributes among 40 possible attributes. Thus, we create 40 clients, give unique attribute to each, and then show them the subset with this attribute. Note that this is not partition, as some data will be repeated many times, but we want to see if the client will learn a biased style towards the assigned attributes.
To make the simulation feasible, we create 10 workers with their models, each handling a unique subset of clients, and let cach worker sample their clients in a round-robin fashion. Thus, each worker will always see least one of attributes. We train our model for approximately 200 communication rounds, with 2 epochs per round, and train the discriminators 5 times as frequently the generator (i.e. Tp 5).
24

---

Published in Transactions on Machine Learning Research (04/2024)
Figure 17: Samples from a condooniiiioall GAN on CIFAR-10 10 (skipping the first client corresponding zoom-in augmentation). For each client, the first rows share the same z", while the last 4 rows share a different z.. Rows 1, 2, 5, and 6 share the same z", while rows 3, 4, 7, and 8 share different z".
25

---

Published in Transactions on Machine Learning Research (04/2024)
Accuracy
Cross Entropy Loss
1.0
12
content style
0.8
10
Accuroty
0.6
0.4
4-
0.2
solc 10
0.0
s0lc 50
10
20
30
40
50
60
70
10 20
30
40
50
60
70
Epochs
Epochs
Figure 18: Accuracy and loss after training linear classifier to predict the celebrity's identity. Note that there are 10,177 identities in total, which makes this task hard, so we also show top-10 and top-50 accuracies.
Given some content. latent, we can see that the client-specific attributes become more obvious in some clients than others. For example, lipsticks, mustaches, blonde hair, and other attributes are clearer in some clients than in others. Soe Figs.
for samples from cach worker.
For completeness and better understanding of the generation quality, list the altributes assigned to each worker:
Worker 1: ['Mustache', 'Mouth_ 'Mouth_Slightly Slightly_Opentt 'Sideburns', 'Big_Lips']
2. Worker 2: ('Attractive', "harrow_Eye'' Hair', 'Bald'].
3. Worker 3: ['Arched_Es] Eyttboww_yyrrows', 'Bangs', 'Chubby', 'Eyeglasses'.
4. Worker 4: ['Male', Mlosy_Cheeke "learing_Necktie" 'High_ (heekhoheebb
Worker 5: ('Straight_Haie Hair', "Wearing Earrings', 'Black 'Black_Hair', 'No [N__eerrd'] '].
Shadow', 6. Worker 6: ['5_o_Clock____ _e0__Cloc_Shhaorr, 'Young', NearinngNecklace,, Maav__Hair']
7. Worker 7: "RecedinggHaairlinee (Bushy_Eyebrove 'Goatee', 'Hevv "eaavy_Makeup"))
Worker 8: ['Pointy_Nose', "Blond_Hair' Double_Chin'' "Dval_Face")
9. Worker 9: ['Big_Noee Nose', 'Smiling', 'Blurry', Brrow_HHir']]
10. Worker 10:
'Pale_Sking (Bags_Under_
"Wearing_Har Hat '.
Celebrity identification from representation. We also test the content and style features on a celebrity- identification task. The construction of the dataset is done so that we only test whether the style introduces a bias towards some attributes given some content, so we do not expect to see disentanglement between content and style. If we see one and only one attribute per client, then we might expect an attribute be a style in the sense that it constitutes a domain, in our MNIST example. Tn Fig. [8 show the accuracy of lincar classifier and 79%, resp.), good 90% in 10 accuracies (95% and 93%, resp.). Note that using top-10 and top-50 is approximately within the same proportion as using top-1 1 and top-5 for ImageNet, which has 1000 classes.
the content and style reprosentations. Both achieve docent top-1 accuracies (84%
26

---

Published in Transactions on Machine Learning Research (04/2024)
(a) Changing z per row, fixing z", (b) Changing z per row, fixing z" (c) Changing both of and z
Figure 19: Workers at the top, 2 in the middle, and 3 at the bottom.
27

---

Published in Transactions on Machine Learning Research (04/2024)
(a) Changing z per row, fixing z², (b) Changing zf per row, fixing z". (c) Changing both zf and z
Figure 20: Workers 4 at the top, 5 in the middle, and 6 at the bottom.
28

---

Published in Transactions on Machine Learning Research (04/2024)
(a) Changing x² per row, fixing x.. (b) Changing x fixing x (c) Changing both xf and x².
Figure 21: Workers 7 at the top, 8 in the apper-middle, 9 in the lower middle, and 10 at the bottom.
29


Reviews:
[{"review_id": "CF6YbTe0Aq", "reviewer": "Reviewer_kUTC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n- The proposed architecture combines global client-agnostic and local client-specific generative models, which achieves privacy and personalization in federated learning.\n- The use of disentanglement in personalized generative models provides benefits such as generating locally-unseen labels or attributes and removing private information.\n- The experimental evaluation on various datasets shows the effectiveness of the proposed approach in disentangling content from style and achieving high accuracy on downstream tasks.\n- The paper provides theoretical justifications for the proposed approach and suggests future research directions.\n\nCons:\n- The experimental evaluation is limited to a few datasets (only two), and the generalizability of the proposed approach to other datasets or real-life applications is not fully explored. \n- The paper does not provide a detailed comparison with other existing approaches in federated learning or personalized generative models. Basically, only GAN based works are discussed. Other possible generative models such as normalizing flow, VAE, Autoregressive, diffusion model works on Federated learning are not discussed and compared.\n- Meanwhile, more thorough literature review is also suggested. Some SOTA federated learning papers are suggested to be included in the related work such as the following:\nPersonalized Federated Learning under Mixture of Distributions. ICML'23.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper explores the potential of federated learning in personalized generative models. The authors propose a novel architecture that combines global client-agnostic and local client-specific generative models to achieve privacy and personalization in federated learning. The paper also highlights the benefits of using disentanglement in personalized generative models and provides experimental evaluation and theoretical justifications for the proposed approach.\nOverall, the paper provides a comprehensive exploration of the potential of federated learning in personalized generative models. The proposed architecture is innovative and provides a promising approach to achieving privacy and personalization in federated learning. The experimental evaluation and theoretical justifications provided in the paper further strengthen the proposed approach's validity and potential for future research. However, the experiments are not sufficient. Basically, only two datasets are used. Meanwhile, more thorough literature review is also suggested.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "However, the experiments are not sufficient. Basically, only two datasets are used. Meanwhile, more thorough literature review is also suggested.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZvJjZjK0P0", "reviewer": "Reviewer_eMad", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths：\n+ It addresses the challenge of learning personalized generative models in a federated learning setting, which has been unexplored.\n+ The paper achieves partial decoupling of content and style, allowing the model to distinguish between globally consistent features (content) and client-specific variations (style). This decoupling allows the model to share general knowledge across different clients while maintaining personalisation.\n+ Experimental results show that the content features learnt through PaDPaF are able to support downstream tasks even in an unsupervised setting.\n\n\n\nWeaknesses：\nThis paper could further be enhanced from the following perspectives:\n- The experimental component lacks comparisons with other related work and there are no indicators to quantify the effect of the generating experiments\n- The abstract section references the validation of the theory, yet this aspect is not adequately addressed or demonstrated within the main content of the text\n- The paper mentions being able to address domain adaptation, but the treatment of data heterogeneity is not captured\n- The paper discusses transformers in the introduction and also touches upon diffusion, which is deemed unnecessary for the context of this study.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a novel framework that aims to enhance federated learning with generative adversarial networks for personalized model generation. The framework combines global, client-agnostic models with local, client-specific models to achieve privacy and personalization.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As in Weaknesses.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rWKWZwpHIQ", "reviewer": "Reviewer_gPAJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The idea makes sense and works well\n- The theoretical understanding is not that sufficient", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a novel framework for Partial Disentanglement with Partially-Federated Learning using Generative Adversarial Networks (GANs). In this approach, specific components of the model are federated, allowing the GAN's discriminator to partially disentangle representations into global and local factors. The primary focus is on global factors, crucial for inference. The disentanglement is reinforced by a \"contrastive\" regularization term inspired by the Barlow Twins methodology.\n\nThe proposed model achieves the ability to learn globally-consistent representations of data content, such as labels, leading to high-accuracy label classification. Notably, in supervised scenarios, the model demonstrates the capability to generalize to locally-unseen labels within individual clients concerning both classification and data generation.\n\nExtensive experimentation supports the paper's claims, illustrating the seamless integration of techniques from federated learning, GANs, and representation learning. The results highlight the robustness of the model in real-world scenarios, showcasing its effectiveness in both classification tasks and data generation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- It would be great if the authors can explain whether the suggested method can be applied to generative models other than GANs. Some readers may want to apply this idea to more stable generative models, so it would be great to give some examples. \n- The authors claim that \"partial theoretical justifications for the proposed approach\", but lack of some formal mathematical statement, i.e., Section 4 has some \"discussions\", but not proofs. I think it is better not to over-claim and tone down a bit regarding the theoretical contribution.\n- Any baselines to compare empirically? Current experimental results are mostly showing the performance of the proposed method, but I want to know whether there are possibilities of comparing with existing methods that can be used for personalized generative models.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers raised the following concerns about the original submission that:
1) it was not comparing sufficiently with the related work
2) the experimental evaluation was limited to only two datasets, and did not consider whether the approach could generalize to other generative models (e.g. VAEs)
3) some claims were not well supported (e.g. "theoretical justifications" in the abstract or the mention of transformers in the introduction as potential direction of future work)

All these concerns were addressed appropriately in the revision (see author's response): 
1) they added comparison to related work (Ditto and FredProx)
2) they added one dataset (CIFAR) and also an experiment with VAEs, showing similar results
3) they removed the claims that were not well-supported.

Reviewer kUTC was happy with the changes and recommended a strong acceptance. The other two reviewers did not recommend acceptance, but justified their recommendation with the same arguments that they had put in the original review ignoring the important revision that the authors had uploaded (and not explaining why it was not sufficient). I have thus judged that the revision actually addressed their original concerns and that they should have updated their recommendation accordingly.

The authors should carefully proofread their paper for the camera ready version (perhaps with a spellchecker) as there are still some typos left.
Decision: Accept (poster)