Wasserstein Fair Autoencoders
Authors: Sungdong Lee
Hyunjong Lee
Joong-Ho Won
Venue: Submitted to ICLR 2023 (2023)
Keywords: conditional generation
fair representation
disentanglement
wasserstein autoencoder

Abstract: Autoencoders, or nonlinear factor models parameterized by neural networks, have become an indispensable tool for generative modeling and representation learning in high dimensions. Imposing structural constraints such as conditional independence on the latent variables (representation, or factors) in order to capture invariance or fairness with autoencoders has been attempted through adding ad hoc penalties to the loss function mostly in the variational autoencoder (VAE) context, often based on heuristic arguments. In this paper, we demonstrate that Wasserstein autoencoders (WAEs) are highly flexible in embracing structural constraints. Well-known extensions of VAEs for this purpose are gracefully handled within the framework of the seminal result by Tolstikhin et al. (2018). In particular, given a conditional independence structure of the generative model (decoder), corresponding encoder structure and penalties are induced from the functional constraints that define the WAE. This property of WAEs opens up a principled way of penalizing autoencoders to impose structural constraints. Utilizing this generative model structure, we present results on fair representation and conditional generation tasks, and compare them with other preceding methods.

Under review as a conference paper at ICLR 2023
WASSERSTEIN FAIR AUTOENCODERS
Anonymous authors Paper under double-blind review
ABSTRACT
Autoencoders, or nonlinear factor models parameterized by neural networks, have become an indispensable tool for generative modeling and representation learn- ing in high dimensions. Imposing structural constraints such as conditional in- dependence on the latent variables (representation, or factors) in order to capture invariance fairness with autoencoders has been attempted through adding ad hoc penalties to the loss function mostly in the variational autoencoder (VAE) context, often based on heuristic arguments. In this paper, demonstrate that Wasserstein autoencoders (WAEs) are highly flexible in embracing structural con- straints. Well-known extensions of VAES for this purpose are gracefully handled within the framework of the seminal result by Tolstikhin et al. (2018). In particu- lar, given conditional independence structure of the generative model (decoder), corresponding encoder structure and penalties are induced from the functional constraints that define the WAE. This property WAEs opens up a principled way of penalizing autoencoders to impose structural constraints. Utilizing this generative model structure, we present results on fair representation and condi- tional generation tasks, and compare them with other preceding methods.
INTRODUCTION
The ability to learn informative representation of data with minimal supervision is key challenge in machine learning (Tschannen et al., 2018), toward obtaining which autoencoders have become an indispensable toolkit. An autoencoder consists of the encoder, which maps the input to low- dimensional representation, and the decoder, that maps a representation back to a reconstruction of the input. Thus autoencoder can be considered nonlinear factor analysis model as the latent variable provided by the encoder carries the meaning of "representation" and the decoder can be used for generative modeling of the input data distribution. Most autoencoders can be formulated as minimizing some "distance" between the distribution Px of input random variable X and the distri- bution 9; Pz of the reconstruction distribution Pz and is either deterministic or probabilistic decoder (in the latter case is read as the conditional distribution of given Z), which variationally described in terms of an encoder Q For instance, the variational autoencoder (VAE, Kingma & Welling 2014) minimizes
g(Z), where Z is the latent variable representation having
DvAE(Px 9 Pz) inf Epx [DxL (Oxi(2xxlll P2) -EQ x logg((Z)
Q
over the set of probabilistic decoders or conditional densities of G given Z, where DKL. is the Kullback-Leibler (KI.) divergence, and the Wasserstein autoencoder (WAF, Tolstikhin et al. 2018) minimizes
(NAE(Px.9::'" Q
inf Exx EQzix d' (P(Xg(())
over the set of deterministic decoders g. where is the metric in the space of input X and 2 1. Set Q restricts the search space for the encoder. In VAEs, a popular choice is class of normal distributions
2 =[2zix regular conditional distribution -1X = x) ~ N (n(x), M(x)), (u, Σ) ENN
where A/V is class of functions parametrized by neural networks. In WAEs, the choice
Q = [Qzix regular conditional distribution Qz A Epy Qz = Pz}
(3)
makes the left-hand side of Eq. (2) equal to the (p-th power of) the p- p-Wasserstein distance between Px and 93 Pz (Tolstikhin al. 2018 Theorem 1); Qz is called an aggregate posterior of Z. If Q is

---

Under review as a conference paper at ICLR 2023
set of Dirac measures, i..., Q {Q Qzx== = =f(() f NN then minimizing Eq.
reduces to the learning problem of a deterministic unregularized autoencoder.
Of course, the notion of "informativeness" depends the downstream task. The variation in the observations that are not relevant to the particular task is often called "nuisance" and is desirable to be suppressed from the representation. For example, in finding "fair representations." (Zemell 2013) sensitive information such as gender or socioeconomic status should be removed from latent representations; in obtaining representations of facial images, those that are invariant to lighting conditions, poses, or wearing of eyeglasses are often sought. A popular approach to this goal is to explicitly separate informative and nuisance variables in the generative model by factorization. This approach imposes structure on the decoder. Additionally the encoder is further factorized and a penalty promoting independence between the encoded representation and nuisance variable can be added. A well-known example is the variational fair autoencoder (VFAE, Louizos et al.) 2016), in which a variant of the "M1+M2" graphical model (Kingma et al., 2014) is used to fac- torize the decoder and a resembling factorization of the encoder (variational posterior) is assumed. Independence of the representation from nuisance variable is encouraged by adding maximum mean discrepancy (MMD, Gretton et al. 2007) between conditional variational posteriors: in Lopez (MMD,,Gretton
let al. al. 2007). Other authors employ penalties derived from the mutual information (MI) (Moyer
(2018), MMD is replaced by the Hilbert-Schmidt Independence Criterion (HSIC, Gretton
et al. 2018, Song et al. 2019 Creager et al. 2019). Another example is the Fader Networks (Lam- ple al. (2018), in which the deterministic decoder takes an additional input of the attribute (such
as whether or not eyeglasses are present in a portrait) and an adversarial penalty that hinders the accurate prediction of the attribute by the deterministic, unfactorized encoder.
These examples illustrate that, while the generative model (decoder structure) can be chosen suit- way
ably for the downstream task, there no principled way of imposing the corresponding encoder structure. In this paper, we show that the WAE framework allows us to automatically determine the encoder structure corresponding to imposed decoder structure. Specifically, when the deterministic decoder in Eq. (2) is modified to handle the conditional independence structure of the imposed generative model, then the constraint set (amounting to the Q in Eq. (3)) that makes the LHS of Eq. 22) a proper (power of) Wasserstein distance determines the factorization of the (deterministic) encoder. In practice, the hard constraints in Q is relaxed and Eq. (2) is solved in penalized form. Following the approach Tolstikhin et al. (2018), the cited constraint set can be systemically trans- lated to penalties. Therefore, in addition to the theoretical advantage that the penalized form equals a genuine distributional distance for sufficiently large penalty parameter while that of Eq. ① remains a lower bound of the negative log-likelihood of the model, the ad hoc manner of designing penalties prevalent the VAE literature can be avoided in the WAE framework. Further, the allowance of deterministic encoder/decoder promotes better generation performance in many downstream tasks.
We explain how the WAE framework leads structured encoders given a generative model through examples reflecting downstream tasks in Sect. 3 after providing necessary background in Sect. 2 We would call these structured uses of WAEs the Wasserstein Fair Autoencoders (WFAEs). Af- ter reviewing relevant ideas in Sect. 4 WFAEs are experimented in Sect. 5 for datasets including VGGFace2 (Cao et al. 2018). We conclude the paper Sect.
2 PRELIMINARIES
In fitting given probability distribution Px of a random variable X on measurable space (X, B())), where X C RD equipped with metric d, by a generative model Pa of sample G on the same measurable space, one may consider minimizing the (pth power of) p- Wasserstein distance between the two distributions, i.e.,
min POEM
PP), := inf TEP(Px.Po)
ES d'
Here, M is the model space of probability distributions, P(Px. Pa) is the coupling the set of all
joint distributions on (X' x X, B(x x X))) having marginals Px and Pa. Often the sample G is
generated by transforming variable in latent space. When G = g (Z) a.s. for latent variable Z in a probability space (Z, Z C R', and measurable function g. then Pg is denoted by g; Pz, where is the push forward operator. In this setting, as discussed in Sect. Tolstikhin et al.
2

---

Under review as a conference paper at ICLR 2023
Zz
Z
8
Z1
8
Z
$
9
(a) Example
(b) Example 2
(c) Example
Figure 1: Examples of generative models for WFAEs
(Eq. 2 with the constraint set Q on probabilistic encoders Qz given in Eq. (3. It is further claimed by Patrini et al. (2020) that the set of conditional distributions Qzix can be reduced to be deterministic, i.e., 7. = f(X) a.s. for measurable. However, this claim
2018) show that WP(Px, 9: Pz) DWAF Px)
general true unless g is injective:
Let d(x,, y) |z vlle for y, y X. If Pxx has density with respect the Lebesgue measure, and the measurable function g :: z X is injective, then
Theorem
W Pz) inf Ex (X..(((X)))
(4)
SEQ
where the set of all measurable functions from X to 2 such that f2Px Pz.
The proof of this result is provided in Appendix A of the Supplement.
Remark In Patrini el al. (2020) Theorem A.2), is incorrectly claimed that for the right in- verse ÿ of when the codomain of the latter is restricted to its range, (ğ g);P2(F) is equal to Pz(ğ ( (F)), instead of the correct Pz(g- (Ā (F)).. This confusion invalidates the rest of the argument of the cited theorem.
In practice the set Q can be relaxed to F, a class of all measurable functions parameterized by deep neural networks, which contains minimizer the right-hand side (RHS) of Eq. 44; the constraint fpPx = Pz be met by adding penalty XD(f, Sifffxxl P2) ) for sufficiently large multiplier À and divergence 'D between two distribution. Thus we define the distortion criterion
8(f g) Epx & (X, ((f(x)) XD(f:Px||22)
then the generative modeling problem based on Wasserstein distance can be formulated as
inf inf 8(f.g), geg fer
(5)
for G set of injective measurable functions from Z to X, typically parameterized by deep neural networks. The function f X' Z has an interpretation of an encoder and : Z - has an interpretation of decoder. Typically I < D.
LEARNING INVARIANT REPRESENTATIONS WITH WFAES
Often generative modeling is more complicated than just involving a latent variable Z in 2 and its reconstruction G in X. For example. data may come with labels, which can employed in the generation process learn invariant representations.
Example Let begin with simple generative model shown in Fig. la Louizos et al. 2016 Fig. 1); see also KKngmaa et al. 2014 M2). Here, variable S C R represents the observed nuisance variation, and Z models the remaining information on G (with which we want to mimic the observable variable X) that independent of S. Thus the Z encodes the representation invariant to the unwanted variation in S. Denoting the marginal distribution of the nuisance variable S by Ps. the distribution of model is g3 (Pz & Ps). where & is used to denote a product distribution. The goal is to make the joint distribution Pas of (G, S) close to Pxs (X,)) the observable.
3

---

Under review as a conference paper at ICLR 2023
If we let ((2,s) = (g((,, s), then Pas 0:(Pz Ps). Recall that X' is equipped with metric d. Equip S with another metric d' and X S with d v + (d')². Then, by applying Theorem Pxs. Pz @ Ps. and 9. we obtain
(Pxs, (P2 8 Ps)) inf Exx d (Ix,5),8(i(x,5))
feF
inf E Pxs d (Ix, ()(( = inf Pxxs df (X, JEF
fer
where =
2: (f.lls) P2 U
Pz & Ps). and lIs x + S IIs is the orthogonal projection from X x S onto
S. The second equality holds by noting that f(r,s) sf( s)) and taking h Ils. The latter constraint set F means that
-
f(X,S) S.
(6)
Following formulation equation the unstructured case, we can incorporate constraint equation into the learning problem in a penalized form
min min Exs de (x, (Xxx( + A D(SSPsss (Tss(() g
where D is an appropriate divergence between two probability distributions such as MMD or the generative adversarial network (GAN) loss as suggested by Tolstikhin et al. (2018) and H. promotes independence between two random variables f(X,S) S) and 8, such as the HSIC (Lopez al. 2018
Example 2 Consider more involved generative model shown in Fig. Tb which is employed by semi-supervised model Kingma et al. 2014). This graphical model actually describes the conditional distribution Pxis of X given S, since 5 and Y are allowed to be correlated. Instead, it is required
the VFAE (Louizos et al. 2016 Fig. 2) as an extension of the "MI + M2"
21 S
in order to impose invariance to the nuisance variable S. Let xX as g(y, 22, s) (gr (g2(y, 2))..). U, s). Denoting the marginal distribution of the nuisance variable S by Ps and the joint distribution of Y and S by Pys, the distribution of model G (Pys @ Pz₂). The goal is to make the joint distribution Pas of (G.S) close to Pxs of (X,S) when Y is not observed, and PaYs of (G, Y,S) to Pxys of (X, S) when the data is fully observed.
First consider the case that Y is missing. Let IIxs be the orthogonal projection operator from X' x y S onto X x S. Then by applying Theorem I Pxs. Pys @ Pz,, and Ixxgg, we obtain
inf w3 (Pxs, Ix Pys @ P2.)) fEFanots
Exxs d (xxx
inf Exxs d (1,S), inn (g2(s(X S)), S), S)) JEFunobe
inf Exxo dp (Xx ,1 (92(S ()) S)), feFunabe
where
Funobs = {(Luubbe S ]lnobe)| funobs : X x S y Junobs
X x S +
Z2, (funobs Mss f2 funobe ) Pxs Pys @ Pz₂ The latter constraint set means
(funabs S)S), (Y,S),
f Funobe (X,S) _ Zz, (Jrmobs(X (X,S), 1 5 pumobs (X, (8)
Now consider the case is observed. Equip y' with a metric d" and X y S with d = Vi + (d')2 + (d')z. Then by applying Theorem to PxYs. Prs & Pz,. and g. we obtain
9P @ P2.))
inf EF
Erxve d ([X,Y, robs (X, S))
inf Erxys X (I Y. (g (g2 / (x, S),,
inf Epxye EFot
4

---

Under review as a conference paper at ICLR 2023
where Fobs {f2ths X x V x x + 22 (Ily, IIs. f2 " Pys @ Pza and Ily (x, y, 8) + y, IIs (2, y. 8) + s are projections. The latter constraint set means
12 (x, XYY _ Z2, (V.S) L 1 (X,Y,))
(9)
In order to combine the two Wasserstein losses and constraints Eq. 7 to 9). let us extend y to
D y U (*), where '*' represents the missing value. For any (f1 ffmmass junnoss) E Funobs and
J_2
f2 E Fobs. define fi : .. x y x S + Y and f2 X' x y x S 7 Zz as
ys (simaba (I, 8),
y vサ* *, y ,,
5 (r, 8),
y #
fi 8)
f2 v.
[2mobe (x, 8),
y =
Then we can formulate the learning problem for the WFAE in penalized form
min min Exxys dl d²(x (X.g1(e((Y
+ Exxs d + x, S)) (UJi, (s)),,,,ss))sPPsll/s/sl + (2D2((2 + (U), H,H * (i)xs Pzz));
,, ())
mad f1.12
where 92* * fi (x,y, y, 22, 22, (92 (2r, y, 22), D1 and D2 are appropriate divergences between two probability distributions, and H3. Hi promotes independence between two random variables. Note, unlike Example in which only the encoder f is constrained, Eq. 7 imposes a constraint on the decoder 92. Also note that, the divergence D2 can be estimated in two-sample fashion, namely from the a sample drawn from Pys, S, i.e., (yt, 8;) with y/: observed, and another sample drawn from (fi, Is } Pxys, either as (y,, { if yj is observed or (fi (x). *, s,),, 8j) otherwise. Hence all the data from the minibatch can be utilized. Likewise, divergence D2 and the independence penalties H3 and H4 can utilize the full minibatch.
Remark 2 VAE-based models, e.g., VFAE, assume a specific factorization of the variational poste- rior (encoder). Since the factor qs (z(2) for imputing Y does not appear in the evidence lower bound (ELBO) of the observed likelihood, an additional penalty on this factor evaluated for the fully ob- served sample is coined (Louizos et al. 2016 Eq. 5), making the bound not tight. In the WFAF, on the contrary, the D1 term that arises naturally from constraint equation for the Wasserstein distance penalizes the imputing encoder Ji for both fully (by requiring S S1 Mj. 83) yj) and partially (by the divergence) observed samples.
The model shown in Fig. exexenns Example 7 with two independent nuisance variables that can be missing. Here Y may represent a person's 's identity in her portrait, which may be missing, and S partially observed attributes (e.g., sunglasses on/off. mouth open/closed, and gender). In this setup we want two different portraits of a person to have similar values of Z, and those of two different people to have quite distinct values of Z, even the encoder does not know whose portraits they are. We may also want Z to represent something immune even to gender switch.
Example 3
Proceeding as Example 2 we obtain for g : x + Xxx X Glu. 2, 8) = (g(y, z, 8),
W Ix0:
Pz Ps))
inf
Epx (X. (X), (X), (X))),
U EFFX
FX = fa) (f1, fa) Py 8 Pz 2 Ps. fi x + f2 X + 8
when both Y and S are unobserved,
W (Pxs, (Py @ Pz Ps)) (//5 inf Dexxs
Ep., (x,)
xs (2) (f1, fe, Us) = Py P fi X - f2: {
when only Y is unobserved,
(Pxy, ( ( & 2 Ps)) = (s JexxY Ep,y (X, NY X (X,
inf
JXY (s/2f18) (Ny, fs); Py Pz 3 Ps, 2: xx - 1
5

---

Under review as a conference paper at ICLR 2023
when only S is unobserved. and
W (Pxrs, (P @ Pz 8 Ps))
inf
Exxss (X, 12, (X,Y, (
JXYS = [/2 (Ily, [s) Py Px Ps. h : 2
when the data are fully observed. If we expand to y = yu and u Su {*), then the learning problem is
min min (J1.J.la)
ExXYS (xx.g(X
+ Epxy 6[
+ Epxs d[xx.9(f((xx,55))).... *,
+ Bpx ((X,,((f((X,+,))), *, x), Sa X, *, +))))
+ ( b22 s P2) +
(11 fs):
where H4 measures dependence of three random variables, e.g., the d-variate HSIC Lopez et al.
2018 with d = 3. and F = {(fi, fa) fi X x V, X y x 2, Xx S + SJ. for
(x),
y=
fg (I),
/=*,8=*
(2,
y ==
fo(2,
PX (I,y),
y ***,8=6
y/.
***
8,
$#*,
and f2(21, U, s) is equal to f2 (x) = to f2 (2, if x,8 y = 8,8 + *, and to fXYS (z. y, s) otherwise.
f2x (2, if
Remark 3 If variable Y is removed and S fully observed, Example 3 reduces to Example 7 where the f2(2. v. from the former corresponds to the fxx from the latter. The Fader Networks
Lample et al. 2018)) implicitly use this model to obtain perributeinnvarie images. The adversarial penalty for training the network (Lample et al. 2018] Eq. 2) can be understood as promoting independence between S and 2 = ((X,S). While in Lample el al. (2018) the encoder f does not depend on S, Example Example[shorr ssows that it is more natural to take S as an input to remove its effect on Z. Example 3 can be considered a generalization of the Fader Networks for missing attributes and unknown identities.
representations of facial
4 RELATED WORK
The literature VAES is vast. 3-VAE (Higgins et al., 2017) is one of well-known ways of adding penalties to the ELBO of a VAE, which adds one proportional to the expected KL divergence be- tween the variational posterior (encoder) and prior Pz. It is observed that this penalty promotes fac- torization of the aggregate posterior Q 2 (Kim& & |mih|22188) In fair representation, VFAE (Louizos) et al. 2016) and HSIC-constrained VAH (HCV, Lopez (HCV,LLopzz semi-supervised disentanglement along this line. Adversarial penalties have been also considered (Edwards & Storkey| 2016; Madras al. 2018). [Song et al. (2019) bring an dnfformation-thhoreee interpretation to these approaches. In this regard, penalizing MI between nuisance variable S and encoded latent variable Z (Moyer al. 2018 Song al., 2019 Creager al., 2019) its tractable upper bounds, e.g., based on a variational approximation
al. 2018) add penalties to the ELOO for
Gálliez et al. (021). has been advocated. Recently proposed FairDisCo (Liu et al., (022)) uses the Le distance between the joint density of S and Z and the product density of their marginals, showing its asymptotic equivalence to the MI. However, as stated in Sect. [ these penalties promoting desired structures are chosen rather ad hoc and loosens the already- encoder structure corresponding the imposed decoder structure.
ELBO. Furthermore, there is no principle for choosing the
The WAE framework discussed in the previous section can overcome these pitfalls in VAEs. The WAE literature has focused on improving the divergence in the penalized form of Eq. ( that
matches the prior Pz and the aggregated posterior. The original proposal by Tolstikhin et al. (2018) is to employ either the MMD or GAN divergence. KKlouri et al. (2019) propose to use the sliced Wasserstein distance in order to simplify computation. Patrini et al. (2020) consider the Sinkhorn

---

Under review as a conference paper at ICLR 2023
Adult Income
Health
person
0.35
0.00
03:01
yAUI
0.00
0.70
0.60
0.66
0.55
Age
App
lighting
0.50
yAUC
0.22
WFAF RirDisSoo
0.70
0.70
0.90
-35
-00
10
15
0.65
1ike
SAUC
SAUC
Figure T-SNE map of Z1 in Ex- tended Yale B
Figure 2: Fair representation trade-off.
divergence (Genvvay al. 2018), computing of which can be boosted by using the Sinkhorn algo- rithm (Cuturi 2013). Xu et al. and Nguyen et al. (202I) propose and improve the relational divergence called the fused Gromov- Wasserstein distance. The latter three works consider the set- ting in which the prior Pz is structured. In contrast, we focus on the setting in which the decoder is structured and nuisance information is (partially) available. According to the taxonomy of Tschan nen et al. (2018), the former is close to the clustering meta-prior whereas the latter is close to the disentangling one. We emphasize that the cited divergences are compatible with framework.
EXPERIMENTS
We experimented WFAEs with various real-world datasets. The generative models for these datasets mainly follow Examples 2 and 3 in most of which variable Y (and sometimes S) has the meaning of a label and thus categorical. In order to embed this variable to the Euclidean space RB where employed the entity embedding network (Guo & Berkhahn (016) for observed labels. The trained embedding network naturally becomes pretrained encoder /1 or fs from Examples and and A by-product of this embedding is that it is even possible to impute categories not present in training data.
B does not necessarily depend on the number of categories,
5.1 FAIR REPRESENTATIONS
To demonstrate the performance of WFAEs on fair representation, we reproduced experiments in Liu et al. (2022) using two categorical datasets, namely the Adult Income and Health datasets. Refer to the appendix for data summary and network implementation. The generative model for the WFAE was the structure of Example 2 With the Zi 92 (Y, f2 S)) encoded from the trained model, we quantified the trade-off between fairness and utility (Zhao et al. 7:: we classified S and Y using random forest method, calculated the area under the ROC curve (AUC) on the test data (sAUC and YACC) as a function of demographic parity Ape, and compared the performance with the HSIC- constrained VFAE (HCV) and the FairDisCo. The results are summarized in Fig. 2 While WFAE shows a clear trade-off, other methods are relatively insensitive to demographic parity.
5.2 INVARIANT REPRESENTATION
The same structure as Sect. 5.1 is used to test the ability of WFAEs to learn invariant representations of controlled photographs. The cropped Extended Yale B dataset Georghiades al. (2001); Lee et al. (2005) comprises of facial images of 38 human subjects in various lighting conditions. For each subject, the pictures the person are split into training and test data with a fixed ratio, resulting in 1,664 and 750 images for the training and test respectively. We set the identity of the image as Y and the lighting condition (elevation and azimuth of the light direction normalized into [-1, x [)) as S. In the training stage, first trained fi to estimate Y, then trained the rest of the network with fi held fixed. In consequence, we were able to encode and decode the test data without the information about Y by replacing it with (X, S). Although we trained the model
7

---

Under review as a conference paper at ICLR 2023
Net- work
ID
Lighting group (Acc.)
Lightnn direction (MSE)
mey
RF
Lagistic
RF
Linear
Extended Yale B
MNIST
Netwark
VAE
0.71
0.74
0.74
0.03
0.07
Sharpness
FID
Sharpeess
FID
HCV
0.75
0.60
0.45
0.13
0.22
Fa
1.0
0.34
0.37
FairDisCo
3,36o-4
63.7
3.45o-2
111.1
WFAF
0.97
0.35
0.28
0.23
0.23
WHAB
4.43e-4
66.5
9.796-2
19.4
Test
3.09e.3
1.83e11
Table 1: Invariant representation of Extended Yale B. Rf=random forest, Logistic=logistic regression, Lin- Table 2: Sample generation quality ear=linear regression. Classification accuracy for discrete measures. and mean squared error for continuous variables.
with continuous S, we present some of the results with S categorized in 5 directions, in Lopez al. (2018). The results are presented [] The Zi encoded by WFAE shows better performance in predicting Y and worse in predicting S than others, suggesting better invariant representation. The t-SNE visualization of Z1 in Fig. Fjacordss with this result, showing noticeable separation with respect Y. but not with respect S. In panel A of Fig. (top left), the green box depicts generated images by encoding the test image X and nuisance data S into 21 = 92 92 (X, *, S), f2(X, *, S)), and then computing 91 (Z1.S) S). Those in the red box were generated by using the same Z1 but setting S = (=0.3, +..3). WFAE produced reconstructions closer to the input than HCV and FairDisCo, and perturbing& S only kept the identity of the input in the generated images. The sharpness and the Fréchet inception distance (FID) scores are shown Table221 to assess the sample generation quality. WFAE produced sharper images than FairDisCo, confirming the visual inspection. The FID scores should be taken with caution. though. Since the sample generation is conducted by varying the "lighting direction" attribute (considered as the 5 variable) the generated samples should be different from the test data with scarce images. Rather, it may indicate samples generated from FairDisCo is less sensitive to S, which can also be verified visually.
5.3 CONDITIONAL GENERATION
We further investigated the conditional generation capability of WFAEs using the MNIST and VG- GFace2 datasets (Cao et al. 2018).
MNIST. We treated the digit attribute as S. The generative model for the data is similar to Ex- ample 3 but without Y. We first trained encoder I's that estimates S, then trained the rest of the network. The final network was tested with images with digit information removed. We also trained network without the encoder f2 for Z, hence it decodes summarize the results, all of which were generated from test data without information of S. Penal A (top right) shows decoded samples from g with estimated S and i) not using f2 (blue box), ii) using encoded = fX, S) from the test data (green box), and iii) using Z sampled from prior Pz (red box). Decoded images with the same S all retained their digit information. Reconstruction without using /a. although recognizable, produced degraded images, implying loss of information. FairDisCo with similar architecture produced quite degraded results; see also Table 2 In panel B (top), we estimated Z from the source and S from the target and generated new images by 9(Z,
image using only estimated S. Fig.
VGGFace2. This dataset contains 3. 3.14M training images of faces of total 8631 subjects and 169k test images of total 500 subjects, with partially observed binary attributes such as gender, wearing of sunglasses, and openness of mouth, available for a subset of 30,000 images. Here, we treat the identity of the image as class Y and the vector of attributes as S. The generative model for this dataset the same as Example 3 The dlassprrsserving generation and style transfer tasks were conducted in the same manner as MNIST. In addition, we also tried generating samples with manipulated attributes. Since the attribute encoder fa embeds S in the Euclidean space, we could extrapolate input S to decoder g beyond 0 and 1. For this attribute manipulation task, we compared results with Fader Networks trained with a similar architecture. Fig. [ shows sample images for all tasks. Although images of persons who were not in the training data were used, the WFAE could successfully generate images retaining the identity while employing other identity invariayinvariaan
In this experiment there is no Y and the digit class plays the role of S. So the generation is usual class- conditional one, hence the FID scores are lower for the WFAF as expected from visual comparison.
8

---

Under review as a conference paper at ICLR 2023
A. Class-preserving generation
MNIST
WFAA
WFAE
Extended Yale B
FairDisCo
Reconstruction (wilhout
5 6
7
8 9
SCo
Test
Test Cata
5 6
7
8 8
9
5 6
7
8
9
Reconstrutinn
56
9
5 6
7
8
9
Reconstructian
Generstion
5 6
7
8
9
6
7
8
9
Generatian
5 6
7
8 9
5
6
7
8
9
5
6
8
9
5 6
7
8
9
B. Style transfer
WFAE
FairDisCo
5
6
7
8
9
5 6
7
8
9
0
5
6
7
8
9
5 6
7
8
9 9 9 9
Test Data
Source Data
5
6
7
8
9
5
6
7
8 8
2
5
6
7
8
9
5
6
7
Reconstruction
3
5
6
7
8 8
9
5
6
7
8
Generation
4
5
6
7
9
5
6
7
8
9
Target Data
C. Attribute manipulation and interpolation
Mouth open +
Test Data
Source Data
Mouth
Fader- Networks
WFAE
Mouth open
Figure Conditional generation examples of WFAE
features, c.g., camera angle, lighting condition (panel A, bottom). Note in the style transfer task (panel B. bottom), the generated images possess the styles from the source data and tend preserve the specified attribute of the target data. For example, the generated images tend to have open mouth the target image has mouth wide open. In the attribute manipulation task, we could successfully generate images with the desired attributes changed. In panel C, letting the "Mouth Open" attribute positive produced decoded images having grinning mouth; making it negative produced images with lips all closed. For the Fader Networks, we extrapolated the attribute scores to large magnitude as far as +4400, but it only caused deformation of the original image; see Remark
CONCLUSION
We have shown that the WAE framework is rich enough to handle various conditional independence structures, leading to much more principled formulation of learning problems than the VAE counter- parts. Importantly, conditional independence structure imposes the decoder automatically de- termines the encoder structure and the associated constraints. We hope this paper stimulates further research extensions of WAEs in this direction, for instance, complex hierarchical structures.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. VGGFace2: A dataset for recognising faces across pose and age. In Proc. 13th IEEE Int. Conf. Automatic Face & Gesture Recognition (FG 2018), Pp. 67-74. IEEE, 2018.
Elliot Creager, David Madras, and Richard Zemel. Flexibly fair representation learning by disentanglement. In International conference on machine learning, pp. 1436-1445. PMLR, 2019.
Jacobsen, Marissa Weis, Kevin Swersky, Toniann Pitassi,
Marco Cuturi. Sinkhorn Distances: Lightspeed Computation of Optimal Transport. In Adv. Neural Inf. Process. Syst. (NIPS 2013), 2013.
Harrison Edwards and Amos Storkey. Censoring representations with an adversary. In International Conference on Learning Representations, 2016.
Aude Genevay, Gabriel Peyre, and Marco Cuturi. Learning Generative Models with Sinkhorn Di- vergences. In Proc. 2/ Int. Conf. Artif. Intell. Statist. (AISTAT' 20/8), pp. 1608-1617. PMLR, 2018. URL https: http:////ooceeddeeee ml wressrrsss/eee genevaylee html ISSN: 2640-3498.
Athinodoros S. Georghiades, Peter N. Belhumeur, and David J. Kriegman. From few to many: Illumination cone models for face recognition under variable lighting and pose. IEEE Trans. Pattern Anal. Mach. Intell., (3(6))6436660 2001.
Arthur Gretton, Kenji Fukumizu, Choon Teo, Le Song, Bernhard Schölkopf, and Alex Smola. A kernel statistical test of independence. Adv. Neural Inf. Process. Syst. (NIPS 2007), 20, 2007.
Cheng Guo and Felix Berkhahn. Entity Embeddings of Categorical Variables. April 2016. URL
wwtp:/wwwww.rg.///////..... 0673777
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick,
Shakir Mohamed, and Alexander Lerchner. 3-VAE: Learning Basic Visual Concepts with Con- strained Variational Framework. In Proc. Int. Conf. Learn. Represent. (ICLR 2017), pp. 22, 2017.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising, In Proc. Int. Conf. Mach. Learn. (ICMI. 2018), pp- 2659-2658. PMIR, 2018
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Proc. Inf. Conf. Learn. Represent. (ICLR 2014), 12 2014.
Diederík Kingma and Max Welling. Auto-encoding variational Bayes. In Proc. Int. Conf. Learn. Represent. (ICLR 2014), 2014.
Diederik P. Kingma, Danilo J. Rezende, Shakir Mohamed, and Max Welling. Semi- Supi-Super Learning with Deep Generative Models. In Adv. Neural Inf. Process. Syst. (NIPS 2014), arXiv, October 2014. Number: arXiv:1106.522 5298 arXiv:1406.5299 [cs, stat].
Soheil Kolouri, Phillip E Pope, Charles E Martin, and Gustavo K Rohde. Slised-wasssrstein auto- encoders. Proc. Int. Conf. Learn. Represent. (ICLR 2019), pp. 19, 2019.
Guillaume Lample, Neil Zeghidour, Nicolas Usunier, Antoine Bordes, Ludovic Denoyer, and Mare Aurelio Ranzato. Fader Networks: Manipulating Images by Sliding Attriutte.. In Adv. Neural Inf. Process. Syst. (NIPS 2018). arXiv, January 2018. URL /attp://axxvv.rrr Number: arXiv:1706 000099 arXiv: 700090000090 [cs].
abs/ abs/1706.co 00409
Kuang-Chih Lee, Jeffrey Ho, and J Kriegman. Acquiring linear subspaces for face recognition under variable lighting. IEEE Trans. Pattern Anal. Mach. Intell., 27(5))684-698, 2005.
Liu, Zenan Li, Yuan Yao, Feng Xu, Xiaoxing Ma, Miao Xu, and Hanghang Tong. Fair represen- tation learning: An alternative to mutual information. In Proc. ACM SIGKDD Conf. Knowledge Discovery and Data Mining (DD 2022), pp. 1088-1097, 1097. Association for Computing Machinery, 2022. TRI.https. / http://////oiiooo...1..1111 1ttpg//....113551111 http://.....rr/..111133333332223 3539302
10

---

Under review as a conference paper at ICLR 2023
Romain Lopez, Jeffrey Regier, Michael Jordan, and Nir Yosef. Information constraints auto- encoding variational Bayes. In Adv. Neural Inf. Process. Syst. (NeurIPS 2018), pp. 6114-6125, 2018.
Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, and Richard S Zemel. The variational fair autoencoder. In Proc. Int. Conf. Learn. Represent. (ICLR 2016), 2016.
David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. Learning adversarially fair and transferable representations. In International Conference on Machine Learning, pp. 3384-3393. PMLR, 2018.
Daniel Moyer, Shuyang Gao, Rob Brekelmans, Aram Galstyan, and Greg Ver Steeg. Invariant representations without adversarial training. Advances in Neural Information Processing Systems, 31, 2018.
Khai Nguyen, Son Nguyen, Nhat Ho, Tung Pham, and Hung Bui. Improving Relational Regular- ized Autoencoders with Spherical Sliced Fused Gromov Wasserstein. Proc. Int. Conf. Learn. Represent. (ICLR 202/), 2021. URL http::/arxiv.orgg/ wttp:rgggbbbs/ooooo.. 01787 arXiv: 2010.01787.
Giorgio Patrini, Rianne van den Berg, Patrick Forre, Marcello Carioni, Samarth Bhargav, Max Welling, Tim Genewein, and Frank Nielsen. Sinkhorn autoencoders. In Uncertain. Artif. Intell. (UAI 2020), pp. 733-743, 2020.
Gabriel Peyré and Marco Cuturi. Computational optimal transport. Foundations and Trends in Machine Learning, 11(5-6):355-4 6):355)-35777 2019.
Borja Rodriguez-Gálvez, Ragnar Thobaben, and Mikael Skoglund. A variational approach to pri- vacy and fairess. In 2021 IEEE Information Theory Workshop (ITW), pp. 1-6. IEEE, 2021.
Jiaming Song, Pratyusha Kalluri, Aditya Grover, Shengjia Zhao, and Stefano Ermon. Learning controllable fair representations. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2164-2173. PMLR, 2019.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schölkopf. Wasserstein auto- encoders. In Proc. Int. Conf. Learn. Represent. (ICLR 2018), 2018.
Michael Tschannen, Olivier Bachem, and Mario Lucic. Recent Advances in Autoencoder- Based Representation Learning. In 3rd Workshop on Bayesian Deep Learning, Montreal, Canada, De- cember 2018. URL http: wwwpa/wwwwii.rraeeeoaaeeeeeeeeee
wwt.oorgg///o.ooooooooooooeeaeae abaa 05069 arXiv: 1812.05069.
Hongteng Xu, Dixin Luo, Ricardo Henao, Svati Shah, and Lawrence Carin. Learning Autoencoders with Relational Regularization. In Proc. Int. Conf. Mach. Learn. (ICMI. 2020), volume 119, pp. 10576-10586. PMLR, 2020. arXiv: 2002.02913.
Omry Yadan. Hydra a framework for elegantly configuring complex applications. Github, 2019.
URL / http://www.....cccccccocceeeee/a
Richard Zemel. Learning Fair Representations. In Proc. Int. Conf. Mach. Learn. (ICML 2013), volume 28 of 3, pp. 325-333. PMLR, 2013.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Learning hierarchical features from deep gener- ative models. In Proc. Int. Conf. Mach. Learn. (ICML 2017), pp. 4091-4099, 2017.
A PROOF OF THEOREM
Proof Under the conditions of the theorem statement, the Konte-kantoo equivalence holds (see, e.g., Peyré & Cuturi 2019 Theorem 2.1):
W2 Pa)= T.X TXX-XT.PxPPC
inf Epx

---

Under review as a conference paper at ICLR 2023
Hence it suffices show that
/ (
inf 1:2 f: Px =
inf Tx x
T())dXx
or equivalently
f:
Pa..
The forward inclusion C holds since for any measurable f such that f&Px = Px8-1= Pz we have J: X - X' measurable and for any Borel set E C X
- fx Px - (p
= =9:1Px9 - 9.
For the backward inclusion D. suppose T - X is measurable and satisfies TyPx = Pg. Since
g is injective, it has a left inverse g¹ :\' 2. Let f = 96 g¹ c Then for any Borel set C 2,
fPP(FF) = Px (F) = (x))
(TPx((1))=(FF)
-Po((o))((F))
- -)(P((())~((F))
-P2 ((0)"
P2((g -g) Pz(F),
which completes the proof.
B ADDITIONAL DETAILS FOR THE FAIR REPRESENTATION EXPERIMENT
Following Liu et al. 2022). fair representation experiment were held for the Adult Income and
Health datasets, whose characteristics are described in Table Note that all variables were catego- rized: one-hot encoding was used for variables with multiple category to make all data either 0 or 1. The encoder-decoder architecture of the network was adopted from (Louizos et al.) 2016) (Table
Training data size
Test Data Covariate Dimension
Response Variable
Sensitive
Data
Number
Variable
Adult Income
30162 44116
15060
115
account hospitalization
gender
Health
11030
254
age
Table 3: Information categorical datasets for fair representation task.
12

---

Under review as a conference paper at ICLR 2023
C FURTHER IMPLEMENTATION DETAILS
In the source code attached, settings for all experiments are gathered as configuration files in direc- tory configs fsnfigg///e tttiiiainiin.....o All the network architecture are listed in py files in src/src//oo prc/model directory, and model and architectur ure keyword in the configuration file specifies which ar- chitecture to use among them. The running script run. sh that states which configuration was used for each experiment, managed by Hydra 1.1.1 Yadan (2019). can be found in the exper experiments directory. The prior Pz for the encoded Z was set the dimension of the latent space E. For the penalty divergences DD we used the generative adver- sarial network (GAN) loss, which requires an additional discriminator (Tolstikhin et al.) 2018). All the networks were trained using ADAM (Kingma & Ba 2014) without any learning rate scheduling.
normal distribution N (0. 20,2 where I is
Extended Yale B The cropped version of the Extended Yale Face Database B dataset (Georghiades et... 2001 Lee et al. 2005)
resized into size of 128 128. The encoder-decoder architecture of the network had total of 18.5M parameters, and the discriminator architecture had 881 parameters (Table 5). After pre-training the Y -encoder with 2,100 iterations, we optimized the network for 5,200 iterations, which took about 40 minutes. The results were compared with the HSIC-contrained variational fair autoencoder (HCV, T and FairDisCo Table
The encoder-decoder architecture of the network had 3.8M parameters, and the discrim- inator architecture had 7.4k parameters (Table Sp. We pre-trained the S-encoder with 6,000 itera- tions, then optimized the rest of the network for 11,700 iterations, which took about half hour. The results were compared with HCV and FairDisCo with the 5 information available for decoding (Table 9 and Table 10).
MNIST
The face region of the collected data were cropped and resized into a size of 128 x 128. The encoder-decoder architecture of the network had 88.4M parameters, and the discriminator ar- chitecture had 206k parameters (Table [1}. We pre-trained the (YnSooen with 3,000 iterations, then optimized the rest of the network for 30,000 iterations, which took 16 hours. The results were compared with the Fader Network having an encoder-decoder architecture with 70.2M parameters and discriminator architecture with 483k parameters (Table trained for 20,000 iterations, which took 11 hours.
VGGFace2
Computing infrastructure We trained the networks with Intel@ Xeon@ CPU E5-2650 v4 @ 2.20GHz processors and Nvidia Titan X Pascal GPUs with 12GB memory. For the VGGFace2 experiments, we trained the network using four GPUs; those for the other experiments were all trained using single GPU. All the implementations were based on Python 3.6, PyTorch 1.10.2, Py PyTorch Lightning 1.5.10, and CUDA 10.2.
13

---

Under review as a conference paper at ICLR 2023
Map
Layer
Operation
Filters
Batch norm
Activation
Linked layer
f2
I
Dense
100
Yes
ReLU
X
Dense
dz
No
1, Y.Y.
d²
Dense
No
1. 1.Y.S
Output (22)X,
Sample Z2 X, Y,
H,
1
Dense
100
Yes
ReLU
92, S
2
Dense
dx
No
Sigmoid
92
1
Dense
100
Yes
ReLU
22, S
"
Dense
No
Dense
No
Output (21 Y)
Sample Z1 (2,, Y
14,
Discriminator
1
Dense
4dz
No
ReLU
2
Dense
4dg
No
ReL.U
3
Dense
4dz
No
ReLU
2
4
Dense
4dz
No
ReLU
3
5
Dense
No
Table 4: WFAE architecture for the faimess representation experiment: for the Adult income dataset, dx 115, dz = 50, and for the Health dataset, dx = 254, dz
Map
Layer
Operation
Filters
Kernel
Strides
Batch
Activation
Linked layer
Convolution
64
5x5
2x2
ReLU
X
2
Convolution
128
5x5
2x2
Yes
ReL.U
3
Convolution
5x5
2x2
ReLU
4
Convolution
512
3x3
2x2
Yes
ReL.U
5
Convolution
1024
3x3
2x2
Yes
ReLU
Dense
fa
1
Convolution
2x2
ReLU
X
2
Convolution
64
5x5
2x2
Yes
ReLUU
3
Convolution
128
5x5
2x2
Yes
ReLU
Convolution
3x3
ReLUU
5
Convolution
512
3x3
2x2
Yes
ReLU
Dense
91
Dense
8x8x1024
9a
2
Transpose Convolution
512
3x3
2x2
Yes
LeakyReLU
3
Transpose Convolution
256
3x3
2x2
Yes
LeakyRel.U
4
Convolution
3x3
Ixl
LeakyReLU
5
Transpose Convolution
128
5x5
2x2
Yes
LeakyReL.U
6
Transpose Convolution
64
5x5
2x2
Yes
LeakyReLU
Convolution
Ixl
LeakyReLU
8
Convolution
5x5
Ixl
No
Sigmoid
1
Dense
50
Yes
LeakyReLU
fi.
Discriminator
1
Danse
16
No
ReLU
fa
2
Dense
16
ReL.U
3
Dense
ReLU
4
Dense
16
No
ReLU
5
Dense
Table 5: WFAE architecture for the Extended Yale B dataset.
14

---

Under review as a conference paper at ICLR 2023
Map
Layer
Operation
Filters
Kermel
Strides
Linked layer
Q21xx
Convolation
64
5x5
2x2
Yes
ReLU
X
Convolation
128
5x5
2x2
Yes
ReLU
Convolaticn
256
5x5
2x2
Yes
ReLU
Convolation
512
3x3
2x2
Yes
ReLU
Convoluticn
1024
3x3
2x2
Yes
ReLU
Dense
No
5.
Dense
No
Output (Zi SX
Sample Zi X,8 8
QaaaLy
Dense
Yes
ReL.U
Z,
Dense
No
Dense
No
Output 77
Sample 7g
14,
Qrizi
Yes
ReLU
2.
2
Donse
No
P2,12,
ReL.U
Zz.
Dense
No
Donse
No
Outpui (2,
Sample Z1 12, Y
Px
Dense
8x8x1024
No
2,.8
Transpose Convolation
512
3x3
2x2
Yes
ReL.U
3
Transpose Convolation
256
3x3
2x2
Yes
ReLU
Convolation
256
3x3
1x1
Yes
ReL.U
Transpose Convolation
128
5x5
2x2
Yes
ReLU
Transuose Convolation
5x5
2x2
Yes
ReL.U
7
Convolation
5x5
1xl
Yes
ReLU
Convolation
5x5
1x1
No
Sigmoid
Table HCV architecture for the Extended Yale B dataset.
Map
Layer
Operation
Filters
Kernel
Strides
Batch
Activation
Linked layer
Q7
Convolation
32
5x5
2x2
Yes
ReLU
X
Convolation
ReLUU
Convolation
128
5x5
2x2
Yes
ReLU
2
Convolation
ReLUU
5
Convolation
512
3x3
2x2
Yes
Rel.1]
d²
Dense
No
S.
Output (Z,]X
Sample Za
o²
Px. 1
Dense
50
Yes
ReLU
2g,
Dense
10
No
Dense
Na
Oulput (2, 12,
Sample 12,,
P.
Pxizus
Dense
8x8x
Na
7,8
Transpose Convolation
512
3x3
2x2
Yes
ReL.U
Transpose Convolation
256
3x3
2x2
Yes
ReL.U
4
Convolation
256
3x3
Ixl
Yos
ReLU
5
Transpose Convolation
128
5x5
2x2
Yes
ReL.U
6
Transpose Convolation
64
5x5
2x2
Yos
ReLU
Convolution
64
5x5
Ixl
Yes
ReL.U
8
Convolation
5x5
Ixl
No
Sigmoid
Table 7: FairDisCo architecture for the Extended Yale dataset.
15

---

Under review as a conference paper at ICLR 2023
Map
Layer
Operation
Filters
Kernel
Strides
Batch norn
Activation
Linked layer
fa
Convolution
64
4x4
2x2
Yes
ReL.U
X
Convolution
128
Yes
ReLU
Convolution
256
4x4
Yes
ReLU
4
Convolution
512
4x4
2x2
Yes
ReLU
5
Dense
No
Ja
Convolution
4x4
Yes
ReLU
X
Convolution
4x4
Yes
ReLU
3
Convolution
64
4x4
2x2
Yes
ReLUU
Convolution
4x4
Yes
ReLU
Dense
Dense
7x7x256
fa, fa
2
Transpose Convolution
128
4x4
2x2
Yes
LeakyRel.U
Transpose Convolution
4x4
Yes
LeakyReLU
Convolution
4x4
Yes
LeakyReLU
5
Convolution
4x4
1x1
No
Sigmoid
Discriminator
Dense
48
No
ReLU
fa
Dense
ReLU
Dense
ReLU
Dense
48
No
ReL.U
Dense
Table 8: WFAE architecture for the MNIST dataset.
Map
Layer
Operation
Filters
Kemel
Strides
Batch
Activation
Linked layer
Convolution
64
4x4
2x2
Yes
ReLU
X
2
Convolution
128
4x4
2x2
Yes
ReL.U
Convolution
256
4x4
2x2
Yes
RcLU
4
Convolution
512
4x4
2x2
Yes
RcLU
100
Yes
ReLU
S
Dense
6
No
Dense
No
Output (Z) S)
Sample Zi|X,
14,
Pxiz,s
7x7x256
No
2,
Transpose Convolution
128
4x4
2x2
Yes
ReLU
Transpose Convolution
4x4
2x2
Yes
ReLU
Convolation
4x4
1xl
Yes
ReLU
Convolution
4x4
1xl
No
Sigmoid
Table 9: HCV architecture for the MNIST dataset.
Map
Layer
Operation
Filters
Kemel
Strides
Batch
Activation
Linked layer
Qaxxs
Convolution
4x4
2x2
Yes
ReI.U
X
2
Convolution
128
4x-
2x2
Yes
ReLU
Convolution
256
2x2
Yes
ReLU
4
Convolation
512
4x4
2x2
Yes
ReLU
j
No
d²
Dense
No
Output (Z) S)
Sample S
o²
Pxies
Dense
7x7A256
No
S,5
Transpose Convolution
128
2x2
Transpose Convolution
64
4x4
2x2
Yes
ReLU
Convolution
4x4
1xl
Yes
ReLU
Convolution
4x4
1xl
No
Sigmoid
Table 10: FairDisCo architecture for the MNIST dataset.
16

---

Under review as a conference paper at ICLR 2023
Map (a)
Layer
Operation
Filters
Kemel
Strides
Batch
Activation
Linked layer
Convolution
128
5x5
2x2
Yes
ReLU
X
Convolution
256
5x.5
2x2
Yes
ReLU
Convolation
5x5
Yes
ReLU
Convolation
1024
5x5
2x2
Yes
ReLU
Dense
71
fa
Convolution
128
5x5
Yes
ReLU
X
Convolation
128
5x.5
1xl
Yes
ReLU
Convolution
256
5x55
2x2
Yes
ReI.U
Convolation
5x5
Yes
ReLU
Convolution
512
5x55
2x2
Yes
ReL.U
Convolation
3x3
Convolation
1024
3x3
2x2
Yes
ReLUU
Convolution
1024
3x3
1xl
Yes
ReLU
Dense
9
Dense
8x8x
No
fa.
Transpose Convolation
512
5x5
2x2
Yes
LeakyReL.U
Residual Block
LenkyRel.U
4
Transpose Convolution
256
5x5
2x2
Yes
LeakyReLU
Residual Block
256
5x5, 1x1
1xl
Yes
LenkyRel.U
Transpose Convolution
128
5x5
2x2
Yes
LeakyReLU
Residual Block
128
3x3. 1x1
1x1
Yes
LeakyReLU
Transpose Convolation
LeakyReLU
Residual Block
64
3x3, lxl
Ixl
Yes
LeakyReLU
10
Convolation
3x3
No
Sigmoid
Discriminator
Dense
No
ReLU
fa
Dense
256
No
ReLU
Dense
256
No
ReLU
Dense
256
No
ReLU
Dense
No
Table 11: WFAE architecture for the VGGFace2 dataset.
Map
Layer
Operation
Kemel
Strides
norm
layer
Encoder
Coavolution
128
5x5
2x2
Yes
ReLU
X
Convolution
128
5x5
1xl
Yes
ReLU
Convolution
256
5x5
Yes
ReLU
Convolation
256
5x5
Ixl
Yes
ReLU
Convolution
512
5x5
2x2
Yes
ReLU
Coavolution
512
3x3
Yes
ReLU
Convolation
1024
3x3
2x2
Yes
ReLU
Convolution
1024
3x3
Yes
ReLU
Dense
96
Decoder
Dense
8x8x1024
No
8,2
Transpose Convolation
512
5x5
Yes
LeakyReLU
Residual Block
512
5x5, Ixl
1xl
Yes
LeakyRel.U
Transpose Convolation
256
5x5
2x2
Yes
LeakyReLU
Residual
LeakyRel.U
Transpose Convolation
128
5x5
2x2
Yes
LeakyReLU
Residual Block
128
3x3, 1x1
1xl
Yes
LeakyReL.U
Transpose Convolution
5x5
2x2
Yes
LeakyReLU
Residual Block
64
3x.3, 1x]
1xl
Yes
LeakyReL.U
Convolution
No
Sigmoid
Discriminator
Dense
No
ReLU
Dense
384
No
ReLU
Dense
384
No
ReLU
Dense
384
No
ReL.U
Dense
7
No
Table 12: Fader Network architecture for the VGGFace2 dataset.
17


Reviews:
[{"review_id": "M3gltbDxsV", "reviewer": "Reviewer_4HcR", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new analysis of Wasserstein Autoencoders and claims the new analysis reveals a learning objective form that naturally can be optimized without the addition of ``ad-hoc\" penalties. The paper describes this anaylsis and presents a full derivation of their construction in three prototypical generative model examples with side or nuisance information. They evaluate their method in a few settings where conditional generative models are deployed.", "summary_of_the_review": "It is not clear what the paper is proposing and the main claim is not justified at all. The paper seems to have derived some formulations and then tacked on arbitrary regularizers, completely antithetical to the claims in the abstract and introduction.", "strength_and_weaknesses": "Strengths\n1) The paper is detailed in its derivation for the numerous examples it presents.\n\n2) The paper does a good job of setting up the problem and describing necessary background in a way that flows nicely with the story of the paper.\n\n3) The notation, analysis, and arguments are typically very clear, and a careful reader would be able to follow most of the discussion with mild background in Wasserstein method and VAEs.\n\nWeaknesses\n1) The paper does not provide a solution for the problem and claim suggested in the abstract or introduction. The authors claim that penalties and constraints can be derived directly from, or induced from, the conditional independence structure of the WAE. The introduction describes a number of prior literature that use a variety of metrics to enforce or push independence, and claim that none of these are a ``principled way ofo imposing the encoder structure\". The end of the preliminaries describes a funtion $\\delta$ that already has a new, arbitrary divergence $\\mathcal{D}$. The core of the paper, the three examples presented, all have additional regularization/constraint terms arbitrarily appended to their objectives, explicitly suggesting they can take the form of an MMD loss, GAN loss, HSIC loss, etc.\n\n2) There is no general algorithmic or analytic procedure presented that might generalize past the three problems described here. The experimental setups shown follow the examples shown, but it's not clear what the examples are providing over the existing methods when the arbitrary regularizers are not added.\n\n3) The experimental evaluation in general is not very convincing. The most important piece in experimental setups similar to this is choosing the regularization weights. This can drastically change the results of all models tested, and no discussion is provided that clarifies how or which weights were chosen. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "DAFzrL8MQ7", "reviewer": "Reviewer_ernN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors argue that there is a principled way to handle structural constraints on the latent variables in Wasserstein AutoEncoders. They argue that this is in contrast to the VAE where numerous tricks have been employed on the objective function in order to impose similar structural constraints, e.g., the semi-supervised work and beta-VAE.", "summary_of_the_review": "Interesting paper. Concerns can be alleviated through a rebuttal of the comparison to VAEs and by expanding on the empirical finding.", "strength_and_weaknesses": "Thank you for a very interesting read. The research is of high interest to the research community and shows a clear strength of using WAEs. To my knowledge and review, the theory is sound. The experiments are meaningful and provide good empirical evidence for the claims in the paper.\n\nThe weakness from the initial review regards the strong advocation that this is in contrast to VAEs. I agree with the fact that many research papers have employed tricks on the objective function that are far from principled in the VAEs.\n\nRegarding the comparison to VAEs:\n1) Have you, in fact, shown that WFAEs are preferred over the VAE framework? If so, a response in the rebuttal would be much appreciated.\n2) Is the comparison to the VAE papers fair? For example, some of the very deep VAE papers, e.g., \"Very Deep VAEs ...\" by Child et al. or \"BIVA ...\" by Maaloe et al. shows principled approaches to the VAE in cohesion to image generation examples that could suggest that they learn structural constraints on the latent variables. What would happen if you employ a structural constraint on these models only by altering the graphical model (no changes on the objective function) and how would that compare to the WFAE? \n\nRegarding the general empirical findings:\n- It would be useful to see many more examples of conditional generation examples, e.g., figure 5 in \"Autoencoding beyond pixels using a learned similarity metric\" by Larsen et al.\n- What about semi-supervised classification evaluations? Wouldn't it be possible to report on these and compare to state-of-the-art?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7PAziPZnCv", "reviewer": "Reviewer_Zcji", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper extends the Wasserstein autoencoder framework for the unconditional generative model to the setting of having structural constraints e.g., conditional independence on the latent variables (representation, or factors). In greater detail, the authors consider three examples of having additional information to the generator. The first example is the \"M2\" model (Kingma et al., 2014). In this setting, there exists an observed nuisance variable $S$, and the latent variable $Z$ wants to encode the representation invariant to the unwanted variation in $S$. The second example is the extension of the variational fair autoencoder (an extension of the “M1 + M2” semi-supervised model) that has partially observed label variable $Y$ in addition to the nuisance variable  $S$. The final example is the extension of the first example with two independent nuisance variables that can be missing. The main modeling techniques in the paper are factorizing the couplings between the joint observed data variables distribution and the joint model variables distribution by the chain rule and matching the aggregated posterior to the prior (which is used in WAE and is equivalent to Kantorovich optimal transport via the gluing lemma). The ground metric on the joint space is the square root of the square of distances on marginal variables. The authors compare the proposed framework with the HSICconstrained VFAE (HCV) and the FairDisCo in learning fair representation on Adult Income and Health datasets, learning invariant representation on the Extended Yale B dataset, and conditional generation on MNIST and VGGFace2 datasets.", "summary_of_the_review": "The paper should include a graphical model for the inference model (encoder) and compare the proposed framework with recent papers (mentioned in the weaknesses part).", "strength_and_weaknesses": "## Strength\n\n* The paper is the first work that extends WAE to the setting of the \"M1\" and \"M2\" models. \n* The experimental results indicate that the proposed framework is better than the previous approach such as variational fair AE and FairDisCo.\n\n## Weaknesses\n\n* The writing is hard to follow. The author can simply replace complicated notations by showing the graphical model of the inference/recognition/encoding model.\n* The chosen ground metric e.g., $\\tilde{d}$ in Example 1 is not a valid metric on the joint space of supports of random variables for generalized choices of  $d$ and $d'$.  Therefore. the final objective is not the \"Wasserstein\" distance. The specific settings of those distances are missing in experiments.\n* The duality gap could be large for the Lagrange duality. The tuning of various Lagrange multipliers is a negative point.  Moreover, the choices of the divergences e.g., $D$ and $H$ in Example 1 are not discussed.\n* The baselines of experiments are not strong. There are more recent works on the same problem e.g., [1],[2]. Also, the performance of the proposed framework is not clearly superior compared to the chosen baselines e.g., Figure 2, Table. 2.\n\n[1] Hierarchical VampPrior Variational Fair Auto-Encoder, Philip Botros and Jakub M. Tomczak\n[2] Flexibly Fair Representation Learning by Disentanglement, Elliot Creager et al.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject