ADIS-GAN: Affine Disentangled GAN
Authors: Letao Liu
Martin Saerbeck
Justin Dauwels
Venue:  (2021)
Keywords: Deep Learning
Disentangled Representation
Generative Adversarial Network
Computer Vision

Abstract: This paper proposes Affine Disentangled GAN (ADIS-GAN), which is a Generative Adversarial Network that can explicitly disentangle affine transformations in a self-supervised and rigorous manner.  The objective is inspired by InfoGAN, where an additional affine regularizer acts as the inductive bias.  The affine regularizer is rooted in the affine transformation properties of images, changing some properties of the underlying images, while leaving all other properties invariant. We derive the affine regularizer by decomposing the affine matrix into separate transformation matrices and inferring the transformation parameters by maximum-likelihood estimation.  Unlike the disentangled representations learned by existing approaches, the features learned by ADIS-GAN are axis-aligned and scalable, where transformations such as rotation, horizontal and vertical zoom, horizontal and vertical skew,  horizontal and vertical translation can be explicitly selected and learned. ADIS-GAN successfully disentangles these features on the MNIST, CelebA, and dSprites datasets.

Under review as a conference paper at ICLR 2021
ADIS-GAN: AFFINE DISENTANGLED GAN
Anonymous authors
Paper under double-blind review
ABSTRACT
This paper proposes Affine Disentangled GAN (ADIS-GAN), which a Gener- ative Adversarial Network that can explicitly disentangle affine transformations in a self-supervised and rigorous manner. The objective is inspired by InfoGAN, where an additional affine regularizer acts as the inductive bias. The affine regu- larizer is rooted in the affine transformation properties of images, changing some properties of the underlying images, while leaving all other properties invariant. We derive the affine regularizer by decomposing the affine matrix into separate transformation matrices and inferring the transformation parameters by maximum likelihood estimation. Unlike the disentangled representations learned by exist- ing approaches, the features learned by ADÍS-GAN are axis-aligned and scalable, where transformations such rotation, horizontal and vertical zoom, horizontal and vertical skew, horizontal and vertical translation can be explicitly selected and learned. ADIS-GAN successfully disentangles these features on the MNIST, CelebA, and dSprites datasets.
INTRODUCTION
In a disentangled representation, observations are interpreted in terms of few explanatory factors. Examples of such factors are the rotation angle(s), scale, or position of an object in an image. Disen- tangled variables are generally considered as the abstraction of interpretable semantic information and reflection of separatable factors of variation in the data. Many studies have explored the ef-
fectiveness of disentangled representations (Bengio et al. 2013 N al., 2017 LeCun et al.) 2015) Lake et al. 2017 Tschannen et al., 2018). The information presented in observations is encoded in an interpretable and compact manner, e.g., the texture style and the orientation of the objects
(Bengio et al., 2013 LeCun et al.) 2015 Lake et al. 2017 Tschannen al. 2018). The learned representation are more generalizable and can be useful for downstream tasks, such as classification
and visualization (Bengio al., 2013 N et al. 2017 Chen et al. 2016).
The concept of disentangled representation has been defined in several ways in the literature Lo- catello et al., 2019 Higgins al.) 22018] Eastwood & Williams 2018). The necessity of explicit inductive biases both for learning approaches and the datasets is discussed in Locatello et al.) (2019). Inductive bias refers the set of assumptions that the learner
predict outputs of given inputs that it has not encountered. For instance, in the dSprites dataset objects are displayed at different angles and positions; such prior knowledge helps to detect and classify the objects. However, the inductive biases in existing deep learning models are mostly implicit. The proposed ADIS-GAN utilizes relative affine transformations (see Section as the explicit inductive bias, leading to axis- aligned and scalable disentangled representations.
The issue of axis axis-alignment addressed in Higgins et al.) (2018), where each latent dimension should have pre-defined unique axis-alignment. Without axis-alignment, the features learned by disentangled representation need to identified with expert knowledge after the training, which could be cumbersome process when dealing with a large number of features. The axis-alignment property also helps to discover desired but non-dominant attributes (e.g., roll angle of face in CelebA dataset).
Axis-alignment:
The scalability property allows us to make a trade-off between the compactness and expressivity of the disentangled representation. For example, the zoom attribute can be decomposed as horizontal and vertical zoom. A more compact representation encodes the zoom attribute by one latent dimension, while more expressive representation decomposes the zoom attribute as
Scalability:

---

Under review as a conference paper at ICLR 2021
C1
3
3
3
3
3
3
3
3
3
3
6
6
6
6
6
6
6
C3
6
6
6
6
6
6
6
6
6
6
C4
5
5
5
5
5
5
5
5
5
5
C5
5
5
5
5
5
5
5
5
5
5
C6
0
0
O
O
O
O
O
O
0
O
O O
O O
O
Figure I: Disentangled representation of affine transformations on the MNIST dataset. c1 rotation, C2: horizontal zoom, C3: vertical zoom, c²: horizontal skew, cs: vertical skew, cg: horizontal trans- lation, e7: vertical translation. To the best of our knowledge, ADIS-GAN the first algorithm that can disentangle an entire affine transformation in a self-supervised manner.
horizontal and vertical zoom, encoded by two latent dimensions. The scalability property provides a solution to an open question related to disentangled representations: Should we capture a data generative factor (e.g., 200m) by one or multiple latent dimensions? Many disentanglement metrics
Higgins et al. y; Chen et al. p0 Kim & Mnih (2018); Eastwood & Williams (2018) rely on single latent dimension, while others Denton & Birodkar (2017); Ridgeway & Mozer (2018) Higgins et al. 2018) encode latent factors via multiple latent dimensions.
We motivate the importance of axis-alignment and scalability in particular for affine transformations (see Figure J where disentangling object poses from texture and shape is an attractive property of an algorithm in the imaging domain (Jaderberg et al. 2015 Bepler al.! 2019 Engstrom et al., 2019). In supervised learning tasks, Spatial Transformer Network Jaderberg et al. (2015) can ac- tively spatially transform an image by providing proper affine transformation matrix. In unsuper- vised learning tasks, few algorithms have successfully disentangled the affine transformation. In Bepler et al. (2019), an algorithm introduced that disentangles rotation and translation but not an entire affine transformation.
We propose ADIS-GAN, which a Generative Adversarial Network that utilizes the affine regu- larizer (see Section 4 as an inductive bias to explicitly disentangle the affine transformation. The affine regularizer rooted in the affine transformation properties of the images, that affect certain properties of the underlying images, while leaving all other properties invariant. We derive the affine regularizer by decomposing the affine matrix into separate transformations and inferring the trans- formation parameters by maximum likelihood estimation. Unlike the disentangled representations learned by existing approaches, the features learned by ADIS-GAN are axis-aligned and scalable, where transformations such rotation, horizontal and vertical zoom, horizontal and vertical skew, horizontal and vertical translation can be explicitly selected and learned (see Figure I..
In the remainder of the paper, review related work in Section 2 We then compare the difference
between GAN, InfoGAN and the proposed method in Section 3 We introduce the ADIS-GAN in
Section 4 while in Section [ show numerical results showing the axis-aligned and scalable disentangied representation learned by ADIS-GAN, We offer concluding remarks in Section
Our contributions:
1. To the best of our knowledge, ADIS-GAN is the first algorithm that can disentangle an entire affine transformation, including rotation, horizontal and vertical zoom, horizontal and vertical skew, horizontal and vertical translation in a self-supervised manner.
2. The disentangled representations obtained by ADIS-GAN are axis-aligned. The advantages are two-folds: a. The attributes be learned can be pre-defined, which saves the effort to identify the attributes after the training. b. Desired but non-dominant attributes can be learned, in parallel with the dominant attributes.
3. The disentangled representations obtained by ADIS-GAN are scalable. The scalability property makes it possible to make a trade-off between the compactness and expressivity of the learned representation.
2

---

Under review as a conference paper at ICLR 2021
2 RELATED LITERATURE
Recent approaches learning disentangled representations are largely based on Variational Autoen- coders (VAEs) Kingma Welling (2013) and InfoGAN Chen et al. (2016). In a standard VAE, observation z is encoded to latent representation 2 from variational distribution (2)(z) using a deep neural network. In the generative process, the latent variable sampled from a prior distribution P(z) and uses another deep neural network to reconstruct the observation from conditional distri- bution P(z|=). To achieve the disentanglement, a factorized aggregated posterior (=)x=))) P(x)dd is encouraged during the training. In standard GAN Goodfflllow et al. (2014). latent representation 2 is sampled from a prior distribution P(z). The fake data Tjake is generated by 2 from distribution P(a|z). A discriminator is introduced to encourage the generated data Tlake to be close to the ob- servation Treat- To achieve disentanglement, InfoGAN Chen et al.] (2016) proposes to maximize the mutual information between subset of latent representation and the generated data Tfake
Recently, much attention has been paid to regularizers that promote disentanglement. The 8- 8-VAE Higgins et al. (2017) encourages the disentanglement by increasing the weight of the KI. regular-
izer, thus promoting the factorization of the posterior (z][z). Both FactorVAE Kim & Mnih (2018)
and (-TCVAAE/Chn et al. (2018) penalize the total correlation, while the former relies on versar-- ial training and the latter directly calculates the total correlation through the decomposition of the
3-VAE objective function. The HFVAE Esmaeili al. (2019) proposes a two-level hierarchical ob- jective to control relative degree of statistical independence. In the Chy Ansari & Soh (2019), an inverse-Wishart (IW) prior on the covariance matrix of the latent code is augmented to promote statistical independence. The DIP-VAE Kumar et al. (2018) penalizes the difference between the aggregated posterior and a factorized prior. In the Annealed VAE Huang et al.) (2018), the encoder can concentrate on learning individual factors and variation by gradually increasing the bottleneck capacity. The IB-GAN Jeon et al.] (2019) is an extension to InfoGAN rooted in information bot- tleneck theory, which includes mutual information upper bound and forms mutual information bottleneck. The InfoGAN-CR [in et al. (219) adds a contrastive regularizer on top on InfoGAN, that compares the changes between the image and latent space.
Even though the aforementioned methods have achieved better disentanglement performance com- pared the baseline established by VAE and InfoGAN, none of them have successfully disentangled an entire affine transformations in a scalable and axis-aligned way, which is a desirable property in the imaging domain. Moreover, our affine regularizer is orthogonal those approaches. thus makes it possible to integrate our affine regularizer with other methods.
In Gidaris et al. (2018); Chen et al. (2019); Wang et al. (2020); Zhang et al. (2019). self- supervised regularization is applied, where they compare the difference of images before and after the affine/projective transformation. The transformation loss is define as: L= |IM - (0) and parameterized matrix M E IR3333, We make more specific assumption about the matrix and further decompose it to achieve disentanglement (see Section 4).
3 BACKGROUND: GAN, INFOGAN AND ADIS-
In a standard GAN Goodfellow et al. (2014), latent representation is sampled from a prior dis- tribution P(z). The fake data Piake is generated from z from conditional distribution P(a(2). A discriminator is introduced to encourage the generated data Tiake to be close to the observation Preat:
min max V(D, G) = Ladv (D.
(1)
To achieve disentanglement, InfoGAN CheaannCCheeeeeeee Chen et al. (216)) maximizes the mutual information between a subset c of latent representation z and the generated data Diake:
min max V(D, = Iudy (D, G) - AT(': G D
(2)
To explicitly disentangle affine transformation attributes in a scalable and axis-aligned manner, ADIS-GAN has one additional term called affine regularizer on top of InfoGAN:
min max V G) L (D, XI xisee) BL
(3)
G D
3

---

Under review as a conference paper at ICLR 2021
4
AFFINE REGULARIZER
Compared to InfoGAN, the major difference of ADIS-GAN is the affine regularization loss Laffine (see 2. To calculate Lutliee three modifications to the network have been made. 1: Besides generating Tiake and calculating mutual information loss I(c'; Ttake) like InfoGAN, the semantic latent vector e of ADIS-GAN has an additional task: to generate random affine transformation pa- rameters. We now rename the latent vector с as "tranform to describe the state before and after the affine transformation, where superseript "real" stands for the initial state before the transformation and subscript "transform" stands for the final state after the transformation, more details regarding the transformation states will be introduced in Section 4.2 rraa
deal
"transfomm is sampled from a uniform distri- transform using Flow (see Figure 4). 2: Affine transfor- mation augmented image Firanfform is introduced. Firansform is obtained by multiplying M transform and Freal sampled from training data (see Figure B.. Unlike InfoGAN where "Ireal" is the positive sample fed to the discriminator. In ADIS-GAN, Twansform is the positive sample fed to the discriminator. This guarantees the affine transformations are observed by the network. 3: Affine reugularization loss Lamime is calculated by comparing the ground truth affine transformation latent vector c Granstorm and the reconstructed affine transformation latent vector deal"
bution, then converted to an affine matrix M Mrani
"Irnssormm which is obtained by calculating the relative difference between Freal and its affine augmented pair Trranffor (see Figure
G
Take
D
Lav
real transform
1 (Cintoi Ifake)
E
Ftransform
Lattine
Freal
The main framework of ADIS-GAN. stands for discriminator, E stands for encoder, G stands for generator. Tisk is the generated image, Treal is the image sampled from training dataset, Transfor,, the affine transformed image. The generation process of Frransfoem is demonstrated in Figure 2 is latent noise sampled from normal distribution. Trnnstom is the semantic latent vectors sampled from uniform distribution. cleal Gransfor has two usages: 1. It is used to generate Traake and compute mutual information loss. 2. used to generate V'ransform together with Freal and compute affine regularization loss (see Figure 33.. The meaning of the superscript "real" refers to the initial state before the transformation and subscript "transform" refers to the final state after the transfor- mation. i Ttake) is the mutual informaion loss. Laav (D, G)) is the GAN loss. Lattine is the affine regularization loss. Figure 3 Billustrates how to calculate the affine regularization loss Latline-
Figure 2:
The working principle of affine regularizer is illustrated in Figure 3 First, the latent vector Gramsform is sampled from a uniform distribution. It is converted to an affine matrix Mrrsal transfiom through "Flow 1" (see Section 4.1.T) Next, the real image "Xreal sampled from training data is affine transformed with M to obtain Trransform- Both Treal and its transformed pair Transform encoded to latent Gransform respectively. The encoded latent vectors bassss and Gransform are further con- verted to affine matrix Mbasis and Mrbasi trunstiorm through "Flow I" (see Section 4.1.1 Then the relative tramsform is computed using both basis and M basis greal' tion 9). Finally, the reconstructed latent vector Gransform is obtained using M through "Flow 2" (see Section 1.2).. Here use rotation (0), horizontal and vertical zoom (p, q), horizontal and vertical translation (z, y) as an example (more affine parameter combinations can be found in Ap- pendix H). The affine regularizer loss Laftine = Σ: ( transform -c(i -c(i)reall y I12, i E {0, q, x. y-
vectors Greal and /bsss
transformation matrix Mreal
transform (see Section 4.2 Equa-
You may refer to Algorithm] T and Figure for more details.
4

---

Under review as a conference paper at ICLR 2021
transform
Treal
F
hasis real
Flow
M
Flow
Mreal
Affine transform
Relative transform
real
Flow 2
transform
transform
transform
E
Transform
Flow
Mbusis
transform
transform
Transform sampled from Unif(-1,1) and Treal sam- pled from training data. Output: Turnsform and rral "ransform" The affine regularizer loss is: Lamine = <( transform 122 i E {0, I, v. 0: rotation, y: horizontal zoom, g: vertical z00m, x: horizontal translation, y: vertical translation. stands for encoder, Flow and Flow are defined in Section 4.1 and Figure Relative transform is defined in Section 4.2 Equation g
Figure Affine regularizer diagram. Inputs:
real
Σ; ||[()) )
transform
Algorithm 1: Affine Regularizer
Input: Sampled images: Treals latent vectors: c(i
real transform
Unif(-1,1), i (0.p.
Output: Latiine
Matrix from sampled latent vector: M Meal
gansform Flow (c(
transtorm ), i E (0.p.9.2,8)
Affine transformation: Transffrm M
trassoom
Zreal
Latent vectors encoded from images: Gransform Encoder @transform) ), Geaa Encoder(Fxea)
Matrices from encoded latent vector: M
= Flow I [c( busis (ransfoorm.. M M
Flow (c(i) (e)s) E Q, Relative transform matrix: Mreal'
transform M hasss
transform x Maasss
Reconstructed latent vectors: c(i) (())tall transform
Flow 2 Prefr
transform
Affine reuglarizer loss: Laftine
real E; ||((1)
c(i
real
2. i {0, (0,p44 9,,,,
transform
transform
Flow 1:
NormLa
M Initialization
M
Flow 2:
Ao
MLE
M
Figure 4: Conversion between latent vectors and affine matrix. Flow 1: convert the latent vectors to affine matrix M. Flow convert the affine matrix Mto latent vectors c. stands for semantic latent vectors. č stands for affine parameters such as rotation (0), zoom (p, and translation (x, y). M stands for affine matrix. "NormLa" and "M Initialization" are described in Section 4.1.1 "NormaL" and "MLE" are described in Section 4.2.2
4.1 CONVERSION BETWEEN LATENT VECTORS AND AFFINE MATRIX
4.1.1 LATENT VECTORS TO AFFINE MATRIX: FLOW 1
To conduct affine transformation and calculate the relative difference, we need to convert latent vector to affine matrix M. The "Flow 1" in Figure illustrates the process.
NormLA (Latent to Affine): First, we need to normalize the latent vector c to the affine trans- formation parameter ē. For instance, if we set the affine transformation range as rotation 0 E -101 N x/10], horizontal and vertical zoom p. E [0.8, 1.2], horizontal and vertical translation z,y E [-0.2, 0.2), and 5-d latent vector
sampled from uniform distribution Unif(-1. 1). Norm, (Latent to Affine) refers to the following normalization: z(0) = c(0) x (T/10), c(p) =
<(p) x 0.2 1, ē(g) e(g) x 0.2 ē(x) c(x) 0.2, ē(y) c(y) 0.2.

---

Under review as a conference paper at ICLR 2021
M Initialization: Next, the affine parameter è needs to be properly arranged in the affine matrix M. By default, the affine matrix M is organized as in Equation
A11
A12
A13
cos
sin 9 cos 0
0
07
[ 0 x
M
A21
A22
Ag3
sin
0
0
1
y
(4)
0
0
0
0
0
0
1
4.1.2 AFFINE MATRIX TO LATENT VECTORS: FLOW 2
To obtain the reconstructed affine transformation latent vector "C "rransoomm we need to convert the affine matrix to latent vector. The "Flow 2" in Figure iillustrates the process.
MLE (Maximum Likelihood Estimation): First, we need to calculate the affine parameter ? using maximum likelihood estimation (see Appendix A).
0 arelan 2(An An - A122
1i1 A22 A/1 Ag
p A1 cos 0 A21 sin A
ā(i)
q
-A sin 0 A21 sin 0,
(5)
x A1acos A4 sine
y A1g sin A2 co0
where E Q,
NormAL (Affine Latent): Next, need normalize the affine parameter latent vector c. If we apply the same affine transformation setting mentioned in Section Flow 1. NormAL (Affine Latent) refers the following normalization: c(0) 2(0) (10/x). (ő(p) x5,
c(g) (Ō(g) - 1) x 5, c(x) = ē(x) 5, c(y) ē(y) x 5.
4.2 RELATIVE AFFINE TRANSFORMATION
M transform
M basis real
M
transform
Pbasis
Freal
Firansform
Figure 5: Illustration of relative affine transformation. Solid line stands for the affine transformation between two real images. Dashed line stands for the affine transformation between one real image and the imaginary image basis.
We assume, without loss of generality, that images can be expressed as the multiplication of the affine transformation M busis 33x3 (affine matrix is a matrix by default) and an image basis Tbasis (see Figure 5):
M M/bais
x Tbasis Tyeal-
The superscript "basis" Mhasis stands for the initial state and subscript "real" stands for the trans- formed state of the image The image basis Trusis is the average manifold of all images within the same category. For instance, the digits "0" "1", "9" in MNIST are different categories. Pracis does not refer to a particular image the dataset and is purely learned from the data.

---

Under review as a conference paper at ICLR 2021
If purposely apply known affine transformation obtain an new image Ftranstorm
M
transform
sampled image Treals
M
real transform
X Treal
Ttransform
According to the definition in equation 6 the transformed image can also be expressed the multi- plication of the M hassis transform and the image base Thasis
M
Mbassi basis frrmsfomm
X Ubnsis
Trransform
(8)
By solving the simultaneous formula from equation to
real transform
M transtiom
M basis
(9)
real
M
transoomm is the relative affine transformation between affine matrices transform
Mbasis
and M real
5 NUMERICAL RESULTS
The goal of the experiments in this section is to investigate, both qualitatively and quantitatively, the disentangled representations obtained by ADIS-GAN. The axis-alignment of the disentangled representations is demonstrated CelebA dataset Liu al. (2015), while the scalability of the disentangled representation is illustrated on MNIST dataset. Next quantitative results for ADIS- GAN are presented and compared benchmarks on the dSprites /Sprites/Matthey al. (2017) dataset. The parameters of the affine transformation are selected as follows: rotation range: 1818 18% zoom range: [0.8, 1.2], and translation range: [-0.2, 0.2]. For all the experiments we use Adam optimizer with the learning rate of 0.0002 for discriminator and 0,0001 for generator and encoder. The batch size 128 for MNIST, for dSprites, and 16 for CelebA. The regularization weight in Equation is 1 for both À and B. You may refer to Appendix G for network structure details.
5.1 QUALITATIVE RESULTS
The axis-alignment and scalability properties achieved by selecting different matrix initializa- tions and their corresponding maximum likelihood estimation. You may refer to Appendix for various ways of matrix initializations and their maximum likelihood estimation.
5.1.1 AXIS-ALIGNMENT
In CelebA dataset, typical dominant attributes are azimuth, sunglasses, emotion, etc. Existing meth- ods Chen et al. (2016) Kim & Mnih (2018); (0he)) Chen (2018) have successfully disentangled those attributes (see Appendix E). However, non-dominant attributes such as the roll, width and length of the face and relative position of face in the frame are rarely tackled. Thanks to the exis-aligned property, ADIS-GAN can explicitly learn those non-dominant attributes (see Figure 6 and 8). interesting note that ADIS-GAN -ANN does not rigidly perform affine transformations images but understands the relationship between different facial components. For example, when try to shorten the length of the face (vertical zoom and translation, see left images on both Fig- ure 7 and (8) ADIS-GAN knows how to complete the bottom of the images with a neck to make the images look realistic. To disentangle facial attributes the CelebA data, choose model the latent space with one categorical code, Cea Cat(K = 10, = 0.1) and 5 continuous codes Goont 1n..(-
the given
5.1.2 SCALABILITY
Scalability refers to the flexibility to encode one attribute via one latent vector or, alternatively, to decompose one attribute into sub-attributes and encode them by multiple latent vectors. For example, both zoom and translation attributes can be encoded by one latent vector, or they can be decomposed horizontally and vertically encoded by latent vectors (see Figure

---

Under review as a conference paper at ICLR 2021
Figure 6: Roll attribute on CelebA dataset.
Figure 7: Vertical (left) and horizontal (right) zoom attributes on CelebA dataset.
Figure Vertical (left) and horizontal (right) translation attributes on CelebA dataset.
6
6
6
Horizontal
6
6
6
Vertical
6
6
Merged
Zoom
Translation
Illustration of scalability. case 1 (row 1 and 2): use two latent vectors to represent horizon- tal and vertical direction separately. case (row 3): one latent vector to represent the horizontal and vertical as merged representation. You may refer to Appendix I for the derivation.
Figure 9:
5.2 QUANTITATIVE RESULTS
To disentangle object style on dSprites, we choose to model the latent space by one categorical code Geat ~ Cat(K = 3. p = 0.33), and 4 continuous codes Goont ~ Unif(-1,11) that represent rotation, zoom, horizontal and vertical translation. To the best of our knowledge, ADIS-GAN is the first algorithm that can disentangle shape attributes with categorical codes on dSprites datasets. We have observed that shape disentanglement is easier for larger images. We conducted experiments on the
8

---

Under review as a conference paper at ICLR 2021
Table I: Comparison of the axis- all-ageggmeettttt and scalability of the disentangled representation learned by different approaches. ADIS-GAT GAN is the only algorithm that generates disentangled representa- tions with both useful properties.
Factor -VAE
Annealed -VAE
Info -GAN
Info
ADIS
Property
3-VAE
3-TCVAE
-GAN-CR
-GAN
Axis
alignment
n0
no
no
no
no
no
yes
Scalability
no no
no
no no n0 yes
"Four Shapes" dataset (see Appendix D). Compared to dSprites, the major difference of the "Four Shapes" dataset is that the images are larger. We found that ADIS-GAN can successfully disentan- gle shape attribute on "Four Shapes" dataset with categorical codes (see Appendix D). The image enlargement procedures and the disentanglement results of dSprites are described in Appendix C.
Table 2: Disentanglement scores on the dSprites dataset. The reference values are from Lin et al. (201)) Table 1. Note that we do not apply model centrality here, which is hyperparameter tuning method, as our focus is on axis-alignment and scalability. InfoGAN (modified) uses spectrum nor- malization Miyato et (2018) for the discriminator of InfoGAN. For the FactorVAE and Beta BetaVAE scores, we only use continuous codes. For the other scores, we use both continuous and categorical codes. Overall, ADIS- is comparable with the state-of-the-a disentanglement algorithms on the dSprites dataset. We refer to Appendix F for the definations of disentanglement metrics.
Model
FactorVAE
DCI
Explicitness Modularity
MIG
BetaVAE
VAE
0.63 1 .06
0.30 + .10
0.10
3-TCVAE
0.62
0.29 1.10
0.45
HEVAE
0.63 + .08
0.39 +.16
3-VAE
0.63 + .10
0.41 = .11
0.21
VAE
0.77
VAE
DIP-VAE
0.53
Factor VAE
0.82
0.15
VactorVAF (1.0)
0.79 .01
0.67 + .03
0.78 + .01
0.79 + 001
0.27 + .03
0.79 + .02
Factor VAE (10.0)
0.83 :0
0.70 + .02
0.79
+ 00
0.79
.0
0.40 = .01
0.83 00
FAc (20.0)
0.83 183144 .01
0.72 1 .02
0.79
+ .0
0.79
1
0.40
= .01
0.85
+ .00
FactorVAE (40.0)
0.82
0.74 .01
+ .00
0.77
1 .01
0.43
.01
0.84 1
InfoGAN
0.59 1991..0 .70
0.41 1 .05
0.05
IB-GAN
0.80
0.67 .07
GAN
InfoGAN(modified))
0.82 1 .01
0.60 1 .22
0.82 1 .0
0.94 1 .01
0.22 1 001
0.87 1 .01
InfoGAN-CE
0.88 + .01
0.71 = .01
0.85 + .00
0.96 + .00
0.37 = .01
0.95 + .01
0.86 - 02
0.71 .22
0.78 = 00
0.95 01
0.46 = .01
0.89 - .01
CONCLUSION
This paper introduces the Affine Disentangled GAN (ADIS-GAN GANN) that can explicitly learn the affine transformations via disentangled representations. In contrast to earlier approaches disentangle- ment, where inductive biases are not explicit, the disentangled representation obtained by ADIS- GAN axis-aligned and scalable. The affine regularizer introduced in this paper can be applied
to other methods such the VAE family Kingma & Welling (2013): Higgins et al. (0017) Kim & Mnih (2018); Chen et al. (2018).
Another attractive property of the affine regularizer is that it supports the possibility utilize expert knowledge as the inductive bias, as it is model based and exploits maximum likelihood estimation. disentangle the 2D affine transformations with the decomposed relationships. Task-specific explicit regularizers may provide an alternative pathway for disentanglement com- pared existing general implicit regularizers.
For example,

---

Under review as a conference paper at ICLR 2021
REFERENCES
Abdul Fatir Ansari and Harold Soh. Hyperprior induced unsupervised disentanglement of latent representations. AAAI Conference on Artificial Intelligence, 2019.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell., 35(8):1798-1828, August 2013. ISSN 0162-8828. doi: 20.1009/PAAM..013...0.
Tristan Bepler, Ellen Zhong, Kotaro Kelley, Edward Brignole, and Bonnie Berger. Explicitly disentangling image content from translation and rotation with spatial-vae. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 15435-15445. Curran Associates, Inc., 2019.
Ricky T. Q. Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of dis- entanglement in variational autoencoders. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 2610-2620. Curran Associates, Inc., 2018.
Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby. Self-supervised gans via auxiliary rotation loss, In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 12154-12163. Computer Vision Foundation / IEEE, 2019. doi: 10. (000///VR.229..001111
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 2172-2180. Curran Associates, Inc., 2016.
Emily L Denton and vighnesh Birodkar. Unsupervised learning of disentangled representations from video. In I. Guyon, U. Luxburg, S. Bengio. H. Wallach, R. Fergus. S. Vishwanathan, and R. Gamett (eds.), Advances in Neural Information Processing Systems 30, pp. 4414 4423. Curran Associates, Inc., 2017.
Cian Eastwood and Christopher K. I. Williams. framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018.
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Explor- ing the landscape of spatial robustness. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 Proceed- ings of Machine Learning Research, pp. 1802-1811 Long Beach, California, USA, 09-15 Jun 2019. PMLR.
Babak Esmacili, Hao Wu, Sarthak Jain, Alican Bozkurt, N Siddharth, Brooks Paige, Dana H. Brooks, Jennifer Dy, and Lun-Willem van de Meent. Structured disentangled representations. volume 89 of Proceedings of Machine Learning Research, pp. 2534.2534. PMLR, 16-18 Apr 2019.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - -May 3, 2018, Conference Track Proceedings. OpenRe- view.net, 2018
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Pro- cessing Systems 27, pp. 2672-2680. Curran Associates, Inc., 2014.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In ICLR. OpenReview.net, 2017.
10

---

Under review as a conference paper at ICLR 2021
Irina Higgins, David Amos, David Pfau, Sébastien Racanière, Loic Matthey, Danilo J. Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. CoRR, tbs//882.00220,, 2018.
Chin-We Huang, Shawn Tan, Alexandre Lacoste, and Aaron C Courville. Improving explorabil- ity in variational inference with annealed variational objectives, In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Infor- mation P Processing Systems 31, pp. 9701-9711. Curran Associates, Inc., 2018.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and koray kavukcuoglu. Spatial transformer networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett )) Advances in Neural Information Processing Systemy 28, pp. 2017-2025. Curran Associates, Inc., 2015.
Insu Jeon, Wonkwang Lee, and Gunhee Kim. IB-GAN: Disentangled representation learning with information bottleneck GAN, 2019.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Pro- ceedings of Machine Learning Research, pp. 2649-2658, Stockholmsmässan, Stockholm Swe- den, 10-15 Jul 2018. PMLR.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2013.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. VARIATIONAL INFERENCE OF DISENTANGLED LATENT CONCEPTS FROM UNLABELED OBSERVATIONS. In Interna- tional Conference on Learning Representations, 2018.
Brenden M. Lake, Tomer Ullman, Joshua B. Tenenbaum, and Samuel J. Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 40:e253, 2017. doi: 1410017/0040525516000133
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436 2015. doi: 10. 1..08//aauure15539
Zinan Lin, Kiran Koshy Thekumparampil, Giulia C. Fanti, and Sewoong Oh. Infogan-er: Disen- tangling generative adversarial networks with contrastive regularizers. CoRR, abs/1906.06034, 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Francesco Locatello, Stefan Bauer, Mario Lučié, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Frederic Bachem. Challenging common assumptions in the unsuper- vised learning of disentangled representations. In International Conference on Muchine Learning, 2019. Best Paper Award.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset.
2017.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. International Conference on Learning Representations, 2018.
Siddharth N, Brooks Paige, Jan- Willem van de Meent, Alban Desmaison, Noah Goodman, Pushmeet Kohli, Frank Wood, and Philip Torr. Learning disentangled representations with semi-supervised deep generative models. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp- 5925-5935. Curran Associates, Inc., 2017.
Karl Ridgeway and Michael C Mozer.
Learning deep disentangled embeddings with the f-statistic loss. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Cesa-Bianchi, and R. Gamett (eds.), Advances in Neural Information Processing Systems 31, pp. 185-194, Curran Associates, Inc., 2018
11

---

Under review as a conference paper at ICLR 2021
Michael Tschannen, Olivier Bachem, and Mario Lucic. Recent advances in autoencoder-basee based representation learning. CoRR, abs/1812.05006 2018.
Jiayu Wang, Wengang Zhou, Guo-Jun Qi, Zhonggian Fu, Qi Tian, and Houqiang Li. Transforma-
tion gan for unsupervised image synthesis and representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Liheng Zhang, Guo-Jun Qi, Liqiang Wang, and Jiebo Luo. AET vs. AED: unsupervised repre- sentation learning by auto-encoding transformations rather than data. IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation IEEE, 2019, doi: 10. (09/CVP..201..00066.
12


Reviews:
[{"review_id": "V-MeUVO9kr", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This manuscript presents ADIS-GAN (affine disentangled GAN), a method for learning disentangled affine transformations of images such as rotation, zoom, and translation. The authors enforce this disentanglement using a regularizer that penalizes the difference between affine transformations on latent representations. The method is difficult to follow with many critical details left out. Without a complete description, it’s impossible to fully understand and evaluate this work. Some specific comments and questions follow below.\n1.\tHow is the latent vector, C, used? How does it interact with W_basis and W_real? Must it be 5-d to match with rotation, zoom, and translation? How are additional factors of variability encoded?\n2.\tHow is W_basis learned?\n3.\tWhat is Norm_AL\n4.\tWhat are the encoder/generator/adversary model architectures and how were they trained? For that matter, what is the encoder listed in algorithm 1? I can’t find any reference to it in the text.\n5.\tHow do you ensure that affine transformations of the latent vector actual produce affine transformations of the image? In Figure 5, it looks like the faces aren’t simply being transformed by zoom, because other attributes clear change. In column 5 from the right, the top face has short hair, an exposed ear, and visible teeth. The bottom face, in contrast, has long hair, covered ear, and no visible teeth. This does not appear to be a simple zoom transformation.\n\nThings that would improve my score:\n1.\tFully explain the method, neural network architectures, and training parameters.\n2.\tExplain how affine transformations of the latent vector are forced to produce matching affine transformations of the image.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ql1nkM67HfK", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method that can explicitly learn the affine transformations via disentangled representations in a self-supervised manner. The proposed ADIS-GAN learns to extract affine parameters by adding an affine regularizer on the top of InfoGAN. It seems to me that the main contribution of the paper is to learn the explicit parameterization of the affine transformation, such as rotation, zoom, and translation, by maximum likelihood estimation.  Experiments show that the proposed method is successful in estimating individual parameters of the affine transformation. \n\n**Strengths**\n+ The paper represents the first method that can learn explicit parameters of the affine transformation in a self-supervised manner.\n+ The experiments show that the proposed method can learn a disentangled representation with explicit affine parameters. \n\n**Weakness** \n- Experiments are not convincing enough. The paper only contains experiments on simple/synthetic datasets. The benefits of the proposed method are not completely clear. For the dSprites dataset, the proposed method does not outperform InfoGAN-CR, while InfoGAN-CR can perform disentanglement beyond affine transformations. For scalability, the paper only presents one example for zoom. It is also unclear how to merge zoom parameters into one. It is also helpful to give real applications that can benefit from the explicit disentanglement. \n- Writing of the paper can be improved. For example, Section 4.1 needs improvements. The notations are not completely clear, and their roles in the whole algorithm are not clear. It would be better to strengthen the connection of Equation (7) to the affine regularizer loss defined at the end of Section 4. \n- The factorization of the affine matrix and derivation of the maximum likelihood estimator are straightforward, making the paper's technical contribution less significant. \n\n**Minor issues**\n- The paper emphasizes scalability, but it is not completely clear what scalability the proposed method achieves and how it is achieved. For example, is it possible to merge the translation parameters into a single one? If so, how to do it, and how well the combined parameter performs?\n- It is unclear whether the proposed method can factor out skews. Figure 1 shows examples for skews, but the paper does not present how the skew parameters are estimated. \n- In Appendix A, the paper assumes the noise statistics for all parameters A_{ij}. It seems unrealistic. \n- The citations are correctly formatted. The parentheses are missing for citations.\n\n**Post-rebuttal**\n\nThank the authors for the rebuttal. It addresses parts of the raised issues. However, my rating keeps the same after reading the rebuttal and other reviews because\n1. the contribution and utility of the proposed method are not significant;\n2. the writing needs improvement; and\n3. the experiments are not convincing enough, and its advantages over previous methods are not clear. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "MBlHWV98toi", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper summary:\n\nThis paper seems to be about aligning latent variables in a GAN with certain affine transformations of the image. This is done by adding an additional \"affine regularization\" on top of the InfoGAN formulation. This affine regularization seems to be the L2 distance between the randomly sampled c (interpretable latent variables?) and some transformed c'. How the transformed c' is computed was not understood by the reviewer, but it seems to depend on a set of explicitly chosen affine transformations. The affine transformations specified are rotation, horizontal and vertical zoom, horizontal and vertical translation. The authors experiment on MNIST, dSprites, four shapes, and CelebA data. Quantitative metrics are reported for dSprites.\n\nStrong points:\n\nNot many papers tackling disentanglement in GANs.\n\nThe paper seems to show nice visual results and seems to be working at recovering the selected affine transformations of images.\n\nWeak points:\n\n(My overall concerns are about the additional supervision required by the method and the clarity of the writing, in particular the method itself.)\n\nThe paper mentions this is a self-supervised approach, but it seems closer to a supervised approach where the affine loss can only help recover specific latent factors that must be known beforehand. Would ADIS-GAN be capable of discovering unknown latent factors?\n\nThe paper repeatedly claims in the text and in Table 1 that previous works do not have axis-alignment or scalability. Does ADIS-GAN guarantee alignment in some way that is more reliable than in previous works? Isn't the scalability aspect (modeling horizontal and vertical zoom) explicitly encoded by ADIS-GAN, whereas previous works attempt to learn this from data?\n\nI could not follow the exposition in the paper. For instance:\n  - There seems to be a complete change in notation from Sections 1-3 to Section 4. No mention of the previously defined x_real, x_fake, z, or c are used in Section 4, where the core contribution is described. \n  - Many sentences end in \"see Appendix\", but I have looked at the Appendix and still come away confused..\n  - Why is M 3x3?\n  - Where do the A_{ij}'s come from in equation 9? I guessed it might be elements of c, but there are 5 elements in c and 6 A_{ij}'s.\n  - There seems to be derivations of a maximum likelihood estimate for constructing c', but these depend on knowing what the A_{ij}'s are.\n  - The purpose of Norm_{AL/LA} is never explicitly mentioned, though I imagine it's for scaling the interval [-1, 1] to specific bounds so that M can be applied?\n  - What is Encoder in Algorithm 1 and why does it return a linear operator? (Is this different from the encoder in InfoGAN which outputs a distribution over the latent variables?)\n\nAdditional feedback:\n\n - There is mention that ADIS-GAN used categorical codes as well. I assume these do not contribute to the affine loss?\n\n - There are multiple rows in Table 2 that reference models not explained in the text. e.g. references for HFVAE and ChyVAE, and what is \"InfoGAN(modified)\"?\n\n - Please use \\citep to include brackets around your citations, when not explicitly using them in a sentence.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Most of the reviewers had serious problems with clarity to start out. 
The authors have addressed some, but not all of these problems. 

More importantly, there were issues of significance and experimental evaluation.
I concur with r4 on the experimental evaluation. 
I think if you're going to explicitly specialize toward disentangling affine transform parameters, 
that's fine, but then you're in application-paper land, and I think there needs to be more of an attempt to show
that it will work "in the wild". 
For this reason, and for the general reason that reviewers unanimously voted to reject, I am recommending rejection.
Decision: Reject