Group Equivariant Generative Adversarial Networks
Authors: Neel Dey
Antong Chen
Soheil Ghafurian
Venue: ICLR 2021 Poster (2021)
Keywords: Group Equivariance
Geometric Deep Learning
Generative Adversarial Networks

Abstract: Recent improvements in generative adversarial visual synthesis incorporate real and fake image transformation in a self-supervised setting, leading to increased stability and perceptual fidelity. However, these approaches typically involve image augmentations via additional regularizers in the GAN objective and thus spend valuable network capacity towards approximating transformation equivariance instead of their desired task. In this work, we explicitly incorporate inductive symmetry priors into the network architectures via group-equivariant convolutional networks. Group-convolutions have higher expressive power with fewer samples and lead to better gradient feedback between generator and discriminator. We show that group-equivariance integrates seamlessly with recent techniques for GAN training across regularizers, architectures, and loss functions. We demonstrate the utility of our methods for conditional synthesis by improving generation in the limited data regime across symmetric imaging datasets and even find benefits for natural images with preferred orientation.

Published as a conference paper at ICLR 2021
GROUP EQUIVARIANT GENERATIVE ADVERSARIAL NETWORKS
Neel Dey
Antong Chen & Soheil Ghafurian!
New York University neel deel.dey@nyu.ede
Data Science & Scientific Informatics, Merck & Co., Inc. a antong. www.@merck......m.m com, sche anw.mma com
ABSTRACT
Recent improvements in generative adversarial visual synthesis incorporate real and fake image transformation in self-supervised setting, leading to increased stability and perceptual fidelity. However, these approaches typically involve im- age augmentations via additional regularizers in the GAN objective and thus spend valuable network capacity towards approximating transformation equivariance in- stead of their desired task. In this work, we explicitly incorporate inductive sym- metry priors into the network architectures via group-equivarintt convolutional networks. Eroup-convolutions have higher expressive power with fewer sam- ples and lead to better gradient feedback between generator and discriminator. We show that group-equivariance integrates seamlessly with recent techniques for GAN training across regularizers, architectures, and loss functions. We demon- strate the utility of our methods for conditional synthesis by improving generation in the limited data regime across symmetric imaging datasets and even find bene- fits for natural images with preferred orientation.
INTRODUCTION
Generative visual modeling is an area of active research, time and again finding diverse and creative applications. prevailing approach is the generative adversarial network (GAN), wherein density estimation is implicitly approximated by min-max game between two neural networks (Good-) fellow et af.) 2014). Recent GANS are capable of high-quality natural image synthesis and scale dramatically with increases in data and compute (Brock al. 2018). However, GANs are prone to instability due to the difficulty of achieving local equilibrium between the two networks. Frequent failures include one or both networks diverging or the generator only capturing few modes of the empirical distribution. Several proposed remedies include modifying training objectives (Arjovsky) al. 2017| Maatticu 2018). hicrarchical methods (Karras al. (017). instance selec- tion (inha et al.) 2020, latent optimization (Wu et al., (019). and strongly regularizing one or both networks (Gulrajani et al. 2017 Miyato et al. [018]] Dieng et al. 2019). among others. In practice, one or all of the above techniques are ultimately adapted to specific use cases.
Further, limits on data quantity empirically exacerbate training stability issues more often due to discriminator overfitting, Recent work on GANs for small sample sizes can be roughly divided into transfer learning approaches (Wang et al. 2018 Noguchi & Harada 2019, Mo et al., 2020 Zhao et al. 2020a) or methods which transform/ augment the available training data and provide the dis- criminator with auxiliary tasks. For example. Chen et al. (2019) propose a multi-task discriminator which additionally predicts the degree by which an input image has been rotated, whereas Zhang et al. (2020); Zhao et al.) (2020c) incorporate consistency regularization where the discriminator is penalized towards similar activations for transformed/auggmetee real and fake images. However, with consistency regularization and augmentation, network capacity is spent learning equivariance transformation as opposed to the desired task and equívariance is not guaranteed.
In this work, we consider the problem of training tabula rasa on limited data which possess global and even local symmetries. We begin by noting that GANs ubiquitously use convolutional layers
Work started and partially done during an internship at Merck & Co., Inc.
Work done while employed Merck & Co., Inc.

---

Published as a conference paper at ICLR 2021
0 90"
180"
270
0F
90'
180F
270"
Natural Pathology
Satellite Astronou
Figure 1: Several image modalities have no preferred orientation for tasks such classification. We improve their generative modeling by utilizing image symmetries within a GAN framework.
which exploit the approximate translation invariance and equivariance of image labels and distri- butions, respectively. Equivariance to geometric transformations is key understanding image representations (Bietti & Mairal 2019). Unfortunately, other symmetries (e.g., rotations and re- flections) inherent to modalities such as astronomy and medical imaging where galaxies and cells can be in arbitrary orientations are not accounted for by standard convolutional layers. To this end, Cohen & Welling (2016) proposed group-theoretic generalization of convolutional layers (group- convolutions) which in addition to translation, exploit other inherent symmetries and increase the expressive capacity of network thereby increasing its sample efficiency significantly in detection (Winkels & Cohen 2019). classification (Veeling et al., 2018). and segmentation (Chidester et al., 2019). Importantly, equivariant networks outperform standard CNNs trained with augmentations from the corresponding group (Veeling et al., 2018 Table 1), (Lafarge et al. [2020a] Fig. 7). See
Cohen et al. (2019); Esteves (2020) for a formal treatment equivariant CNNs.
Equivariant features may also be constructed via scattering networks consisting of non contrrainable Wavelet filters, enabling equivariance to diverse symmetries (Mallat 2012 Bruna Mallat 2013] Sifre & Mallat (013). Generative scattering networks include Angles & Mallat (2018) where standard convolutional decoder optimized to reconstruct images from an embedding generated by fixed scattering network and Oyallon et al. (2019) who show preliminary results using stan- dard convolutional GAN to generate scattering coefficients. We note that while both approaches are promising. they currently yield suboptimal synthesis results not comparable modern GANs. Cap- sule networks (Hinton et al.) 201T; Sabour et al., (017) are also equivariant and emerging work has shown that using capsule network for the GAN discriminator (Jaiswal al., 2019 Upadhyay & Schrater] 2018) improves synthesis on toy datasets. However, capsule GANs and generative scatter- ing approaches require complex training strategies, restrictive architectural choices not compatible with recent insights in GAN training, and have not yet been shown to scale to real-world datasets.
In this work, improve the generative modeling of images with transformation invariant labels by using an inductive bias of symmetry. We replace all convolutions with coup-convvlutions thereby admitting higher degree of weight sharing which enables increased visual fidelity, especially with uimieed-sample datasets. To our knowledge, we are the first to use croup-equivarie layers in the GAN context and to use symmetry-diie -diven considerations in both generator and discriminator archi- tectures. Our contributions are as follows,
1. We introduce symmetry priors via group-qquivariance generative adversarial networks.
2. We show that recent insights in improving GAN training fully compatible with group- equivariance with careful reformulations.
3. We improve class-conditional image synthesis across diversity of datasets, architectures. loss functions, and regularizations. These improvements are consistent for both symmetric images and even natural images with preferred orientation.
METHODS
2.1 PRELIMINARIES
Groups and group-convolutions. A group is set with an endowed binary function satisfying the properties of closure, associativity, identity, and invertibility. A two-dimensional symmetry group is
2

---

Published as a conference paper at ICLR 2021
~
Sample
Upoampied P4
Project Reshape
feature map
P4 feutuure map
rotatioos
Repeat: P4-P4 conv upearaple
cont
Upeamppee 2x
until desired resolution
Figure An abbreviated illustration of group-convolutions used in our generator networks.
the set of all transformations under which a geometric object is invariant with an endowed operation of composition. Given group G and a map & X + Y between two G-sets X and Y, & is said to be equivariant i.f.f. Φ(y -)) = g- ((x), Vo E X, Vg E G. Colloquially, an equivariant map implies that transforming an input and applying the map yields the same result applying the map and then transforming the output. Analogously, invariance requires that (g x) = (xx,, Va E X, Vg E G. In deep networks, equivariance to planar symmetry group can be achieved by either transforming filters (Cohen & Welling 2016) or feature maps (Dieleman et al. 2016
Our work utilizes the plane symmetry groups p4 (all compositions of 90-degree rotations and transla- tions) and p4m (all compositions of 90-degree rotations, reflections, and translations) (Schattschnei These groups can be parameterized neatly following Cohen & Welling (2016).
der 1978).
COo
sin cos
w
(-1) 1) cos sin ra
(-1) sin( 2
u
g(r, u, v)
sin
v 1
g ((m,r,u,)
tx cos
2
V
where g(r, u, parameterizes p4, g'mm, u, parameterizes p4m, degree rotations), m E {0,1} 1} (the number of reflections), and (u,v) E 2 (integer translations). The group operation is matrix multiplication for both groups. The matrix g(r, u, v) rotates and translates a point (expressed as homogeneous coordinate vector) in pixel space via left-muliplication. Analo- gous intuition follows for g'(m,r, u, v).
(the number of 90-
We now briefly define G-equivariant convolutions. We note that formally these are correlations and convolutions and that the literature uses the terms interchangeably. A G-convolution between vector-valued K-channel image f 2 + RR* and filter vb 2 R.K with (f1, f2, .......................... fk)
and 3 (4/, 2/2,.... ... ...,,,))))))))) can be expressed standard reference, one considers G to be the translation group on Z we have g y = y 9 and recover the standard convolution. After the first layer of G-CNN, we see that (ʃ ) is a function on G, necessitating that filter banks also be functions on Subsequent G-convolutional layers are therefore defined as [f * (g) = Eneo Σ fu(h)vx(g -1)..
()(g) Σ EEEZ Σ Sk(y) (g y) For
h). Finally, for tasks where the output is an image, it is necessary to bring the domain of feature maps from G back to Z We can pool the feature map for each filter over the set of transformations, corresponding to average or max pooling
over the group of rotations (or roto-reflections as appropriate).
GAN optimization and stability. As we focus on the limited data setting where training instability is exacerbated, we briefly describe the two major stabilizing methods used in all experiments here. We regularize the discriminator by using a zero-centered gradient penalty (GP) on the real data as proposed by Mescheder et al. (2088) of the form, R₁ -Em.-P IV [z)(( where y is the regularization weight, x i8 sampled from the real distribution Preat, and D is the discriminator. This GP has been shown to cause convergence (in toy cases). alleviate catastrophic forgetting (Thanh- Tung & Tran 2018), and strongly stabilize GAN training. However, empirical work has found that this GP achieves stability at the cost of worsening GAN evaluation scores (Brock et al. 2018).
3

---

Published as a conference paper at ICLR 2021
A widely used technique for GAN stabilization is spectral normalization (Myytto et al. 2018 which constrains the discriminator to be 1-Lipschitz, thereby improving gradient feedback to the generator (Zhou et al. 2019: Chu et al.] 2020). With spectral normalization, each layer is rescaled as, W W/)(W), ), where W is the weight matrix for a given layer and o (W) is its spectral norm. In practice, a (W) is estimated via power iteration method opposed to computing the full singular value decomposition during each training iteration. Finally, applying spectral normalization to both generator and discriminator empirically improves training significantly (Zhang et al. 2018
2.2 GROUP EQUIVARIANT GENERATIVE ADVERSARIAL NETWORKS
Here, outline how to induce symmetry prior into the GAN framework. Implementations available at h /githbb thub com wwwoo//eell wwwpde/eeewwaaareeeeeeeeee
The literature has developed several techniques for normalization and conditioning of the individual networks, along with unique architectural choices we extend these developments to the equivariant setting. We start by replacing all convolutional layers with group-convolutional layers where filters and feature maps are functions on a symmetry group G. Batch normalization moments (loffe& & Szegedy 2015) are calculated per group-feature map as opposed to spatial feature maps. Pointwise nonlinearities preserve equivariance for the groups considered here. Pre-activation residual blocks common to modern GANs are used freely as the sum of equivariant feature maps on is also equivariant.
high-level in Figure 2 We use fully connected layer to linearly project and reshape the concatenated noise vector z .N I) and class embedding into spatial feature maps Z². We then use enetralllynormmaizerr group-convolutions, inter- spersed with pointwise-none www.iiie-ooniinaaiiies.s. and nearest-neighboutt upsampling to increase spatial extent. We use upsampling followed by group-convolutions instead of transposed group- conpoloonvolutions to reduce checkerboard artefacts (Odena et al.) (016). We further use a novel group-equivariat class- conditional batch normalization layer (described below) normalize and class- condiionnoiion image generation while also projecting the latent vector to each level of the group-convolutional hierar- chy. We finally max-pool over the set of transformations to obtain the generated image xr
Generator.
The generator is illustrated
The group-equivarian discriminator receives an input x, which it maps to a scalar indicating whether it real or fake. We do this via spectrally normalized group-convolutions contwise-nomlinerrite
Discriminator:
and spatial-pooling layers to decrease spatial extent. After the final group- convolutional layer, we pool over the group and use global average pooling to obtain an invariant representation at the output. Finally, we condition the discriminator output via the projection method proposed by Miyato Koyama| (2018). Importantly, the equivariance of group-convolutions de- pends the convolutional stride. Strided convolutions were commonly used for downsampling in early GANs (Radford et al. 2015). However, stride values must be adjusted to the dataset to preserve equivariance, which makes comparisons to equivalent non-equivariant GAN architectures difficult. We therefore use pooling layers over the plane (commonly used in recent GANs) to downsample in all settings preserve equivariance and enable fair comparison.
Spectral Normalization. As the singular values of matrix are invariant under compositions of 90- degree rotations, transpositions, and reflections spectral normalization a group-weight matrix preserves equivariance and we use freely.
Class-conditional Batch Normalization. Conditional batch normalization (Perez al. 2018) places the scale and shift of features with an affine transformation learned from the class label (and optionally from the latent vector well (Brock et al. 2018) via linear dense layers, and is widely used in generative networks. We propose a group-equivariane preserving conditional normaliza- tion by learning the affine transformation parameters per group-feature map, rather than each spatial feature. As we use fewer group-filte filters than equivalent non-equivariant GANs, we use fewer dense parameters learn conditional scales and shifts.
EXPERIMENTS
Common setups. In each subsection, we list specific experimental design choices with full details available in App. C For each comparison, the number of group-filters in each layer is divided by the square root of the cardinality of the symmetry set to ensure a similar number parameters to the standard CNNs enable fair comparison. We skew towards stabilizing training over absolute
4

---

Published as a conference paper at ICLR 2021
Table 1: A summary of the datasets considered in this paper. The right-most column indicates whether the dataset has a preferred pose.
Dataset
Resolution
Delnsses
Dtraining
Avalidation
Pose Preference
Rotated MNIST
(28, 28)
10
12,000
50,000
No
ANHIR
(128, 128, 3)
5
28,407
9,469
No
LYSTO
(256, 256, 3)
20,000
No
CIFAR-I0
(32, 32, 3)
10
50,000
10,000
Yes
Food-101
(64, 64, 3) 101
75,747
25,250
Yes
Table 2: Min. & mean Fréchet distances (lower is better) of generated RoMMNIST samples, evalu- ated at every IK generator iterations. All evaluations are visualized in Appendix A Figure
Min. Mean Fréchet Distance
Available Training Data
Loss
Setting
10%
33%
66%
100%
Real data
0.6854
0.3208
0.1324
0.1296
RAGAN
CNN in G & D
(2.04, 11.40)
(1.42, 11.65)
(1.20, 11.10)
(1.36, 11.68)
CNN in G & G-CNN
(1.84, 4.26)
(0.88, 3.26)
(0.52, 2.85)
(0.53, 3.12)
G-CNN in G & CNN in D
(1.49, 9.75)
(1.08, 9.29)
(0.90, 8.70)
(0.95, 9.62)
G-CNN in G & D
(1.61, 4.25)
(0.76, 3.40)
(0.54, 2.92)
(0.53, 2.90)
NSGAN
CNN in & CNN in G & G-CNN in
(1.00, 7.02)
(0.74, 8.25)
(0.84, 8.07)
(0.97, 8.49)
2.488
(1.02, 3.51)
(0.55, 2.85)
(0.54, 3.08)
G-CNN & CNN D
(1.00, 7.00)
(0.96, 7.42)
(0.87, 6.83)
(0.94, 7.52)
G-CNN in G &
(2.85, 5.67)
(1.04, 4.24)
(0.82, 3.27)
(0.64, 3.32)
CNN in
(3.42, 16.21)
(3.90, 18.32)
(3.87, 17.81)
(4.88, 19.40)
WGAN
CNN in G & G-CNN D
(2.87, 5.98)
(0.76, 4.11)
(0.50, 3.57)
(0.39, 3.51)
G-CNN CNN in
(2.67, 16.02)
(3.40, 17.03)
(3.77, 17.76)
(3.74, 17.82)
G-CNN in &
5.677
(0.58, 3.32)
(0.56, 3.52)
(0.54, 3.76)
performance compare models under the same settings to obviate extensive checkpointing typically required for BigGAN-like models. Optimization performed via Adam (Kingma & Ba 2014) with 31 0.0 and B2 = 0.9, as in Zhang et al. (2018) Brock et al. (2018). Unless otherwise noted, all discriminators are updated twice per generator update and employ unequal learning rates for the generator and discriminator following Heusel et al. (2017). We use exponential moving average
(a = 0.9999) of generator weights across iterations when sampling images as in Brock et al. (2018). All initializations use the same random seed, except for RotMNIST where we average over 3 random seeds. An overview of the small datasets considered here is presented in Table
Evaluation methodologies. GANs are commonly evaluated by embedding the real and gener- ated images into the feature space of an ImageNet pre-trained network where similarity scores are computed. The Fréchet Inception Distance (FID) (Heusel al.) (017) jointly captures sample fi- delity and diversity and is presented for all experiments. To further evaluate both aspects explicitly, present the improved precision and recall scores (Kynkäänniemi et al. 2019) for ablations on real-world datasets. As the medical imaging datasets (ANHIR and LYSTO) are not represented in ImageNet, we finetune Inception-v3 (Szegedy al.] (016)) prior feature extraction for FID cal- culation as in |uang et al. (2018). For RoIMNIST, we use features derived from the final pooling layer of the p4-CNN defined in Cohen Welling (2016) to replace inceptionffeaturization analogous approach
taken in Binkkowki et al. (2018) in their experiments on the canonical MNIST dataset. Natural image datasets (Food-101 and CIFAR-10) are evaluated with the official Tensorfles Inception-v3 weights. Importantly, we perform ablation studies all datasets to evaluate group-equivariance in either or both networks.
We note that the FID estimator is strongly biased (Binkowski al. 2018 and work around this limitation by always generating the same number of samples as the validation set as recommended in Binkowski al. (201). An alternative Kernel Inception Distance (KID) with negligible bias has
5

---

Published as a conference paper at ICLR 2021
Inter-label Interpolation (coustant noise)
Latent Space Interpolation (constant label)
Boscltee
Beselliw
Propased
Inter-Label Interpolation (constant noise)
H1AEE
Ki-47
Proposed Hoseiiie
a ceeeer
WWWWNN
(a) RotMNIST
(b) ANHIR
Figure 3: Qualitative GAN interpolation (White 2016) results. (a) Selected spherical interpolations between generated RoMMNIST samples in either latent space (top) between labels (bottom). Equivariant GANs interpolate intuitively between samples, whereas standard GANs do not. (b) Selected inter-label linear interpolations between two staining dyes in synthesized ANHIR images. The standard model (top) changes both structure and dye between the generated samples, whereas the equivariant model (bottom) better preserves structure while translating between dyes.
been proposed (Binkowski et al. 2018), yet large-scale evaluation (Kurach et al. 2019) finds that KID correlates strongly with FID. We thus focus on FID in our experiments in the main text.
3.1 SYNTHETIC EXPERIMENTS: ROTATED MNIST
Rotated MNIST Larochelle et al. 2007) provides random rotations of the MNIST dataset and is a common benchmark for equivariant CNNs which
to measure sensitivity to dataset size, loss function, and equivariance either network to motivate choices for real-world experiments. We ex- periment with four different proportions of training data: 10%, 33%, 66%, and 100%. Additionally, the non-saturating loss (Goodfellow et al. 2014) (NSGAN), the Wasserstein loss (Arjovsky et al., 2018) (RaGAN) are tested.
2017) (WGAN), and the relativistic average loss Jolicoeur-Martingg For the equivariant setting, all convolutions are replaced with p4-convolutions. p4m is precluded as some digits do not possess mirror symmetry, All settings were trained for 20,000 generator iterations with batch size of 64. Implementation details are available in Appendix C.2.1
Results. Fréchet distance of synthesized samples the validation set is calculated every thou- sand generator iterations. As shown in Table we find that under nearly every configuration of loss and data availability considered, using p4-convolutions and minimum Fréchet distance. As data availability increases, the best-case minimum and mean FID scores improve. With f33%, 66%, 100%} the data, most improvements come from using p4-discriminator, with the further usage of p4-generator only helping in few cases. At 10% data, having an equivariant generator more impactful than an equivariant discriminator. These where
either network improves both the mean
trends are further evident from App. A Fig. 6 verge faster than non-equivariant counterparts. The NSGAN-GP and RAGAN-GP losses perform similarly, with WGAN-GP underperforming initially and ultimately achieving comparable results. Qualitatively, the equivariant model learns better representations as shown in Figure 3(a). Holding the class-label constant and interpolating between samples, we find that the standard GAN changes the shape of the digit in order to rotate it, whereas the equivariant model learns rotation in the latent space. Holding the latent constant and interpolating between classes shows that our model learns an intuitive interpolation between digits, whereas the standard GAN transforms the image immediately.
that GANs with pildiisrimmnntoos
3.2 REAL-WORLD EXPERIMENTS
Datasets. p4 and p4m-equivariant networks most useful when datasets possess global roto(- reflective) symmetry, yet have also been shown to benefit generic image representation due to local symmetries (Cohen & Welling 2016 Romero et al. 2020). To this end, experiment with two

---

Published as a conference paper at ICLR 2021
Table 3: FID evaluation (lower is better) of all real-world datasets across ablations and augmentation-bues basedd baseline comparisons. indicates an inapplicable setting for the method. Setting
ANHIR
LYSTO
CIFAR-10
Food-101 101
Ablation
CNN in G & D
7.32
7.27
20.89
27.34
G-CNN in G; CNN in D
6.93
6.68
21.20
24.16
CNN in G; G-CNN in D
5.56
5.02
17.09
16.91
G-CNN in G & D
5.54
3.90
17.49
17.73
CNN in G & D Standard Aug.
7.57
6.59
37.41
35.18
Aug.
CNN in G & D bCR (Zhao et al. 2020ch
5.86
4.78
19.64
21.18
CNN in G & D AR (Chen al. 2019)
19.59
20.39
G-CNN in G & D + bCR Zhao et 2020c)
5.19
4.53
17.94
15.55
CIFAR-10
Food-101 101
LYSTO
Place
Torare
Cls
Cooque Modlame
Biied
Edommme
Dod
Frgg
Bos Soup
Hoese
Eshimbhpp
Boot
Figure 4: Selected generated samples using the best performing equivariant models with no aug-
mentation. Random samples are available in App. Layout inspired by Karras al. (2020a).
types of real-world datasets detailed in Table (1) sets with roto(-reflective) symmetry, such that the image label is invariant under transformation; (2) natural images with preferred orientation (c.g., the boat class of images in CIFAR-I0 10 cannot be upside-down). Briefly, they
ANHIR provides high-resolution pathology slides stained with 5 different dyes to highlight different cellular properties (Borovec al. 2020 2018). We extract 128 128 foreground patches from images of different scales, as described in App. C.1.2 We use the staining dye conditioning.
LYSTO is multi-organ pathology benchmark for the counting of immunohistochemistry stained lymphocytes (Ciompi et al., 2011)). We re-purpose it here for conditional synthesis at a higher resolution of 256 256. As classification labels are not provided, we use the organ source as class labels. The use of organ sources as classes is validated in App. C.I.I The high image resolution in addition to the limited sample size of 20,000 make LYSTO a challenging dataset for GANs.
CIFAR-1O natural image vision benchmark of both small resolution and sample size (rizhevsky al. 2009). Previous work (Weiler Cesa 2019 Romero al. 2020) finds that equivariant-
networks improve classification accuracy CIFAR-10 and we include here it GAN benchmark.
Food-101 is small natural image dataset of a 101 categories food taken in various challenging settings of over/under exposure, label noise, etc. (Bossard et al. 2014). Further, datasets with high number classes are known to be challenging for GANs (Odenal 2019). Importantly, even though the objects in this dataset have a preferred pose due to common camera orientations, speculate that roto ooto-equivariance may be beneficial here as food photography commonly takes an en face or mildly oblique view. We resize the training 64 64 resolution for our experiments.

---

Published as a conference paper at ICLR 2021
ANHIR
LYSTO
Food-101
CIFAR-1O
[lecul
Clitic bock
144
179
Chilee & G
130
1900 2000
Gravatic Besuiica
Cravaia
Improved Precision and Recall KKnkääänniemi et al. 2019 analysis of ablations for all snapshots of trained models in each setting (closer to top-right is better). Bottom: GAN convergence (FID vs. generator updates) of standard GANs vs. our proposed models for all datasets. For visual clarity, we show only a subset of comparisons with convergence plots for all methods provided in App. Fig. Readers are encouraged to zoom-in for better inspection.
Figure 5: Top:
Baseline architecture. To produce a strong non-equivariant baseline, we face several design choices. State-of-the-ar GANs follow either BigGAN (Brock et al. 2018) StyleGAN2 (Karras) cet al. (020b) in design. As StyleGAN2 has not yet been demonstrated to scale to conditional gen- eration with large number of classes (to our knowledge), we follow BigGAN-like construction despite the stability of StyleGAN2. For our small datasets, we make the following modifications: (1) we use fewer channels; (2) we do not use orthogonal regularization; (3) we do not use hicrarchi- cal latent projection as we find in early testing that projecting the entire latent to each normalization layer achieves similar results; (4) we do not use attention as equivariant attention is an area of active research (Romero& & Hoogendoom 2019 Romero et al. 2020) but currently has prohibitively high memory requirements and may not yet scale to GANs. Further details are available in App. C.2
We then modify either generator (G) and/or discriminator (D) as in Section 2.2 to obtain the corre- sponding equivariant settings. We note that a discriminator invariant to rote-reflections would assign the same amount of realism to an upright natural image versus a rotated/reflected copy of the same image, allowing the generator to synthesize images at arbitrary orientations. Therefore, for CIFAR- 10 and Food-101 we pool over rotations before the last residual block to enable the discriminator to detect when generated images are not in their canonical pose while maintaining most of the bene- fits of equivariance studied in Weiler & Cesal (2019). We use p4m-equivariance for ANHIR and LYSTO and p4-equivariance for CIFAR-10 10 and Food-101 to reduce training time.
Comparisons. A natural comparison would be against standard GANs using augmentations drawn from the same group our model is equivariant to. However, augmentation on the real images alone would lead the augmentations "leaking" into the generated images, e.g., vertical flip augmentation may lead to generated images being upside-down. Zhao et al. (2020c) propose balanced consistency regularization (bCR) for augmentations of both real and generated samples to alleviate this issue, and we thus use it as comparison. We restrict the augmentations used in bCR to 90-degree rotations or 90-degree rotations and reflections as appropriate to enable fair comparison against equivari- ant GANs. Using additional augmentations would help all methods across the board. We further compare against auxiliary rotations (AR) GAN (Chen et al., 2019) where real and fake images are augmented with 90-degree rotations and the discriminator is tasked with predicting their orientation. We do not use AR for ANHIR and LYSTO as they have no canonical orientation. For completeness, also evaluate standard augmentation (reals only) for all datasets.
Results. Quantitative FID results of ablations and comparisons against baselines are presented in Ta- B Equivariant networks (G-CNNs outperform methods which use standard CNNs with with- out augmentation across all datasets. For ANHIR and LYSTO, we find that -quivvarraee in ei- ther network improves FID evaluation, with the best results coming from modifying both networks.
8

---

Published as a conference paper at ICLR 2021
However, for the upright datasets CIFAR-10 and Food-101, find that having a p4-equivariant discriminator alone helps more than having both networks be pl-equivariant. We speculate that this effect is in part attributable to their orientation bias. With bCR and AR GANs, we find that standard CNNs improve significantly, yet are still outperformed by equivariant nets using no augmentation. We include mixture of equivariant GANs and bCR for completeness and find that for ANHIR and Food-101, they have an additive effect, whereas they do not for LYSTO and CIFAR-10, indicating a dataset-sensiitvityy Of note, we found that bCR with its suggested hyperparameters lead im- mediate training collapse on ANHIR, LYSTO, and CIFAR-IO, which was fixed by decreasing the strength of the regularization substantially. This may be due to the original work using several dif- ferent types of augmentation and not just roto-reflections. Standard augmentation (i.e., augmenting training images alone) lead to augmentation leakage for CIFAR-10 10 and Food-101.
Qualitatively, as class differences in ANHIR should be stain (color) based, we visualize inter-class interpolations between synthesized samples in Figure bb) We find that our model better pre- serves structure while translating between stains, whereas the non-equivariant GAN struggles to do so. In our ablation study in terms of precision and recall in Figure 5 using p4m-equivariance in G and D achieves consistently higher recall for ANHIR and LYSTO. For Food-101, we find that G-CNN in G and D achieves higher precision, whereas CNN in G and G-CNN in D achieves higher recall. For CIFAR-10 precision and recall, find no discernable differences be- tween the two settings with lowest FID. Interestingly, for CIFAR-10 adding p1-equivariance to G but not worsens FID but noticeably improves precision. These observations are consistent with our FID findings as FID tends to correlate better with recall (Karras et al. [2020a). Finally. we plot FID vs. generator updates in Figure [ finding that the proposed Tramework converges faster than the baseline as a function of training iterations (for all datasets except ANHIR). Convergence plots for all datasets and all methods compared can be found in App. Figure 7 showing similar trends.
DISCUSSION
Future work. We present improved conditional image synthesis using equivariant networks, open- ing several potential future research directions: (1) As efficient implementations of equivariant at- tention develop, we will incorporate them model long-range dependency; (2) Equivariance to continuous groups may yield further increased data efficiency and more powerful representations. However, doing so may require non-trivial modifications current GAN architectures as memory limitations could bottleneck continuous group-equivariant GANs at relevant image sizes. Further, adding more discretizations beyond 4 rotations on a continuous group such has SF) may show diminishing returns (Lafarge et al.) 2020a F)g.7); (3) In parallel to our work, Karras et al. (2020a) propose a differentiable augmentation scheme for limited data GANs pertaining to which transfor- mations to apply and learning the frequency of augmentation for generic images, with similar work presented in Zhao et al. (2020b). Our approach is fully complementary to these methods when em- ploying transformations outside the considered group and will be integrated into future work; (4) Contemporaneously. Lafarge et al.] (2020b) propose equivariant variational autoencoders allowing for control over generated orientations via structured latent spaces which may be used for equivari- ant GANs as well: (5) The groups considered here do not capture all variabilities present in natural images such as small diffeomorphic warps. Scattering networks may provide an elegant framework to construct GANs equivariant to wider range of symmetries and enable higher data-efficiency.
Conclusion. We present a flexible framework for incorporating symmetry priors within GANs. In doing so, we improve the visual fidelity of GANs in the limited-data regime when trained on sym- metric images and even extending to natural images. Our experiments confirm this by improving on conventional GANs across a variety of datasets, ranging from medical imaging modalities to real-world images of food. Modifying either generator or discriminator generally leads to improve- ments in synthesis, with the latter typically having more impact. To our knowledge, our work is the first to show clear benefits of cquivariant learning over standard GAN training on high-resolution conditional image generation beyond toy datasets. While this work is empirical, we believe that it strongly motivates future theoretical analysis of the interplay between GANs and equivariance. Fi- nally, improved results over augmentation-basse strategies are presented, demonstrating the benefits of explicit transformation equivariance over equivariance-approoinnee
regularizations.
Acknowledgements. Neel Dey thanks Mengwei Ren, Axel Elaldi, Jorge Ono, and Guido Gerig.

---

Published as a conference paper at ICLR 2021
REFERENCES
Tomás Angles and Stéphane Mallat. Generative networks as inverse problems with scattering transforms. In International Conference on Learning Representations, 2018. URL https:
//openzeviess ew.
fbbR-
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv:1701.07875. 2017.
Wasserstein gan.
arXiv preprint
Alberto Bietti and Julien Mairal. Group invariance, stability to deformations, and complexity of deep convolutional representations. The Journal of Machine Learning Research, 20(1):876-924, 2019.
Mikolaj Binkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018.
Jirí Borovec, Arrate Barrut,, and Jan Kybic. Benchmarking of image registration methods for differently stained histological slides. In 2018 25th IEEE International Conference on Image Processing (ICIP), pp- 3368 -372. IEEE, 2018.
Jiří Borovec, Jan Kybic, Ignacio Arganda- Carreras, Dmitry V Sorokin, Gloria Bueno, Alexander Khvostikov, Spyridon Bakas, Eric, Chao Chang, Stefan Heldmann, al. Anhir: automatic non- rigid histological image registration challenge. IEEE Transactions on Medical Imaging, 2020.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 -mining discriminative com- ponents with random forests. European Conference on Computer Vision, 2014.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint 18Xi:: 1096, 2018.
J. Bruna and S. Mallat. Invariant scattering convolution networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 3878)1187- 2013. doi: 10.1 10.1009//AMM...01.223...
Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby. Self-supervised gans via auxiliary rotation loss. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12154-12163, 2019.
Benjamin Chidester, That- Vinh Ton, Minh-Triet Tran, Jian Ma, and Minh N Do. Enhanced rotation- cquivariant u-net for nuclear segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 0-0, 2019.
Casey Chu, Kentaro Minami, and Kenji Fukumizu. Smoothness and stability gans. In Interna- tional Conference Learning Representations, 2020. URL http:::///ppeeeeeeeeee.. net/ Forum? R.JunekkkRJeDDekHKk
Francesco Ciompi. Yiping Jiao, Jeroen van Laak. Lymphocyte assessment hackathon (lysto), October 2019. URL htt.t:ts::/d...rggg/// http:////1..212//ecc/cee 3tt.://2enood..13.5... 3513571
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International confer- ence on machine learning, pp. 2990-2999, 2016.
Taco S Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant cnns homo- geneous spaces. In Advances Neural Information Processing Systems, pp. 9142-9153, 2019.
Sander Dieleman, Jeffrey De Fauw, and Koray Kavukcuoglu. Exploiting cyclic symmetry in convo- lutional neural networks. arXiv preprint arXiv:1602.02660 2016.
Adji B Dieng, Francisco JR Ruiz, David M Blei, and Michalis Titsias. Prescribed generative adversarial networks. arXiv preprint 1rXiv:1910.043000 19101940.000....... 2019.
Carlos Esteves. Thcoretical aspects of group equivariant neural networks. arXiv preprint a55 2020.
10

---

Published as a conference paper at ICLR 2021
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farkey Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-6622222 2680, 2014.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. In Advances in neural information processing systems, pp. 5767-5777, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
Geoffrey E Hinton, Alex Krizhevsky, and Sida Wang. Transforming auto-encoders. Interna- fional Conference on Artificial Neural Networks, pp. 51. Springer, 2011.
Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to- image translation. In European Conference on Computer Vision, pp. 179-196. Springer, 2018.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 2015.
Phillip Isola, Jun- Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-imant translation with conditional adversarial networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1124-1134, 2017.
Ayush Jaiswal, Wael AbdAlmageed, Yue Wu, and Premkumar Natarajan. Capsulegan: Generative adversarial capsule network. In Laura Leal-Taixé and Stefan Roth (eds.), Computer Vision ECCV 2018 Workshops, Cham, 2019. Springer International Publishing.
Alexia
The relativistic discriminator: a key element missing from standard gan. arXiv preprint arXiv:1807.00734, 2018.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arXiv: 17000117/0 2017.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data, 2020a.
Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020b.
Junho Kim, Minjae Kim, Hyeonwoo Kang, and Kwang Hee Lee. U-gat-it: Unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-imane translation. In International Conference on Learning Representations, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69853 2014.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Karol Kurach, Mario Lučić, Xiaohua Zhai, Marcin Michalski, and Sylvain Gelly. A large-scale study regularization and normalization in GANs. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3581-3590, Long Beach, California, USA, 09- 15 Jun 2019. PMLR.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In Advances in Neural Information Processing Systems, pp. 2939-3938, 2019.
Maxime W. Lafarge, Erik J. Bekkers, Josien P. W. Pluim, Remco Duits, and Mitko Veta. Roto- translation equivariant convolutional networks: Application to histopathology image analysis, 2020a.
11

---

Published as a conference paper at ICLR 2021
Maxime W. Lafarge, Josien P. W. Pluim, and Mitko Veta. Orientation-disentannge unsupervised representation learning for computational pathology, 2020b.
Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, and Yoshua Bengio, An empir- ical evaluation of deep architectures on problems with many factors of variation. In Proceedings of the 24th international conference on Machine learning, pp. 473 473-480. ACM, 2007.
Jac Hyun Lim and Jong Chul Ye. Geometric gan. arXiv preprint 1eXivv1705.02840
Stéphane Mallat. Group invariant scattering. Communications on Pure and Applied Math-
ematics, 65(10):133--1398, 2012. doi: https://0i..gg///.000//g/ggg222
URL https:
//onlinelibrary wil..wwlli wiley. com doi /ttp://ooo/oo/1.00////..paa../
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? arXiv preprint arXiv:1801.04406 2018.
Takeru Miyato and Masanori Koyama. cgans with projection discriminator. arXiv preprint arXiv:1802.05637, 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv: arXiv: 180200595 2018
Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Freeze the discriminator: a simple baseline for fine- tuning gans, 2020.
Atsuhiro Noguchi and Tatsuya Harada. Image generation from small datasets via batch statistics adaptation. arXiv preprint arXiv:1904.0177 2019.
Augustus Odena. Open questions about generative adversarial networks. Distill, 2019. doi: 10. 23915/distill.0001.
Augustus Odena, Vincent Dumoulin, and Chris Olah. Deconvolution and checkerboard artifacts. Distill, 1(10)ee3, 2016.
E. Oyallon, S. Zagoruyko, G. Huang, N. Komodakis, S. Lacoste-Julien, M. Blaschko, and E. Belilovsky. Scattering networks for hybrid representation learning. IEEE Transactions Pattern Analysis and Machine Intelligence, 41(9):220--222 2222222, 2019. doi: 10. 19110////MM...11.. 2855738.
Ethan Perez, Florian Strub, Harm Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: (5t:://111.64444 2015.
David W Romero and Mark Hoogendoorn. Co-attentive equivariant neural networks: Focusing equivariance transformations co-occurring in data. arXiv preprinl arXiv:1911.07888 2019.
David W. Romero, Erik J. Bekkers, Jakub M. Tomezak, and Mark Hoogendoom. Attentive group
equivariant convolutional networks, 2020.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules, In Advances in neural information processing systems, pp. 3856-3866, 2017.
Doris Schattsehneider. The plane symmetry groups: their recognition and notation. The American Mathematical Monthly, 85(6):439-45 450, 1978.
Laurent Sifre and Stephane Mallat. Rotation, scaling and deformation invariant scattering for tex- ture discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2013.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
12

---

Published as a conference paper at ICLR 2021
Samarth Sinha, Han Zhang, Anirudh Goyal, Yoshua Bengio, Hugo Larochelle, and Augustus Odena. Small-gan: Speeding up gan training using core-sets. arXiv preprint arXiv:1910.1 2019.
Samarth Sinha, Anirudh Goyal, Colin Raffel, and Augustus Odena. Top-k training of gans: Improv- ing generators by making critics less critical. arXiv preprint arXiv:2002.066224 22222002..0622,.. 2020.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826 2016.
Hoang Thanh- Thanh-Tung and Truyen Tran. On catastrophic forgetting in generative adversarial networks. arXiv preprint 18w0:04011011 2018.
Dustin Tran, Rajesh Ranganath, and David Blei. Hierarchical implicit models and likelihood-free variational inference. In Advances in Neural Information Processing Systems, pp. 5523-5533,
2017.
Yash Upadhyay and Paul Schrater. Generative adversarial network architectures for image synthesis using capsule networks. arXiv preprint arXiv: 87880060776 2018.
Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equiv- ariant cnns for digital pathology. In International Conference on Medical image computing and computer-assinne -assiste intervention, pp. 210-218 Springer, 2018.
Yaxing Wang, Chenshen Wu, Luis Herranz, Joost van de Weijer, Abel Garcalez-Garcia, and Bog- dan Raducanu. Transferring gans: generating images from limited data. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 218 2018.
Maurice Weiler and Gabriele Cesa. General e (2)-equivariant steerable cnns. In Advances Neural Information Processing Systems, pp. 14334-14345, 2019.
Tom White. Sampling generative networks. arXiv preprint arXiv: 2016
Marysia Winkels and Taco Cohen. Pulmonary nodule detection in scans with equivariant cnns. Medical image analysis, 55:15-26, 2019.
Yan Wu, Jeff Donahue, David Balduzzi, Karen Simonyan, and Timothy Lillicrap. Logan: Latent optimisation for generative adversarial networks. arXiv preprint 1/20v:19/,.00953, 2019.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. arXiv preprint arXiv: 2018.
Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lee. Consistency regularization for
generative adversarial networks. In International Conference on Learning Representations, 2020. URL /RRLL/ttpps:/ppeeeeeeee...c wtwp:enrwviieeeeeeeerrneee
Miaoyun Zhao, Yulai Cong, and Lawrence Carin. leveraging pretrained gans for limited-data generation. arXiv preprint arXiv: / 2020a.
Shengyu Zhao, Zhijian Liu, Ji Lin, Man-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training, 2020b.
Zhengli Zhao, Sameer Singh, Honglak Lee, Zizhao Zhang, Augustus Odena, and Han Zhang. Im- proved consistency regularization for gans, 2020c.
Zhiming Zhou, Jiadong Liang, Yuxuan Song, Lantao Yu, Hongwei Wang, Weinan Zhang, Yong Yu, and Zhihua Zhang. Lipschitz generative adversarial nets. In Proceedings of the 36th International Conference on Machine Learning, pp. 7584-7593, 2019.
13

---

Published as a conference paper at ICLR 2021
A
SUPPLEMENTARY RESULTS
RAGAN-GP
NSGAN-GP
WGAN-GP
100% data H-écheet Dist.
62
30
50
20
20
18
2.00
5003
1500
10010
12000 (5000 17000 20000
2500
5000
7500
10003 12500 15000 17500 20000 Gccerntar Dirrotices
2500 3000 7300 10000 12000 15800 17300 20030
08% data Préchet Diitt.
GACNN
GONN
GAONN
GAONN
GANNN UNNM
GAUNN
CANN GANN
CON L.C-UND to
CHAN C, C-C-NNN
60
CNN la boch
60
CNN both
CAN In both
30
32
30
lheratioes
Toonloos
33% data Hréchet Dist.
GACNN 1603
GLLNN II (o4)
GALNN LE BOIL
70
G-CNN a in
70
GANN IN CN is
GLENN G CONN
ONN La CKN i
CON C. CNN is.
COSS la O. CANN
CNN Ea bvck
CNN both
CNEN In both
52
40
40
00
18
5002
10000 12000 15000 17500 20000
2507
1520
10003 12500 17500 20000
5200 7500 10000 12500 15500 17500 20010 Geeeeetor Persitooee
Genemttr Throstoee
Caneeator Terrafooee
80
82
10% data Fréchet Dist.
CONN
56
53
03
22
10
18
2303 5002
7500
10000
32000 15000 37500 20000) Thersdions
2500
3000
2500 30003
12500 15000 17500 20000 Groseato Ihrrailona
2200 3000 7320 10000 12200 13000 17300 20030
Figure 6: Convergence plots of all GAN ablation settings on Rotated MNIST across data availabil- ities (rows) and loss functions (columns). Fréchet distance the validation set is evaluated every 1,000 generator iterations, for 20,000 iterations total. Experiments are repeated with 3 different ran- dom seeds and average trajectories are reported with standard deviation error bars. This figure best interpreted alongside Table which
which lists best performances for each configuration.
14

---

Published as a conference paper at ICLR 2021
Comparisons
Ablations
200
CNK in and D SCR
G-CNN CNN in
175
175
G-CNN in and D
150
LYSTO ANHIR
125
FID
100
100
75
75
25
0
30000
20000
Generator Ibenations
Goneratoo Sterations
120
GNNNN in and
UNN In and
GNN in and + bCR
UNN G. GONN
80
FID
FID
80
60
40
20
0
10000
20000
300U0
QUUDO
50000
10000
20000
30000
4U000
50000
Generator lterations
Generator Sterations
200
CNN in G. G-CNN in D
175
CNN in and
CNN in and D MCR
CNN in G, G-CNN in
150
Food-100 101
150
125
100
75
75
50
25
25
0
10000
20006
30000
40000
10000
20000
30000
40000
Generater Iteatton
Ganerater Sterations
80
CNN G-CNN
CNN ta and D Aux. Rot
G-CNN G. CNN in
CNN and D bCR
G-CNN
CIFAR-10
CNN and DCR
G-CNN in and D
60
FID
30
FID
40
30
20000
600D0
RD000
100000
100000
Generator Teratonn
Figure 7: GAN convergence (FID generator updates) for baseline comparisons of the best per- forming methods (left) and ablations (right) for all datasets, Readers are encouraged to zoom-in for better inspection.
15

---

Published as a conference paper at ICLR 2021
Dompling piings
Oysters
Soaweed Salad
Gnooehi
Omelette
Babyback
Spaghetti
Belognese
Deviled Dgpss
Missella
Sachimi
Filet Mignon
Gyoza
Figure 8: Random 64 x 64 Food-101 samples from arbitrarily chosen classes with no truncation taken from the best performing model snapshot with p4-equivariance (without augmentation) in the discriminator.
16

---

Published as a conference paper at ICLR 2021
Real
Synthesized
Cc10
CD3
Ki67
Figure 9: Random 128 128 ANHIR samples with no truncation taken from the best performing model snapshot with pam-equivariiace in both generator and discriminator (without augmentation). Selected real samples are shown in the left column for reference.
17

---

Published as a conference paper at ICLR 2021
Real
Synthesized
Colon
Breast
Prostate
Figure 10: Random 256 256 LYSTO samples with no truncation taken from the best performing model snapshot with pam-equivariiace in both generator and discriminator (without augmentation). Selected real samples are shown in the left column for reference.
18

---

Published as a conference paper at ICLR 2021
(a) RotMNIST
b CIFAR-10
5
6
w
1
2
Truck Bost Tore Fox Box
R
8
8
8
P
Figure 11: Random samples for RoMMNIST (28 28) and CIFAR-10 (32 32) sampled with
0.75 truncation trained without augmentation.
Table 4: Kernel Inception Distance results for Map2Sat translation on the Maps dataset. Lower is better.
Setting
KID
Pix2Pix (Isola et al.
2017
0.1584 +: 0.0026
Pix2Pix Isola al.
2017
(optimized)
0.0663 +:2 0.0038
CNN in G. G-CNN m D
0.0333 + 0.0005
G-CNN in and D
0.0399 + 0.0024
B IMAGE-TO-IMAGE TRANSLATION
To show the generic utility of equivariance in generative adversarial network tasks, present pilot study employing p4-equivariance in supervised image-to-image translation to learn mappings between visual domains. Using the popular P Pix2Pix ix model of Isola et al. (2017) as baseline, we replace both networks with p4-equivariant models. For completeness, we also evaluate whether employing p4-equivariance in just the discriminator achieves comparable results modifying both networks, as in the natural image datasets in the main text.
We the 256 256 Maps dataset first introduced in (Isola et al. 2017), consisting of 1096 training and 1098 validation images of pairs of Google maps images and their corresponding satellite/acrial view images. As FID has highly biased estimator, its use for evaluating generation with only 1098 validation samples is contraindicated (Binkowski et al. 2018). We instead use the Kernel Inception Distance (KID) proposed by Binkowski al. (2018) which exhibits low bias for small sample sizes and is adopted in recent image translation studies (Kim et al. 2020). Briefly, as in FID, KID embeds real and fake images into the feature-space of an appropriately chosen network and computes the squared maximum-meat discrepancy (with polynomial kernel) between their embeddings. Lower values of KID are better. We use the official Tensorfl floorrrlow Low implementation and weights
For baseline P 11x2Pi ix, we use pre-trained weights provided by the authors Interestingly, find that their architectures can be optimized for improved performance by replacing transposed convolu- tions with resize-convoluttion,, reducing the number of parametens by swapping 4 x convolutional
eval eval/inccooo w/wnc//nepiiinnncm httppppcciioneerrssso...o
19

---

Published as a conference paper at ICLR 2021
Input
Pix2Pix
Ours
Real
Figure 12: Arbitrarily selected sample translations from input map images (Col. 1) using either base- line Pix2Pix with publicly available pre-trained weights (Col. 2) or Pix2Pix with a p4-equivariant discriminator (Col. 3). Real aerial images are shown in Col. 4.
kernels for 3 x kernels, and removing dropout. For equivariant models, we replace convolutions with polcconolutioons in this optimized architecture and halve the number of filters to keep the num- ber of parameters similar across settings. Architectures are given in Tables [5] and [6] We leave all other experimental details identical to Isola al. (2017) for all models, such training for 200 epochs with random crops under a cross-entropy GAN loss.
Quantitative results are presented in Table 4 which shows that p4-equivariance in either setting improves over both original baseline and optimized baseline by wide margin, with the best results coming from p4-equivariance in the discriminator alone. Qualitative results presented in Figure showing improved translation fidelity, further supporting our hypothesis that equivariant networks benefit GAN tasks generically.
20

---

Published as a conference paper at ICLR 2021
C EXPERIMENTAL DETAILS
C.1 DATA PREPARATION
C.I.1 LYSTO CLASS CONDITIONING
To validate the assumption of the organ source being discriminative feature, suitable test would be to train a classifier to distinguish hetween sources. We partition the original training set with 60/40 train/test split. The original testing set is not used as it has no publicly available organ source information. The dataset has classes colon, breast, and prostate. Holding out 20% of the new constructed training set for validation, we fine-tune ImageNet-pretrained V (Simonyan & Zisserman 2014) and achieve 98% organ classification test accuracy, thus validating our assumption.
C.1.2 ANHIR PATCU EXTRACTION
To extract patches for image synthesis, we choose the lung-lesicn images from the larger AN- HIR dataset, these images are provided different scales and possess diverse staining. The images were cropped to the nearest multiples of 128, and 128 x 128 patches were then extracted. Foreground/background masking was performed via K-means clustering, followed by morphologi- cal dilation. The images were then gridded into 128 x 128 patches, i.e., there was no overlap between patches. If patch contained less than 10% foreground pixels, it was excluded from consideration.
C.2 ADDITIONAL IMPLEMENTATION DETAILS
The following subsections list sataset- training strategies. Unless noted, all layers use or- thogonal initializations. Batch normalization momentum is set to 0.1, and LeakyReLU slopes set to 0.2 (if used). Spectral normalization is used everywhere except for the dense layer which learns the class embedding specified in the BigGAN PyTorch GitHub repository
For ablation studies, as GANs consist of two networks (the generator and discriminator), we replace equivaraui layers (convolutional, normalization, and pooling) with the corresponding stan- dard layers in either generator or discriminator to evaluate which network benefits the most from equivariant learning. When remove equivariant layers from both networks, recover our base- line comparison. All settings use roughly the same number of parameters, with a very small differ- ence in parameter count arising from the p4 (or p4m) class-conditional batch normalization layers requiring fewer affine scale and shift parameters than their corresponding standard normalization layers. Tangentially, we note that the equivariant networks require higher amounts of computation time. For example, for a fixed number of training iterations on ANHIR, pdm-equivariant GANs currently require approximately four times the amount of computation time.
To identify a common shared stable hyperparameter configuration for all ablations of our method on real datasets, a grid search was performed for the ANHIR dataset over learning rates for generator and discriminator (Mg Na) ((10-1 4 4 x 10-1), }, (5 x 10-3 2 x 10-4)), gradient penalty strengths (7 = {0.01, 0.1, 1.0, 10.0)), and binary choices as to whether to use batch normalization in the discriminator or not, whether to use average-pooling discriminator, and whether to use a Gaussian latent space or a Bernoulli latent space. We use the identified hyperparameter configuration as an initial starting point for all datasets, modifying them as appropriate as described below.
max-pooling reduce spatial extent in the
For ANHIR, LYSTO, and Food-101 use the relativistic average adversarial loss Jolicoeur- Martineau 2018) for its stability and for CIFAR-10 we use the Hinge loss (Lim & Yel 2017; Tran aux- iliary rotations GAN (Chen et al. 2019), we use the suggested regularization weights For balanced consistency regularization (bCR) (Zhao et al. 2020c), we find that dataset-specific tuning of the regularization strength was required.
al. (2017) to remain consistent with the literature for that dataset. For our implementation
21

---

Published as a conference paper at ICLR 2021
C.2.1 ROTMNIST
Given the low resolution of Rotated MNIST, we take straightforward approach to synthesis with- out residual connections. In the generator, we sample from a 64D Gaussian latent space, concate- nate class embeddings, and linearly project as described in Section 2.2 Four pectrally-normaliin convolutional layers are then used with class-conditional batch normalization employed after every convolution except for the first and last layer. The discriminator uses three spectrally normalized convolutional layers, with leaky ReLU non- mon-linearities. Average pooling is used to reduce the spatial extent of the feature maps, with global average pooling and conditional projection used at the end of the sequence. For NSGAN and RaGAN, we use the R₁ GP, conservatively setting 7 0.1. For WGAN, we use the GP defined in Gulrajani et al.] (2017) to ensure the 1-Lipschitz constraint with the recommended weight of 10.0. Learning rates were set GJ 0.0001 and n/ 0.0004, respectively. For the p4-equivariant models, max-pooling over rotations is used after the last group- convolutional layer in both generator and discriminator to get planar feature maps. Architectures are presented in Tables and
C.2.2 ANHIR
We sample from a 1281) Gaussian latent space with batch size of 32. The generator consists of 6 pre-activation residual blocks followed by a final convolutional layer to obtain a 3-channel out- put. We use class-conditions batch normalization after every convolution, except at the final layer. The discriminator uses pre-activation residual blocks, followed by global average pooling and conditional projection. In the equivariant settings, we use residual blocks with pám--cnnvluttinns for relo-reflective symmetries. We train with the relativisitic average loss and use the Rı GP with y 0.1. Learning rates are set N 0.0001 and np 0.0004. All models were trained for approximately 60,000 generator iterations. bCR weights for comparison were set to Areat = 0.1 and Sake 0.05 for roto-reflective augmentations, with higher values collapsing training. Archi- tectures are presented in Tables 7 and g
C.2.3 LYSTO
Implementation for LYSTO is similar to that of App. C.2.2 with some key differences due to the greater difficulty of training. Due to memory constraints, we use batch size of 16. We increase the number of residual blocks to 6 in both generator and discriminator and halve the number of filters. The equivariant settings used the p4m roto-reflective symmetries. We initially experienced low sample diversity across a variety of hyperparameter settings. Contrary to recent literature, we find that using batch normalization in the discriminator in addition to spectral normalization greatly improves training for this dataset. Further, halving the learning rates for both networks to GlG 0.00005 and ViD = 0.0002 and increasing the strength of the gradient penalty to 1.0 were necessary for ensuring training stability. As in App. C.2.2 all models were trained for approximately 60,000 generator iterations and bCR weights were set to Xreat = 0.1 and Xfake = 0.05 for roto-reflective augmentations. As test labels and Recall to the training set itself as done in a subset of experiments within Selicoeur-Martines (2018) and Zhao et al. (2020b). Architectures are presented in Tables 9 and 10
publicly available for LYSTO, we evaluate FID, Precision,
C.2.4 CIFAR-10
For CIFAR-1O, we make the following changes to our training parameters to be in accordance with prior art for BigGAN-like designs for this dataset: (1) layer weights are now initialized from (.0)), (2) average pooling used in the discriminator instead of max pooling: (3) learning rates ae and njn are now equal and set to 0.0002; (4) the discriminator is updated four times per generator update; (5) architectures are modified as Tables [T] and [2] (6) we use the Hinge loss instead of the relativistic average loss. We use batch size of 64. Karras et al. (202aa) suggest Ri GP weight of y = 0.01 for CIFAR-IU 10 which we use here. We train all CIFAR- 10 GANs for 100K generator iterations. bCR weights were set to Xreat 0.1 and Afake 0.1 for 90-degree rotation augmentations.
For the p4-equivariant discriminators, move the pooling the group before the last resid- ual block as stated in the main text. Alternatively, we experimented with using a single additional standard convolutional layer with 32 filters after the p4-residual blocks lightweight alternative
22

---

Published as a conference paper at ICLR 2021
(lateist, class) embedding
projettion
p4m GRatchNorm
p4m GBatchNorm
ReLU
ReLU
Ixl pfm--p4m
Upsample 2x
Upsample 2x
GConv8N
3x3 phm-p4m
GConv8N
3x3 p4m--pm GConvSN
p4m GBatchNorm
1xl p4m-p4m
2x2 plaue-group MaxPool
GConvSN
p4m GBatchhNrm
V.A
prijectim
ReLU
3x3 pém-pp4m
ReLU
GConvSN
3x3 p4m--p4m
2x2 plane-group MaxPool
GConvSN
ResBlockG
ResBlockD
Y
Figure 13: Residual blocks in the group-equivariant settings used in RGB image generation archi- tectures. The choice of p4 or p4m is dataset-specific. The generator uses ResBlockG (left) and the discriminator uses ResBlockD (right). The first residual block in the convolutional sequence in either network uses =2-p4m group-convolutions for the initial layer. The non-equivariant settings re- place all group-convolutions and normalizations within the residual blocks with standard techniques. Visual design inspired by Brock et al. (2018).
to making entire residual block non-equivariant but this worsened FID evaluation. Interestingly, we find that substituting Global Average Pooling for Global Sum Pooling in the CIFAR- 10 discrim- inators lead to improvement of 55 8 in terms of FID across the board. This architectural change to the hesseed GANs from Guirajani et al. (2017) was originally made in Miyato et al. (0018) but our knowledge has not been noted in the literature previously.
C.2.5 FOOD-101
Compared to the residual synthesis models in App. C.2.2 and C.2.3 we make several changes. We sample from 64D latent Gaussian to lower the number of dense parameters and substantially increase the width of the residual blocks to account for the high number of image classes. We find that an 8x increase in the number of channels for the initial projection from the latent vector and class embedding improves training significantly. We use 4 residual blocks each in both generator and discriminator. For the equivariant setting, we use only p4 rotational symmetries to reduce training increase the batch size 64 and the R1 GP to 7 1.0, both of which improve the evaluation of all experimental settings. We train all GANs for 455K generator iterations. The suggested bCR weights of Areat 10.0 and Ayake 10.0 from Zhao al. (2020c) were used here for 90 9e-reee rotation augmentations. However, when bCR with default parameters was combined with p4-eap equivarriaee in and D. augmentations start to 'leak' into the generated images (e.g.. generating upside-down plates), necessitating lower weights of X = 0.5 and Afake 0.5.
time. Importantly,
C.3 ARCHITECTURES
Architectures for the Rotated MNIST experiments are given in Tables and 6 ANHIR in Tables and 8 and LYSTO in Tables and TO The residual blocks used in the ANHIR, LYSTO, CIFAR- 10, and Food- 101 experiments are given in Figure [3] SN refers to spectral normalization and (=2 p4), (p4- p4), (z2 p4m), (p4m p4m) refer to the type of convolution used.
23

---

Published as a conference paper at ICLR 2021
Generator
Sample E R&1 ~ NV (0, 7)
Discriminator
Embed y E (0.....9) 9) into ÿ E IR
Input RGB image x E 2822xxx
Concatenate 2 and à into
R228
x 3 ConvSN, 1 - 128
Project and reshape /1 to 7 X 7 X 128
LeakyReLU, Avg. Pool
3 x 3 ConvSN, 128 - 512
3 x 3 ConvSN, 128 + 256
ReLU: Up 2x
LeakyReLU, Avg. Pool
3 x 3 CNn 512 - 256
3 3 ConvSN, 256 + 512
CCBN( h); ReLU; Up 2x
LeakyReL.U, Avg. Pool
3 ConvSN, 256 + 128
Global Average Pool into
CCBN( h:); ReLU
Embed y E {.,....) . into
3 x ConvSN, 128 +
Projection step(y',.)
tanh()
Table 5: Architectures used for the standard generator and discriminator in the Rotated MNIST experiments.
Generator
Sample 2 E R24 ~ N N(0, I)
Discriminator
Embed y {0,... ..., 9} into I 1864
Input RGB image & E 28x28x1
Concatenate and ÿ into / E R128
3 x 3 22 p4 GConvSN, + 64
Project and reshape h to 7 X X 128
LeakyReLU, Avg. Pool
3 3 x2
p1 GConvSN, 128 - 256
3 p4 p1 GConvSN, 64 + 128
ReLU; Up 2x
LeakyReI.U, Avg. Pool
3 x 3 p4 p4 GConvSN, 256 > 128
3 x 3 p4 p4 GConvSN, 128 + 256
CCBN(-, h): ReLU; Up 2x
LeakyReLU, Avg. Pool
3 3 p4 pd GConvSN, 128 - 64
p4-Max Pool
CCBN(, h); ReLU
Global Average Pool into
x p4 p4 GConySN, 64
Embed y E {0, .... 9) into
p4-Max Pool
Projection step(y',
tanh()
Table 6: Architectures used for the p4-equivariant generator and discriminator in the Rotated MNIST experiments.
Generator
Sample z E IR 128 ~ A(,0 I)
Discriminator
Embed y E {0, ..., 4} into ý E 128
Input RGB image x E 12
Concatenate and ý into h 2,256
z2 z2 ResBlockD, 3 32
Project and reshape / to 4 x 4 x 128
z2 - x2 ResBlockD, 32 - 64
x2 x2 ResBlockG, 128 - 512
z2 x2 ResBlockD, 64 + 128
=2 z2 ResBlockG, 512 + 256
z2 =2 ResBlockD, 128 + 256
z2 x2 ResBlockG, 256 + 128
22 - z2 ResBlockD, 256 + 512
z2 x2 ResBBlocG, 128 + 64
ReLU
=2- 22 ResBlockCi, 64 - 32
Global Average Pool into
BN; ReLU
Embed y E {O..... 4) into j
3 x ConvSN, 32 +
Projection step(y'.
tanh()
Table Architectures used for the standard generator and discriminator in the ANHIR experiments.
24

---

Published as a conference paper at ICLR 2021
Generator
Sample IR 128
N0, (0,4) I)
Discriminator
Embed y E .0..... into I E R
Input RGB image R 12
Concatenate 2 and // into
_256
z2 pAm ResBlockD, 3
Project and reshape h to 4 X 4 128
p4m p4m ResBlockD, 11 22
22 --m ResBlockG, 128 - 181
p4m p4m ResBlockD. 22 - 45
plm plm ResBlockG, 181 + 90
p4m p4m ResBlockD, 45 - 90
pAm p4mm ResBlockG, 90 + 45
pAm p4m ResBlockDD, 90 181
pAm pAm ResBlockG, 45 7 22
ReLU
pAm pAm ResBlockG, 22 +
p4m-Max Pool
pAm-BB: ReLU
Global Average Pool into
3 x 3 p4m p4m GConvSN, 11 +
Embed y E fO..... 4 into y
p4m-Max Pool
Projection step(y',
tanh()
Table 8: Architectures used for the pıln-equivariant generator and discriminator the ANHIR experiments.
Generator
Sample = E 128 *N(0, T)
Discriminator
Embed E {0, 2] into I 128
Input RGB image x 2556x20xx3
Concatenate and ý into h 256
22 x2
ResBlockD-BAl 3 16
Project and reshape / to 4 x x 128
z2 =2 ResBlockD-BN, 16 - 32
z2 z2 ResBlockG, 128 - 512
z2 =2 ResBlockD-BBN, 32 -> 64
z2 =2 ResBlockG, 512 > 256
22 z2 ResBlockD-B.N, 64 128
=2 - x2 ResBlockG, 256 + 128
z2 z2 ResBlockD-BN, 128 - 256
z2 =2 ResBlockG, 128 64
ResBlockD.BN, 256 512
22 22 ResBlockG, 64 + 32
ReLU
z2 x2 ResBlockG, 32 + 16
Global Average Pool into
BN; ReI.U
Embed E {0, 1, 1,2] into y
x ConvSN, 16 3 tanh()
Projection ft)
Table 9: Architectures used for the standard generator and discriminator in the LYSTO experiments.
Generator
Sample 2 E IR 128 NN(0, I)
Discriminator
Embed E {0, 1, into ÿ 122
Input RGB image x 256x
Concatenate and ý into h 256
22 p4m ResBlockD-BBl BB.. +
Project and reshape / to 4 x 4 x 128
p4m p4m ResBlockD-Bil 5 + 11
p4m ResBlockG, 128 181
p4m p4m ResBlockD-BBN, 11 -> 22
plm- pAm ResBlockG, 181 90
pAm p4m ResBlockD-BBN, 22 45
pAm pAm ResBlockG, 90 + 45
p4m p4m ResBlockD-BN, 45 + 90
pdm p4m ResBlockG, 45 + 22 22
pdm - -pdm ResBlockD-BN, 90 > 181
pAm pdm ResBlockG, 22 11
ReL.U
p4m p4m ResBlockG, 11 +
p4m--Mx Pool
p4m-BN; ReLU
Global Average Pool into
3 x 34m p4m GConvSN, 5 +
Embed y {0, 1,2) into ÿ
PAm-Max Pool
Projection step(0',
tanh()
Table 10: Architectures used for the p4m-equivariant generator and discriminator in the LYSTO experiments.
25

---

Published as a conference paper at ICLR 2021
Discriminator
Input RGB image x E 332x32x3
Generator
Sample 2 IR 128
NO( I)
22 - z2 ResBlockD (avg. pool),
Embed y .....9. 9) into ÿ E R 1
128
Concatenate 2 and ÿ into h E 2255
22 =2 ResBlockD (avg. pool),
128 + 128
Project and reshape to 4 x
256
22 z2 ResBlockD (no downsample).
x2 22 ResBlookG, 256 - 256
128 + 128
=2 =2 ResBlockG, 256 + 256
=2 x2 ResBlockD (no downsample).
z2 =2 ResBlockG, 256 - 256
128 128
BN; ReLU
ReL.U
3 x ConvSN, 256 - 3 tanh()
Global Sum Pool into f
Embed y E {0, .... 9) into Projection st)
Table 11: Architectures used for the standard generator and discriminator in the CIFAR-10 experi- ments.
Discriminator
Generator
Input RGB image x E R64x64x3
Sample z E IR 128 ~ NO, I)
x2 p4 ResBlockD (avg. pool),
Embed y E {0, ..., 9} into ő E R228
3 64
Concatenate and ý into / R256
=2 p4 ResBlockD (avg. pool),
64 + 64
Project and reshape h to 4 x x 256
p4 - p4 ResBlockD (no downsample).
x2 p4 ResBlockG, 256 + 128
64 -
p4 p4 ResBlockG, 128 + 128
p4-Max Pool
p4 p4 ResBlockG, 128 + 128
22 - 22 ResBlockD (no downsample),
p4-BN; ReLU
64 - 128
3 x 3 p4 p4 GConvSN, 128 +
ReLU
p4-Max Pool
Global Sum Pool into
tanh()
Embed y E {0, .... 9) into 7
Projection ft
Table 12: Architectures used for the p4-cquivariant generator and discriminator in the CIFAR-10 experiments.
Generator
Sample Q (0, I)
Discriminator
Embed y E {0, ..., 100} into E R
Input RGB image R64x64x
Concatenate and ý into h 128
x2 x2 ResBlockD, - 128
Project and reshape
x 1024
22 =2 ResBlockD., 128 256
z2 z2 ResBlockG, 1024 + 512
22 22 ResBlockD, 256 512
=2 =2 ResBlockG, 512 + 384
z2 =2 ResBlockD, 512 + 784 ReLU
z2 z2 ResBlockG, 384 + 256
x2 z2 ResBlockG, 256 192
Global Average Pool into f
BN; ReLU
Embed E (O,... 100) into Ø Projection step()',
3 x ConvSN, 192 + tanh()
Table 13: Architectures used for the standard generator and discriminator in the Food-101 experi- ments.
26

---

Published as a conference paper at ICLR 2021
Generator
Sample R64 (0, .... Embed y E {o.... 100} into E Rs
Discriminator
Input RGB image E 164x64x
Concatenate 2 and I into
12
z2 p4 ResBlockD, 3 + 64
Project and reshape
x 4 1024
p4 p4 ResBlockD,
128
22 p4 ResBlockG, 1024 - 256
p -4 ResBlockD, 128 - 256
p4 p4 ResBlockG, 256 - 192
p4-Max Pool
p4 p4 ResBlockG, 192 + 128
22 =2 ResBlockD, 256 784 ReLU
pd p4 ResBlockG, 128 + 96
p4-BN; ReLU
Global Average Pool into /
3 x p4 p4 GConySN, 96
Embed y E (0, .... 100) into g
p4-Max Pool
Projection step(y',
tanh()
Table 14: Architectures used for the pl-equívariant generator and discriminator in the Food-101 experiments.
Generator
Input RGB image E 256x256x3
h1:22 z2 DownBlock, + 64
h2 12::2 z2 DownBlock, 64 128
Discriminator
h2:22 222 DownBlock, 128 + 256
h4:22 =22 DownBlock. 512 + 512
Input RGB image x R256x256x3
his : =2 z2 DownBlock, 512 +12
Input RGB image y TR 256x256x3
h6s : =2 z2 DownBlock, 512 + 512
Concatenate and feature-wise
17:22 x2 z2 DownBlock, 512 - 512
x2 DownBlock, 3 > 64
hg ; x2 =2 DownBlock, 512 +12
22 z2 DownBlock, 64 - 128
22 z2 UpBlock. 512 - 512 512; Concatenate h7
=2 =2 DownBlock, 128 - 256
z2 x2 UpBlock. 512 + 512; Concatenate hg
x2 =22 x 3 Conv 256 + 512,
z2 z2 UpBlock, 512 512;
Concatenate h5
BatchNorm, Leaky ReL.U
22 z2 UpBlock, 512 - 512; Concatenate h4
z2 x2 3 x 3 Conv 512 > 1, tanh()
22 z2 UpBlock, 512 > 256; Concatenate h3
z2 z2 UpBlock, 256 + 128: Concatenate h2
x2 z2 UpBlock, 128 + 64; Concatenate Je1
Upsample 2x, x 3 Conv, 64 3
tanh()
Table 15: Architectures used for the standard generator discriminator the Pix2Pix experi- ments. Each DownBlock consists of 3 Convolution, 2x Average Pool, Batch Normalization, and Leaky ReLU activation. Each UpBlock consists of 2x nearest-neighbors upsampling, x Convolution, Batch Normalization, and Leaky ReLU activation.
27

---

Published as a conference paper at ICLR 2021
Generator
Input RGB image R256x256x3
h1 =2 p4 DownBlock, 3 - 64
h2 : p4 p4 Dook, 64 + 128
Discriminator
h3 p4
p4 DownBlock, 128 + 256
Input RGB image 256xx566
h4 p4 p4 DownBlock, 512 - 512
Input RGB image E R 256x256x3
hs p4 p4 DownBlock, 512 - 512
Concatenate and y feature-wise
hig p4 p4 DownBlock, 512 512
z2 p4 DownBlock, - 64
177 p4 p4 DownBlock, 512 - 512
p4 pd DownBlock, 64 + 128
hg : p4 p4 DownBlock, 512 >>512
p p4 DownBlock, 128 + 256
p4 p4 UpBlock, 512 - 512; Concatenate h7
p4 p4 x 3 Conv 256 + 512,
p4 p4 UpBlock, 512 - 512; Concatenate h6
BatchNorm, Leaky ReLU
p4 p4 UpBlock. 512 > 512; Concatenate h5
p4 p4 3 x Cony 512 1,
p4 p4 UpBlock, 512 - 512; Concatenate h4
p4-average pool tanh()
p4 p4 UpBlock, 512 -> 256; Concatenate h3
p4 p4 UpBlock, 256 > 128; Concatenate h2
p4 p4 UpBlock, 128 64; Concatenate h1
Upsample 2x, 3 x 3 GConv, 64 +
p4-average pool, tanh()
Table 16: Architectures used for the p4-equivari generator and discriminator in the Pix2Pix exper- iments. Each DownBlock consists of a 3 x 3 p4-convolution, 2x Average Pool, p4-Batch Normal- ization, and Leaky ReLU activation. Each UpBlock consists of 2x nearest-neighbors upsampling, x p44Connvluttion, p4-Batch Normalization, and Leaky ReLU activation.
28


Reviews:
[{"review_id": "Jcg8_SRx60u", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method for incorporating inductive symmetry priors into the network architectures of GANs. The authors propose replacing standard CNNs with group-equivariant CNNs in either the generator or discriminator or both. The method is evaluated by comparing generated images to originals using Frechet distance for a number of datasets. \n\nThe paper is well written and comprehensive. Its main contribution seems to be the adaptation of some common GAN techniques to the group-equivariant case. \n\nThe experiments are fairly convincing, although I would like to see more discussion about when to use the group convolution in the discriminator vs generator vs both, as this seems to make a difference for different datasets. I am not an expert and am unsure if Frechet distance is the right metric here, but it seems to be common at least. \n\nThe authors say they compare their method to one of the two main, state-of-the-art GAN designs. It would be interesting to see a comparison with other flavours of GAN also. Given the empirical nature of the paper, I think this would be a more convincing argument (i.e. group-convolutions can improve GANs not just BigGAN).  It would also be interesting to see how the different methods perform given different amounts of training time. \n\nIn general I like this paper, though I think the experiments could be more comprehensive (as noted above). That being said, it is not immediately clear to me that this is enough of a contribution to be accepted, since it is just applying existing methods in the group-equivariant case. As I've said though, I am not an expert in GANs, and I would be open to increasing my rating if more knowledgeable reviewers disagree on this point.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "B0a7UYSvzA8", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Short summary of the paper:\n\nThis paper proposes a new type of architecture that incorporates group-equivarient layers into the discriminator and the generator of a GAN. This group shall be a group of symmetry of the signals to generate, and in this paper, the authors focus on the group generated by translations, pi/2-rotations and reflexions. They also employ linear upsampling, and they show that with no specific ad-hoc tricks(here, class-conditional batch norm & spectral normalisation), it is possible to train competitive GANs with such priors. The experiments are conducted on datasets which have exactly two symmetries (translation or roto-translation), like rotated-MNIST, or approximatively one like Food-101, or CIFAR-10. The major contribution of this paper is experimental, showing better per-class synthesis and claiming one needs less data thanks to those priors.\n\nPros:\n- This is an interesting attempt to incorporate group structure in GANs which can only benefit to the community.\n- Various experiments are conducted which positively support the insights of the paper, measured thanks to a visual fidelity metric.\n\nCons:\n- I think the formalism could be more complete and rich. Several points are not addressed, like dealing with more complicated groups that imply an inexact covariance but are still groups of variability. Currently, I find it too elementary.\n- From the experimental points of view, I thought small data meant to use at least an order of magnitude less samples than the original dataset for training GANs. At least, it would have been nice to show a more complete plot, in order to understand the tendency is a really limited sample setting, rather than 3 data points for 33%, 66% and 100% of the data.\n- I would have appreciated to understand the methodology to pick the hyper parameters. In a supervised context, I understand that one performs a grid search on a validation set which is a subset of the training set. Here, the way to pick an architecture is unclear to me. For instance and in one word, it should be clear if modifying an architecture to make it G-convolutional friendly.\n\nSpecific remarks/questions:\n- I find quite strange to see no Mallat's citation, whereas the first works on invariance in image processing/CNNs is from his group and since at least 2010 if not older ( https://www.di.ens.fr/data/publications/papers/Eusipco2010InterConfPap.pdf ). Furthermore, I noticed some generative models based on a scattering transform ( https://arxiv.org/abs/1805.06621 / https://arxiv.org/abs/1809.06367 ), yet they do not work substantially better than other architectures whereas the scattering transform employed there leads to no loss of information. It indicates that group a framework might not be the ultimate key to generating images. Consequently, I would substantially lower the claim \"to our knowledge, we are the first to introduce group equivariance to GANs, and use geometric considerations in both generator and discriminator\".\n- This leads to a second remark: the groups of equivariance which are used correspond roughly to horizontal and vertical reflexions. They are not natural groups to consider for image variabilities (like small rotations, or small deformations) Consequently, it is natural that applying a data augmentation strategy based on those groups would \"hurt\" the performance.\n- Again, the group-a priori is quite limited: it's not a generic group which is used, but a very small subset subgroup of the rotation. This is probably due to aliasing but one can wonder how to extend this method to real group of variabilities (deformations, rotation, scaling) This should be discuss, or at least tried, even if it doesn't work.\n-  It is written that the authors tried to learn \"a transformation from G to Z^2\" that could be learned. I think the formalism here is important: it is a representation from G to L^2(Z^2) that one would like to design. I think it would be also nice to develop more this aspect in the paper or not to mention it at all.\n- The number of samples used isn't really impressive. 33% of the data isn't order of magnitude smaller than the original dataset thus I find the small data claims strange; what happens if less data is used for instance?\n- Why isn't the linear  averaging along the group in the G-equivariant architectures, and why is it only w.r.t. translations? I'm slightly confuse by the combination of the max-pool and global average pool.\n- I was wondering if the authors tried to manipulate the orbits of the generated signal: could one use the same \"displacement\" (or rotation) in the latent space to sample two rotated images from each other with the generator?\n\nSuggestions for improving the paper:\n- I believe a significant improvement could be obtain from a more careful formalism to handle complex groups of variability of images.\n- I would add slightly more elements about the experimental process, for being more reproducible.\n- I think it'd be okay to show some negative results.\n\n[Post-rebuttal] I've read the rebuttal, which answers to many points I raised. I've reflected it in my score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "6rPDMjVk7G", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Review of ICLR Paper 1928\n\n## Group Equivariant Generative Adversarial Networks\n\nThis manuscript addresses the problem of artificially generating images for which the label should be unaffected by certain symmetries and/or translations. To address this problem, the authors use the Group Equivariant Convolutional Neural Networks of Cohen \\& Welling (2016) as the generator and/or discriminator in a Generative Adversarial Network. They conduct a sequence of experiments on smaller data sets--four real and one synthetic--exhibiting varying levels of symmetry, which together suggest that Group Equivariant GANs may be more effective than traditional convolutional networks in the low data setting.\n\nMy assessment of this manuscript is that it sits right on the threshold separating acceptance and rejection. On one hand, the writing is remarkably clear of grammatical errors and the authors are evidently deeply familiar with GAN literature. On the other hand, the paper's novelty is minimal and the mathematical descriptions of some important parts of the manuscript are insufficient. Group Equivariant Networks were introduced in 2016, and in the original paper those authors demonstrated the efficacy and training efficiency of these networks. In this manuscript, the authors' contribution can be summarized as placing this architecture in a GAN and observing similar improvements, which is unsurprising and in my assessment a marginal contribution.\n\nI've summarized further concerns below.\n\n- Section 2.1, the Preliminaries section, makes the fundamental concepts extremely hard to understand. \n    - \"Equivariant\" and \"invariant\" are not defined.\n    - Please provide a citation for the plane symmetry groups $p4$ and $p4m$.\n    - The phrase \"an image is a function $f$ on a rectangular grid $\\mathbb{Z}^{2}$\" is ambiguous. Use the notation $f: \\mathbb{Z}^{2} \\to \\mathbb{R}$. However, you later mention that $f$ has $K$ channels, in which case $f: [K] \\times \\mathbb{Z} \\times \\mathbb{Z} \\to \\mathbb{R}$. Regardless, I'm confused.\n    - Which operation defines your group? You have sums and product and compositions, so this should be clarified.\n    - Define a filter $\\psi$.\n    - The $g^{-1} y = y - g$ on page 3 is confusing. How can this work when $g \\in p4$ is a rotation and not a translation?\n    - On the whole, I find this section inadequate. The Cohen paper introduces these concepts well, so I encourage you to stick closer to the mathematics introduced there.\n- \"We finally max-pool over the set of transformations to obtain the generated image $x \\in \\mathbb{Z}^{2}$\". An image is a function, not a pair of integers.\n- \"The group-equivariant discriminator receives an inputs $x \\in \\mathbb{Z}^{2}$\" Again, an image is not a pair of integers.\n- Table two is problematic for me. Your Group-Equivariant networks have less parameters, and you are operating in a self-professed limited data environment. Shouldn't I expect that the networks with less parameters exhibit less overtraining?\n- What are the details of how you conducted the ablation study? What layers and/or features did you remove?\n- I am not convinced that, in table 2, the potential for augmentation leaking means that the CNN with standard augmentation is \"inapplicable\". Why don't you just let the FID evaluation measure its performance, which should be higher if the network has captured symmetries incorrectly?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5HINmTMeXP", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe submission concerns an application of group convolutions (Cohen & Welling, 2016) to the image synthesis setting, where images are produced by the generator of a GAN. The two GAN components are augmented mainly by a straightforward replacement of \"regular\" convolutions by group convolutions, in addition to some other training tricks of the trade (gradient penalty, spectral normalization). Experiments indicate somewhat lower FID scores on both synthetic and real settings. The method is seen as useful especially for the low data regime case.\n\nReview:\nDespite the conceptual simplicity of the presented approach (\"replace convolutions by group convolutions\") I found the method itself a novel combination of existing concepts. I can also imagine that getting the such modified architecture(s) to work in practice can be quite tricky; hence the additional GAN training measures that were adopted in the training setup.\n\nOverall, however, I see novelty and impact to be fairly limited, as all insights come from empirical evaluation, which is notoriously difficult for visual synthesis applications. The authors provide both visual results as well as quantitative results mostly on the basis of FID measurements which aim to evaluate the quality of the images. Experiments seem sound and results are clearly presented. \n\nAlthough the authors claim in the discussion section that visual fidelity and sample complexity are meaningfully improved, I miss an attempt at quantitative analysis of this claim, either through algorithmic metrics or user studies. There are also no statements on the trade-off between sample fidelity, training difficult and amount of (training or inference time) compute. Since one of the strengths of the approach may be in the limited data regime, I would have liked to see stronger evidence of a major impact there; I can't quite see such a trend in the numbers of Table 1.\n\nOverall I see the work performed and results achieved by the submission as good, but a stronger verification of the main claims would make the submission even stronger for the main conference track.\n\n[Update post-rebuttal: I thank the authors for addressing some of the concerns raised by the reviewers. My stance remains, also given the outcome of e.g. the 10% experiment -- verification of the main claims remains difficult. My score already reflects that I'd be happy to see the submission accepted.]", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper use Group convolutional neural networks in both generators and discriminator of GANs, and demonstrates advantages of this approach when training with a relatively small sample size. While the novelty is limited in the work  as it simply applies G-CNN for GANs , I believe this application is interesting and  the authors have applied it to many GAN image synthesis applications (conditional generation , pix2pix) on various benchmarks, which gives  evidence of  the potential of GCNNs in generative modeling. Accept
Decision: Accept (Poster)