Group Equivariant Generative Adversarial Networks

1 INTRODUCTION . Generative visual modeling is an area of active research , time and again finding diverse and creative applications . A prevailing approach is the generative adversarial network ( GAN ) , wherein density estimation is implicitly approximated by a min-max game between two neural networks ( Goodfellow et al. , 2014 ) . Recent GANs are capable of high-quality natural image synthesis and scale dramatically with increases in data and compute ( Brock et al. , 2018 ) . However , GANs are prone to instability due to the difficulty of achieving a local equilibrium between the two networks . Frequent failures include one or both networks diverging or the generator only capturing a few modes of the empirical distribution . Several proposed remedies include modifying training objectives ( Arjovsky et al. , 2017 ; Jolicoeur-Martineau , 2018 ) , hierarchical methods ( Karras et al. , 2017 ) , instance selection ( Sinha et al. , 2019 ; 2020 ) , latent optimization ( Wu et al. , 2019 ) , and strongly regularizing one or both networks ( Gulrajani et al. , 2017 ; Miyato et al. , 2018 ; Dieng et al. , 2019 ) , among others . In practice , one or all of the above techniques are ultimately adapted to specific use cases . Further , limits on data quantity empirically exacerbate training stability issues more often due to discriminator overfitting . Recent work on GANs for small sample sizes can be roughly divided into transfer learning approaches ( Wang et al. , 2018 ; Noguchi & Harada , 2019 ; Mo et al. , 2020 ; Zhao et al. , 2020a ) or methods which transform/augment the available training data and provide the discriminator with auxiliary tasks . For example , Chen et al . ( 2019 ) propose a multi-task discriminator which additionally predicts the degree by which an input image has been rotated , whereas Zhang et al . ( 2020 ) ; Zhao et al . ( 2020c ) incorporate consistency regularization where the discriminator is penalized towards similar activations for transformed/augmented real and fake images . However , with consistency regularization and augmentation , network capacity is spent learning equivariance to transformation as opposed to the desired task and equivariance is not guaranteed . In this work , we consider the problem of training tabula rasa on limited data which possess global and even local symmetries . We begin by noting that GANs ubiquitously use convolutional layers ∗Work started and partially done during an internship at Merck & Co. , Inc. †Work done while employed at Merck & Co. , Inc. which exploit the approximate translation invariance and equivariance of image labels and distributions , respectively . Equivariance to geometric transformations is key to understanding image representations ( Bietti & Mairal , 2019 ) . Unfortunately , other symmetries ( e.g. , rotations and reflections ) inherent to modalities such as astronomy and medical imaging where galaxies and cells can be in arbitrary orientations are not accounted for by standard convolutional layers . To this end , Cohen & Welling ( 2016 ) proposed a group-theoretic generalization of convolutional layers ( groupconvolutions ) which in addition to translation , exploit other inherent symmetries and increase the expressive capacity of a network thereby increasing its sample efficiency significantly in detection ( Winkels & Cohen , 2019 ) , classification ( Veeling et al. , 2018 ) , and segmentation ( Chidester et al. , 2019 ) . Importantly , equivariant networks outperform standard CNNs trained with augmentations from the corresponding group ( Veeling et al. , 2018 , Table 1 ) , ( Lafarge et al. , 2020a , Fig . 7 ) . See Cohen et al . ( 2019 ) ; Esteves ( 2020 ) for a formal treatment of equivariant CNNs . Equivariant features may also be constructed via scattering networks consisting of non-trainable Wavelet filters , enabling equivariance to diverse symmetries ( Mallat , 2012 ; Bruna & Mallat , 2013 ; Sifre & Mallat , 2013 ) . Generative scattering networks include Angles & Mallat ( 2018 ) where a standard convolutional decoder is optimized to reconstruct images from an embedding generated by a fixed scattering network and Oyallon et al . ( 2019 ) who show preliminary results using a standard convolutional GAN to generate scattering coefficients . We note that while both approaches are promising , they currently yield suboptimal synthesis results not comparable to modern GANs . Capsule networks ( Hinton et al. , 2011 ; Sabour et al. , 2017 ) are also equivariant and emerging work has shown that using a capsule network for the GAN discriminator ( Jaiswal et al. , 2019 ; Upadhyay & Schrater , 2018 ) improves synthesis on toy datasets . However , capsule GANs and generative scattering approaches require complex training strategies , restrictive architectural choices not compatible with recent insights in GAN training , and have not yet been shown to scale to real-world datasets . In this work , we improve the generative modeling of images with transformation invariant labels by using an inductive bias of symmetry . We replace all convolutions with group-convolutions thereby admitting a higher degree of weight sharing which enables increased visual fidelity , especially with limited-sample datasets . To our knowledge , we are the first to use group-equivariant layers in the GAN context and to use symmetry-driven considerations in both generator and discriminator architectures . Our contributions are as follows , 1 . We introduce symmetry priors via group-equivariance to generative adversarial networks . 2 . We show that recent insights in improving GAN training are fully compatible with group- equivariance with careful reformulations . 3 . We improve class-conditional image synthesis across a diversity of datasets , architectures , loss functions , and regularizations . These improvements are consistent for both symmetric images and even natural images with preferred orientation . 2 METHODS . 2.1 PRELIMINARIES . Groups and group-convolutions . A group is a set with an endowed binary function satisfying the properties of closure , associativity , identity , and invertibility . A two-dimensional symmetry group is the set of all transformations under which a geometric object is invariant with an endowed operation of composition . Given a group G and a map Φ : X → Y between two G-sets X and Y , Φ is said to be equivariant i.f.f . Φ ( g ·x ) = g ·Φ ( x ) , ∀x ∈ X , ∀g ∈ G. Colloquially , an equivariant map implies that transforming an input and applying the map yields the same result as applying the map and then transforming the output . Analogously , invariance requires that Φ ( g · x ) = Φ ( x ) , ∀x ∈ X , ∀g ∈ G. In deep networks , equivariance to a planar symmetry group can be achieved by either transforming filters ( Cohen & Welling , 2016 ) or feature maps ( Dieleman et al. , 2016 ) . Our work utilizes the plane symmetry groups p4 ( all compositions of 90-degree rotations and translations ) and p4m ( all compositions of 90-degree rotations , reflections , and translations ) ( Schattschneider , 1978 ) . These groups can be parameterized neatly following Cohen & Welling ( 2016 ) , g ( r , u , v ) = [ cos ( rπ2 ) −sin ( rπ 2 ) u sin ( rπ2 ) cos ( rπ 2 ) v 0 0 1 ] ; g′ ( m , r , u , v ) = ( −1 ) mcos ( rπ2 ) ( −1 ) m+1sin ( rπ2 ) usin ( rπ2 ) cos ( rπ2 ) v 0 0 1 where g ( r , u , v ) parameterizes p4 , g′ ( m , r , u , v ) parameterizes p4m , 0 ≤ r < 4 ( the number of 90- degree rotations ) , m ∈ { 0 , 1 } ( the number of reflections ) , and ( u , v ) ∈ Z2 ( integer translations ) . The group operation is matrix multiplication for both groups . The matrix g ( r , u , v ) rotates and translates a point ( expressed as homogeneous coordinate vector ) in pixel space via left-multiplication . Analogous intuition follows for g′ ( m , r , u , v ) . We now briefly define G-equivariant convolutions . We note that formally these are correlations and not convolutions and that the literature uses the terms interchangeably . A G-convolution between a vector-valued K-channel image f : Z2 → RK and filter ψ : Z2 → RK with f = ( f1 , f2 , . . . , fk ) and ψ = ( ψ1 , ψ2 , . . . , ψk ) can be expressed as [ f ∗ ψ ] ( g ) = ∑ y∈Z2 ∑K k=1 fk ( y ) ψk ( g −1y ) . For standard reference , if one considers G to be the translation group on Z2 , we have g−1y = y− g and recover the standard convolution . After the first layer of a G-CNN , we see that ( f ∗ ψ ) is a function on G , necessitating that filter banks also be functions on G. Subsequent G-convolutional layers are therefore defined as [ f ∗ ψ ] ( g ) = ∑ h∈G ∑K k=1 fk ( h ) ψk ( g −1h ) . Finally , for tasks where the output is an image , it is necessary to bring the domain of feature maps from G back to Z2 . We can pool the feature map for each filter over the set of transformations , corresponding to average or max pooling over the group of rotations ( or roto-reflections as appropriate ) . GAN optimization and stability . As we focus on the limited data setting where training instability is exacerbated , we briefly describe the two major stabilizing methods used in all experiments here . We regularize the discriminator by using a zero-centered gradient penalty ( GP ) on the real data as proposed by Mescheder et al . ( 2018 ) of the form , R1 : = γ2Ex∼Preal [ ‖∇D ( x ) ‖ 2 2 ] , where γ is the regularization weight , x is sampled from the real distribution Preal , and D is the discriminator . This GP has been shown to cause convergence ( in toy cases ) , alleviate catastrophic forgetting ( ThanhTung & Tran , 2018 ) , and strongly stabilize GAN training . However , empirical work has found that this GP achieves stability at the cost of worsening GAN evaluation scores ( Brock et al. , 2018 ) . A widely used technique for GAN stabilization is spectral normalization ( Miyato et al. , 2018 ) , which constrains the discriminator to be 1-Lipschitz , thereby improving gradient feedback to the generator ( Zhou et al. , 2019 ; Chu et al. , 2020 ) . With spectral normalization , each layer is rescaled as , WSN = W/σ ( W ) , where W is the weight matrix for a given layer and σ ( W ) is its spectral norm . In practice , σ ( W ) is estimated via a power iteration method as opposed to computing the full singular value decomposition during each training iteration . Finally , applying spectral normalization to both generator and discriminator empirically improves training significantly ( Zhang et al. , 2018 ) . 2.2 GROUP EQUIVARIANT GENERATIVE ADVERSARIAL NETWORKS . Here , we outline how to induce a symmetry prior into the GAN framework . Implementations are available at https : //github.com/neel-dey/equivariant-gans . The literature has developed several techniques for normalization and conditioning of the individual networks , along with unique architectural choices - we extend these developments to the equivariant setting . We start by replacing all convolutional layers with group-convolutional layers where filters and feature maps are functions on a symmetry group G. Batch normalization moments ( Ioffe & Szegedy , 2015 ) are calculated per group-feature map as opposed to spatial feature maps . Pointwise nonlinearities preserve equivariance for the groups considered here . Pre-activation residual blocks common to modern GANs are used freely as the sum of equivariant feature maps on G is also equivariant . Generator . The generator is illustrated at a high-level in Figure 2 . We use a fully connected layer to linearly project and reshape the concatenated noise vector z ∼ N ( 0 , I ) and class embedding c into spatial feature maps on Z2 . We then use spectrally-normalized group-convolutions , interspersed with pointwise-nonlinearities , and nearest-neighbours upsampling to increase spatial extent . We use upsampling followed by group-convolutions instead of transposed group-convolutions to reduce checkerboard artefacts ( Odena et al. , 2016 ) . We further use a novel group-equivariant classconditional batch normalization layer ( described below ) to normalize and class-condition image generation while also projecting the latent vector z to each level of the group-convolutional hierarchy . We finally max-pool over the set of transformations to obtain the generated image x. Discriminator . The group-equivariant discriminator receives an input x , which it maps to a scalar indicating whether it is real or fake . We do this via spectrally normalized group-convolutions , pointwise-nonlinearities , and spatial-pooling layers to decrease spatial extent . After the final groupconvolutional layer , we pool over the group and use global average pooling to obtain an invariant representation at the output . Finally , we condition the discriminator output via the projection method proposed by Miyato & Koyama ( 2018 ) . Importantly , the equivariance of group-convolutions depends on the convolutional stride . Strided convolutions were commonly used for downsampling in early GANs ( Radford et al. , 2015 ) . However , stride values must be adjusted to the dataset to preserve equivariance , which makes comparisons to equivalent non-equivariant GAN architectures difficult . We therefore use pooling layers over the plane ( commonly used in recent GANs ) to downsample in all settings to preserve equivariance and enable a fair comparison . Spectral Normalization . As the singular values of a matrix are invariant under compositions of 90- degree rotations , transpositions , and reflections - spectral normalization on a group-weight matrix preserves equivariance and we use it freely . Class-conditional Batch Normalization . Conditional batch normalization ( Perez et al. , 2018 ) replaces the scale and shift of features with an affine transformation learned from the class label ( and optionally from the latent vector as well ( Brock et al. , 2018 ) ) via linear dense layers , and is widely used in generative networks . We propose a group-equivariance preserving conditional normalization by learning the affine transformation parameters per group-feature map , rather than each spatial feature . As we use fewer group-filters than equivalent non-equivariant GANs , we use fewer dense parameters to learn conditional scales and shifts .

Summary: The submission concerns an application of group convolutions (Cohen & Welling, 2016) to the image synthesis setting, where images are produced by the generator of a GAN. The two GAN components are augmented mainly by a straightforward replacement of "regular" convolutions by group convolutions, in addition to some other training tricks of the trade (gradient penalty, spectral normalization). Experiments indicate somewhat lower FID scores on both synthetic and real settings. The method is seen as useful especially for the low data regime case.