Alias-Free Generative Adversarial Networks

1 Introduction . The resolution and quality of images produced by generative adversarial networks ( GAN ) [ 19 ] have seen rapid improvement recently [ 27 , 11 , 29 , 30 ] . They have been used for a variety of applications , including image editing [ 42 , 47 , 37 , 20 , 34 , 3 ] , domain translation [ 62 , 32 , 53 , 36 ] , and video generation [ 49 , 15 , 21 ] . While several ways of controlling the generative process have been found [ 8 , 26 , 10 , 36 , 22 , 2 , 7 , 41 , 6 ] , the foundations of the synthesis process remain only partially understood . In the real world , details of different scale tend to transform hierarchically . For instance , moving a head causes the nose to move , which in turn moves the skin pores on it . The structure of a typical GAN generator is analogous : coarse , low-resolution features are hierarchically refined by upsampling layers , locally mixed by convolutions , and new detail is introduced through nonlinearities . We observe that despite this superficial similarity , current GAN architectures do not synthesize images in a natural hierarchical manner : the coarse features mainly control the presence of finer features , but not their precise positions . Instead , much of the fine detail appears to be fixed in pixel coordinates . This disturbing “ texture sticking ” is clearly visible in latent interpolations ( see Figure 1 and our accompanying videos on the project page https : //nvlabs.github.io/stylegan3 ) , breaking the illusion of a solid and coherent object moving in space . Our goal is an architecture that ∗This work was done during an internship at NVIDIA . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . exhibits a more natural transformation hierarchy , where the exact sub-pixel position of each feature is exclusively inherited from the underlying coarse features . It turns out that current networks can partially bypass the ideal hierarchical construction by drawing on unintentional positional references available to the intermediate layers through image borders [ 25 , 31 , 58 ] , per-pixel noise inputs [ 29 ] and positional encodings , and aliasing [ 5 , 61 ] . Aliasing , despite being a subtle and critical issue [ 38 ] , has received little attention in the GAN literature . We identify two sources for it : 1 ) faint after-images of the pixel grid resulting from non-ideal upsampling filters2 such as nearest , bilinear , or strided convolutions , and 2 ) the pointwise application of nonlinearities such as ReLU [ 52 ] or swish [ 40 ] . We find that the network has the means and motivation to amplify even the slightest amount of aliasing and combining it over multiple scales allows it to build a basis for texture motifs that are fixed in screen coordinates . This holds for all filters commonly used in deep learning [ 61 , 51 ] , and even high-quality filters used in image processing . How , then , do we eliminate the unwanted side information and thereby stop the network from using it ? While borders can be solved by simply operating on slightly larger images , aliasing is much harder . We begin by noting that aliasing is most naturally treated in the classical Shannon-Nyquist signal processing framework , and switch focus to bandlimited functions on a continuous domain that are merely represented by discrete sample grids . Now , successful elimination of all sources of positional references means that details can be generated equally well regardless of pixel coordinates , which in turn is equivalent to enforcing continuous equivariance to sub-pixel translation ( and optionally rotation ) in all layers . To achieve this , we describe a comprehensive overhaul of all signal processing aspects of the StyleGAN2 generator [ 30 ] . Our contributions include the surprising finding that current upsampling filters are simply not aggressive enough in suppressing aliasing , and that extremely high-quality filters with over 100dB attenuation are required . Further , we present a principled solution to aliasing caused by pointwise nonlinearities [ 5 ] by considering their effect in the continuous domain and appropriately low-pass filtering the results . We also show that after the overhaul , a model based on 1×1 convolutions yields a strong , rotation equivariant generator . Once aliasing is adequately suppressed to force the model to implement more natural hierarchical refinement , its mode of operation changes drastically : the emergent internal representations now include coordinate systems that allow details to be correctly attached to the underlying surfaces . This promises significant improvements to models that generate video and animation . The new StyleGAN3 generator matches StyleGAN2 in terms of FID [ 23 ] , while being slightly heavier computationally . Our implementation and pre-trained models are available at https : //github.com/NVlabs/stylegan3 Several recent works have studied the lack of translation equivariance in CNNs , mainly in the context of classification [ 25 , 31 , 58 , 5 , 33 , 61 , 12 , 63 , 51 ] . We significantly expand upon the antialiasing 2Consider nearest neighbor upsampling . If we upsample a 4×4 image to 8×8 , the original pixels will be clearly visible , allowing one to reliably distinguish between even and odd pixels . Since the same is true on all scales , this ( leaked ) information makes it possible to reconstruct even the absolute pixel coordinates . With better filters such as bilinear or bicubic , the clues get less pronounced , but are nevertheless evident for the generator . measures in this literature and show that doing so induces a fundamentally altered image generation behavior . Group-equivariant CNNs aim to generalize the efficiency benefits of translational weight sharing to , e.g. , rotation [ 16 , 57 , 55 , 54 ] and scale [ 56 ] . Our 1×1 convolutions can be seen an instance of a continuously E ( 2 ) -equivariant model [ 54 ] that remains compatible with , e.g. , channel-wise ReLU nonlinearities and modulation . Dey et al . [ 17 ] apply 90◦ rotation-and-flip equivariant CNNs [ 16 ] to GANs and show improved data efficiency . Our work is complementary , and not motivated by efficiency . Recent implicit network [ 45 , 48 , 13 ] based GANs [ 4 , 46 ] generate each pixel independently via similar 1×1 convolutions . While equivariant , these models do not help with texture sticking , as they do not use an upsampling hierarchy or implement a shallow non-antialiased one . 2 Equivariance via continuous signal interpretation . To begin our analysis of equivariance in CNNs , we shall first rethink our view of what exactly is the signal that flows through a network . Even though data may be stored as values in a pixel grid , we can not naïvely hold these values to directly represent the signal . Doing so would prevent us from considering operations as trivial as translating the contents of a feature map by half a pixel . According to the Nyquist–Shannon sampling theorem [ 44 ] , a regularly sampled signal can represent any continuous signal containing frequencies between zero and half of the sampling rate . Let us consider a two-dimensional , discretely sampled feature map Z [ x ] that consists of a regular grid of Dirac impulses of varying magnitudes , spaced 1/s units apart where s is the sampling rate . This is analogous to an infinite two-dimensional grid of values . Given Z [ x ] and s , the Whittaker–Shannon interpolation formula [ 44 ] states that the corresponding continuous representation z ( x ) is obtained by convolving the discretely sampled Dirac grid Z [ x ] with an ideal interpolation filter φs , i.e. , z ( x ) = ( φs ∗ Z ) ( x ) , where ∗ denotes continuous convolution and φs ( x ) = sinc ( sx0 ) · sinc ( sx1 ) using the signal processing convention of defining sinc ( x ) = sin ( πx ) / ( πx ) . φs has a bandlimit of s/2 along the horizontal and vertical dimensions , ensuring that the resulting continuous signal captures all frequencies that can be represented with sampling rate s. Conversion from the continuous to the discrete domain corresponds to sampling the continuous signal z ( x ) at the sampling points of Z [ x ] that we define to be offset by half the sample spacing to lie at the “ pixel centers ” , see Figure 2 , left . This can be expressed as a pointwise multiplication with a two-dimensional Dirac comb Xs ( x ) = ∑ X∈Z2 δ ( x− ( X + 12 ) /s ) . We earmark the unit square x ∈ [ 0 , 1 ] 2 in z ( x ) as our canvas for the signal of interest . In Z [ x ] there are s2 discrete samples in this region , but the above convolution with φs means that values of Z [ x ] outside the unit square also influence z ( x ) inside it . Thus storing an s× s -pixel feature map is not sufficient ; in theory , we would need to store the entire infinite Z [ x ] . As a practical solution , we store Z [ x ] as a two-dimensional array that covers a region slightly larger than the unit square ( Section 3.2 ) . Having established correspondence between bandlimited , continuous feature maps z ( x ) and discretely sampled feature maps Z [ x ] , we can shift our focus away from the usual pixel-centric view of the signal . In the remainder of this paper , we shall interpret z ( x ) as being the actual signal being operated on , and the discretely sampled feature map Z [ x ] as merely a convenient encoding for it . Discrete and continuous representation of network layers Practical neural networks operate on the discretely sampled feature maps . Consider operation F ( convolution , nonlinearity , etc . ) operating on a discrete feature map : Z ′ = F ( Z ) . The feature map has a corresponding continuous counterpart , so we also have a corresponding mapping in the continuous domain : z′ = f ( z ) . Now , an operation specified in one domain can be seen to perform a corresponding operation in the other domain : f ( z ) = φs′ ∗ F ( Xs z ) , F ( Z ) = Xs′ f ( φs ∗ Z ) , ( 1 ) where denotes pointwise multiplication and s and s′ are the input and output sampling rates . Note that in the latter case f must not introduce frequency content beyond the output bandlimit s′/2 . 2.1 Equivariant network layers . Operation f is equivariant with respect to a spatial transformation t of the 2D plane if it commutes with it in the continuous domain : t ◦ f = f ◦ t. We note that when inputs are bandlimited to s/2 , an equivariant operation must not generate frequency content above the output bandlimit of s′/2 , as otherwise no faithful discrete output representation exists . We focus on two types of equivariance in this paper : translation and rotation . In the case of rotation the spectral constraint is somewhat stricter — rotating an image corresponds to rotating the spectrum , and in order to guarantee the bandlimit in both horizontal and vertical direction , the spectrum must be limited to a disc with radius s/2 . This applies to both the initial network input as well as the bandlimiting filters used for downsampling , as will be described later . We now consider the primitive operations in a typical generator network : convolution , upsampling , downsampling , and nonlinearity . Without loss of generality , we discuss the operations acting on a single feature map : pointwise linear combination of features has no effect on the analysis . Convolution Consider a standard convolution with a discrete kernel K. We can interpret K as living in the same grid as the input feature map , with sampling rate s. The discrete-domain operation is simply Fconv ( Z ) = K ∗ Z , and we obtain the corresponding continuous operation from Eq . 1 : fconv ( z ) = φs ∗ ( K ∗ ( Xs z ) ) = K ∗ ( φs ∗ ( Xs z ) ) = K ∗ z ( 2 ) due to commutativity of convolution and the fact that discretization followed by convolution with ideal low-pass filter , both with same sampling rate s , is an identity operation , i.e. , φs ∗ ( Xs z ) = z . In other words , the convolution operates by continuously sliding the discretized kernel over the continuous representation of the feature map . This convolution introduces no new frequencies , so the bandlimit requirements for both translation and rotation equivariance are trivially fulfilled . Convolution also commutes with translation in the continuous domain , and thus the operation is equivariant to translation . For rotation equivariance , the discrete kernel K needs to be radially symmetric . We later show in Section 3.2 that trivially symmetric 1×1 convolution kernels are , despite their simplicity , a viable choice for rotation equivariant generative networks . Upsampling and downsampling Ideal upsampling does not modify the continuous representation . Its only purpose is to increase the output sampling rate ( s′ > s ) to add headroom in the spectrum where subsequent layers may introduce additional content . Translation and rotation equivariance follow directly from upsampling being an identity operation in the continuous domain . With fup ( z ) = z , the discrete operation according to Eq . 1 is Fup ( Z ) = Xs′ ( φs ∗Z ) . If we choose s′ = ns with integer n , this operation can be implemented by first interleaving Z with zeros to increase its sampling rate and then convolving it with a discretized filter Xs′ φs . In downsampling , we must low-pass filter z to remove frequencies above the output bandlimit , so that the signal can be represented faithfully in the coarser discretization . The operation in continuous domain is fdown ( z ) = ψs′ ∗ z , where an ideal low-pass filter ψs : = s2 · φs is simply the corresponding interpolation filter normalized to unit mass . The discrete counterpart is Fdown ( Z ) = Xs′ ( ψs′ ∗ ( φs ∗Z ) ) = 1/s2 ·Xs′ ( ψs′ ∗ψs ∗Z ) = ( s′/s ) 2 ·Xs′ ( φs′ ∗Z ) . The latter equality follows from ψs ∗ ψs′ = ψmin ( s , s′ ) . Similar to upsampling , downsampling by an integer fraction can be implemented with a discrete convolution followed by dropping sample points . Translation equivariance follows automatically from the commutativity of fdown ( z ) with translation , but for rotation equivariance we must replace φs′ with a radially symmetric filter with disc-shaped frequency response . The ideal such filter [ 9 ] is given by φ◦s ( x ) = jinc ( s‖x‖ ) = 2J1 ( πs‖x‖ ) / ( πs‖x‖ ) , where J1 is the first order Bessel function of the first kind . Nonlinearity Applying a pointwise nonlinearity σ in the discrete domain does not commute with fractional translation or rotation . However , in the continuous domain , any pointwise function commutes trivially with geometric transformations and is thus equivariant to translation and rotation . Fulfilling the bandlimit constraint is another question — applying , e.g. , ReLU in the continuous domain may introduce arbitrarily high frequencies that can not be represented in the output . A natural solution is to eliminate the offending high-frequency content by convolving the continuous result with the ideal low-pass filter ψs . Then , the continuous representation of the nonlinearity becomes fσ ( z ) = ψs ∗ σ ( z ) = s2 · φs ∗ σ ( z ) and the discrete counterpart is Fσ ( Z ) = s2 ·Xs ( φs ∗ σ ( φs ∗ Z ) ) ( see Figure 2 , right ) . This discrete operation can not be realized without temporarily entering the continuous representation . We approximate this by upsampling the signal , applying the nonlinearity in the higher resolution , and downsampling it afterwards . Even though the nonlinearity is still performed in the discrete domain , we have found that only a 2× temporary resolution increase is sufficient for high-quality equivariance . For rotation equivariance , we must use the radially symmetric interpolation filter φ◦s in the downsampling step , as discussed above . Note that nonlinearity is the only operation capable of generating novel frequencies in our formulation , and that we can limit the range of these novel frequencies by applying a reconstruction filter with a lower cutoff than s/2 before the final discretization operation . This gives us precise control over how much new information is introduced by each layer of a generator network ( Section 3.2 ) .

Summary: Starting from the observation that current generative models in fact do not synthesize images in a natural hierarchical manner (which they call “texture sticking”), this paper defines the problem systematically, analyze the root of it (unintentional information leaking and aliasing), and proposes a set of simple architectural solutions that are motivated from the signal processing tools.  The key component of the approach are:  - pointing out the presence of “texture sticking” phenomenon that much of the fine details of synthesized images appear to be fixed in the absolute pixel coordinates, not the relative position of the underlying objects being drawn.  - analyzing signals that flow through a network in a continuous domain and interpreting the actual, discrete features as merely a convenient sampled encoding of it - fixing the operations (up/down sampling and nonlinearity) that are the root of the aliasing by applying a proper low-pass filtering.  After applying these fixes, the generator becomes translation- and/or rotation- invariant and provides authentic 2D images as if there is actual 3D model and it is capturing pictures in different views.