Alias-Free Generative Adversarial Networks
Authors: Tero Karras
Miika Aittala
Samuli Laine
Erik Härkönen
Janne Hellsten
Jaakko Lehtinen
Timo Aila
Venue: NeurIPS 2021 Oral (2021)
Keywords: GAN
CNN
signal processing
aliasing
image synthesis

Abstract: We observe that despite their hierarchical convolutional nature, the synthesis process of typical generative adversarial networks depends on absolute pixel coordinates in an unhealthy manner. This manifests itself as, e.g., detail appearing to be glued to image coordinates instead of the surfaces of depicted objects. We trace the root cause to careless signal processing that causes aliasing in the generator network. Interpreting all signals in the network as continuous, we derive generally applicable, small architectural changes that guarantee that unwanted information cannot leak into the hierarchical synthesis process. The resulting networks match the FID of StyleGAN2 but differ dramatically in their internal representations, and they are fully equivariant to translation and rotation even at subpixel scales. Our results pave the way for generative models better suited for video and animation.

Alias-Free Generative Adversarial Networks
Tero Karras
Miika Aittala NVIDIA mait com
Samuli Laine
NVIDIA ckarras@nvdda..com com
NVIDIA
slaine@nvidia.com
Erik Härkönen' Aalto University and NVIDIA erik. harkonen@aalto..ff
Janne Hellsten
NVIDIA
jhellsten@nvidia.ccmm
Jaakko Lehtinen NVIDIA and Aalto University j llehtinen@nvidia. com
Timo Aila
NVIDIA
taila@nvidia.com
Abstract
We observe that despite their hierarchical convolutional nature, the synthesis process of typical generative adversarial networks depends on absolute pixel coor- dinates in an unhealthy manner. This manifests itself as, e-g.. detail appearing be glued to image coordinates instead of the surfaces of depicted objects. We trace the root cause to careless signal processing that causes aliasing in the generator network. Interpreting all signals in the network as continuous, we derive generally applicable, small architectural changes that guarantee that unwanted information cannot leak into the hierarchical synthesis process. The resulting networks match the FID of StyleGAN2 but differ dramatically in their internal representations, and they are fully equivariant to translation and rotation even at subpixel scales. Our results pave the way for generative models better suited for video and animation.
Introduction
The resolution and quality of images produced by generative adversarial networks (GAN) 19
have seen rapid improvement recently [27
I1
29
30 They have been used for variety of
applications, including image editing [2
47
37
20
3 domain translation 662 32 53 36]. and
video generation 449
15 21
While several ways of controlling the generative process have been 6), the foundations of the synthesis process remain only partially
found [8] 26 10 understood.
36 22
In the real world, details of different scale tend to transform hierarchically. For instance, moving head causes the to move, which in turn moves the skin pores it. The structure of typical GAN generator analogous: coarse, low-resolution features are hierarchically refined by upsampling layers, locally mixed by convolutions, and detail is introduced through nonlinearities. We observe that despite this superficial similarity, current GAN architectures do not synthesize images in a natural hierarchical manner: the coarse features mainly control the presence of finer features, but not their precise positions. Instead, much of the fine detail appears to be fixed in pixel coordinates. This disturbing "texture sticking" is clearly visible in latent interpolations (see Figure I and our accompanying videos on the project page sttps://nniiie breaking the illusion of solid and coherent object moving in space. Our goal is an architecture that
gi thub. 1o/ styleeanl)
"This work was done during an internship at NVIDIA.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

StyleGAN2
Ours
StyleGAN2 latent inter polaticl
Ours Intent interpolation
Averaged Central
Figure 1: Examples of "texture sticking". Left: The average of images generated from small neighborhood around central latent (top row). The intended result is uniformly blurry because all details should move together. However, with StyleGAN2 many details (c.g., fur) stick to the same pixel coordinates, showing unwanted sharpness. Right: From a latent space interpolation (top row), we extract short vertical segment of pixels from each generated image and stack them horizontally (bottom). The desired result is hairs moving in animation, creating a time-varying field. With StyleGAN2 the hairs mostly stick to the same coordinates, creating horizontal streaks instead.
exhibits a more natural transformation hierarchy, where the exact sub-pixel position of each feature is exclusively inherited from the underlying coarse features.
turns out that current networks can partially bypass the ideal hierarchical construction by drawing on unintentional positional references available to the intermediate layers through image borders
25 31 58 per-pixel noise inputs [29] and positional encodings, and aliasing [1]. Aliasing, despite being a subtle and critical issue [38. has received little attention in the GAN literature. We identify two sources for it: 1) faint after-images of the pixel grid resulting from non-ideal upsampling filterSsuch such as nearest, bilinear, strided convolutions, and 2) the pointwise application of nonlinearities such as ReLU [52] or swish [40]. We find that the network has the means and motivation to amplify even the slightest amount of aliasing and combining it over multiple scales allows it to build a basis for texture motifs that are fixed in screen coordinates. This holds for all filters commonly used in deep learning [6 51. and even high-quality filters used in image processing.
How, then, do we eliminate the unwanted side information and thereby stop the network from using it? While borders can be solved by simply operating on slightly larger images, aliasing is much harder. We begin by noting that aliasing is most naturally treated in the classical Shannon-Nyqute signal processing framework, and switch focus to bandlimited functions on continuous domain that are merely represented by discrete sample grids. Now, successful elimination of all sources of positional references means that details can be generated equally well regardless of pixel coordinates, which in turn is equivalent to enforcing continuous equivariance to sub-pixel translation (and optionally rotation) in all layers. To achieve this, we describe a comprehensive overhaul of all signal processing aspects of the StyleGAN2 generator [30]. Our contributions include the surprising finding that current upsampling filters simply not aggressive enough in suppressing aliasing, and that extremely high-quality filters with over 100dB attenuation are required. Further, we present principled solution to aliasing caused by pointwise nonlinearities (5) by considering their effect in the continuous domain and appropriately low-pass filtering the results. We also show that after the overhaul, a model based on Ixl convolutions yields strong, rotation equivariant generator.
Once aliasing is adequately suppressed to force the model to implement more natural hierarchical refinement, its mode of operation changes drastically: the emergent internal representations now include coordinate systems that allow details to be correctly attached to the underlying surfaces. This promises significant improvements to models that generate video and animation. The new StyleGAN3 generator matches StyleGAN2 in terms of FID (3), while being slightly heavier computationally. Our implementation and pre-trained models are available at htttss://giiie
Several recent works have studied the lack of translation equivariance in CNNs, mainly in the context
of classification 255 31
58
5
33
61
63 51. We significantly expand upon the antialiasing
*Consider nearest neighbor upsampling. If we upsample a 4x image to 8x8, the original pixels will be clearly visible, allowing one to reliably distinguish between even and odd pixels. Since the same is true on all scales, this (leaked) information makes it possible to reconstruct even the absolute pixel coordinates. With better filters such as bilinear or bicubic, the clues get less pronounced, but nevertheless evident for the generator.

---

X 1s
2
a(2)
No faithful discretization
III.
Z
Z
Z'
Figure 2: Left: Discrete representation Z and continuous representation related to each other via convolution with ideal interpolation filter &, and pointwise multiplication with Dirac comb III.. Right: Nonlinearity o, ReLU this example, may produce arbitrarily high frequencies in the coniinous-domain (2). Low-pass filtering via o, is necessary to ensure that Z' captures the result.
measures in this literature and show that doing so induces a fundamentally altered image generation behavior. Group-equivarian CNNs aim to generalize the efficiency benefits of translational weight sharing to, e.g., rotation [7] 55 54 and scale [56]. Our I x 1 convolutions can be seen an instance of continuously - model [54] that remains compatible with, e.g., channel-wise ReLU nonlincarities and modulation. Dey et al. [17 apply 90 rotation rotation-and-fit and-flip equivariant CNNs [[6] to GANs and show improved data efficiency. Our work complementary, and not motivated by efficiency. Recent implicit network 45 48 [3] based GANs 4 46 generate each pixel independently via similar x convolutions. While equivariant, these models do not help with texture sticking, as they do not use upsampling hierarchy or implement shallow non-antialiased one.
2 Equivariance via continuous signal interpretation
To begin our analysis of equivariance in CNNs, we shall first rethink our view of what exactly is the signal that flows through network. Even though data may be stored as values in pixel grid, cannot naïvely hold these values to directly represent the signal. Doing so would prevent us from considering operations as trivial as translating the contents of a feature map by half a pixel.
a regularly sampled signal can represent any continuous signal containing frequencies between zero and half of the sampling rate. Let us consider a two-dimensional, discretely sampled feature map Z[z) that consists of a regular grid of Dirac impulses of varying magnitudes, spaced 1 /ss units apart where the sampling rate. This is analogous to an infinite two-dimensional grid of values.
According to the Nyquist-Shannen sampling theorem
Given Z[æ] and s, the Whittaker Shannno interpolation formula [4] states that the corresponding continuous representation =<x) obtained by convolving the discretely sampled Dirac grid 2[x) with an ideal interpolation filter Φ.. i.e., x(x) (ø. Z) (z), where * denotes continuous convolution and @
sinc(sxo) sinc(sz)) (sæ1) using the signal processing convention of defining sinc(2) sin( (Tx) /TT).. &. has a bandlimit of s/2 along the horizontal and vertical dimensions, ensuring that the resulting continuous signal captures all frequencies that can be represented with sampling rate s.
Conversion from the continuous to the discrete domain corresponds to sampling the continuous signal 2(x)) at the sampling points of Z[æ] that we define be offset by half the sample spacing to lie the "pixel centers". see Figure 2 left. This can be expressed as pointwise multiplication with a two-dimensional Dirac comb II (xr) Exezz 6(x (X })/s).
We carmark the unit square [0, 1 in z(x)
canvas for the signal of interest. In Z[ø] there are s² discrete samples in this region, but the above convolution with Φ. means that values of Z [xx outside the unit square also influence 2(x) inside it. Thus storing
-pixel feature map is not sufficient; in theory, we would need to store the entire infinite Z [=] As practical solution, we store Z [x]] as two-dimensional array that covers region slightly larger than the unit square (Section 3.2).
Having established correspondence between bandlimited, continuous feature maps =<x) and discretely sampled feature maps Z(x:). we can shift our focus away from the usual pixel-centric view of the signal. In the remainder this paper, we shall interpret =<x) as being the actual signal being operated and the discretely sampled feature map Z [x) as merely convenient encoding for it.
3

---

Discrete and continuous representation of network layers Practical neural networks operate on the discretely sampled feature maps. Consider operation F (convolution, nonlinearity, etc.) operating on a discrete feature map: 7' = (Z). The feature map has corresponding continuous counterpart, so we also have corresponding mapping in the continuous domain: = f()). Now, an operation specified in one domain can be seen to perform a corresponding operation in the other domain:
f(z) = o2' * F(LLI, C 2),
F(Z) = _ @ f(Ø. 2),
(1)
where @ denotes pointwise multiplication and and s' are the input and output sampling rates. Note that in the latter case must not introduce frequency content beyond the output bandlimit s' //.
2.1 Equivariant network layers
Operation f is equivariant with respect to a spatial transformation t of the 2D plane if it commutes with it in the continuous domain: o f t. We note that when inputs are bandlimited to s/2. an equivariant operation must not generate frequency content above the output bandlimit of 6' /2, as otherwise no faithful discrete output representation exists.
We focus on two types of equivariance in this paper: translation and rotation. In the case of rotation the spectral constraint is somewhat stricter rotating image corresponds to rotating the spectrum, and in order to guarantee the bandlimit in both horizontal and vertical direction, the spectrum must be limited to disc with radius s/2. This applies to both the initial network input well as the bandlimiting filters used for downsampling, as will be descrihed later.
We now consider the primitive operations in typical generator network: convolution, upsampling, downsampling, and nonlinearity. Without loss of generality, we discuss the operations acting on a single feature map: pointwise linear combination of features has no effect on the analysis.
Convolution Consider standard convolution with a discrete kernel K. We can interpret K as living in the same grid as the imput feature map, with sampling rate s. The discrete-domait operation simply Foonr (Z) K Z, and we obtain the corresponding continuous operation from Eq. !
f (2) = th (K (III, @ =)) K (ø, (III, =)) K
due to commutativity of convolution and the fact that discretization followed by convolution with ideal low-pass filter, both with same sampling rate s, an identity operation, i.e., Φ. * (III, C 2) 2. In other words, the convolution operates by continuously sliding the discretized kernel over the continuous representation of the feature map. This convolution introduces no new frequencies, so the bandlimit requirements for both translation and rotation equivariance are trivially fulfilled.
Convolution also commutes with translation in the continuous domain, and thus the operation is equivariant to translation. For rotation equivariance, the discrete kernel K needs to be radially symmetric. We later show in Section 3.2 trivially symmetric 1x1 convolution kernels are, despite their simplicity, a viable choice for rotation equivariant generative networks.
Upsampling and downsampling Ideal upsampling does not modify the continuous representation. Its only purpose is to increase the output sampling rate (s' > s) to add headroom in the spectrum where subsequent layers may introduce additional content. Translation and rotation equivariance follow directly from upsampling being an identity operation in the continuous domain. With fup (z) = z, the discrete operation according to Eq. is Fup (2) Ill o O (d, * Z). If we choose s' = nss with integer n, this operation can be implemented by first interleaving with zeros to increase its sampling rate and then convolving it with discretized filter III, C Φs.
In downsampling, we must low-pass filter z to remove frequencies above the output bandlimit, so that the signal can be represented faithfully in the coarser discretization. The operation in continuous domain is (=) vb.- * z, where an ideal low-pass filter vA, == s². oe is simply the corresponding interpolation filter normalized to unit mass. The discrete counterpart is Fdown (2)
I (w.r* (Ø. 2)) = 1/s².. 1/I3.IIII// (1b2/ t/. Z) = (s'/s)2. ////s/.III,, follows from 2/. 3/. "'") Similar to upsampling, downsampling by an integer fraction can be implemented with a discrete convolution followed by dropping sample points. Translation equivariance follows automatically from the commutativity of faown(2) with translation, but for rotation equivariance we must replace the with a radially symmetric filter with disc-shaped frequency response. The ideal such filter [9] is given by (x()) = jino(sslaan = 2J ( ( [|==ll)/////>>>// (T)(/r=||>>)) where J1 is the first order Bessel function of the first kind.
III, .f (Ø., * Z). The latter equality

---

Configuration
FID
BQ-TT BQ-RT
Parameter
RQ-T1
FQ-RT
Time
Viem.
StyleGAN2
5.14
Filter size n 4
4.72
57.49
39.70
0.84x
0.99x
Fourier features
4.79
16.23
10.81
size
4.50
40.48
1.00x
C
No noise inputs
4.54
15.81
10.84
Filter size 8
4,66
65,57
42.09
1.18x
1.01x
Simplified generator
5.21
19.47
10.41
Upsampling i
4.38
39.96
36.42
0.65
0.87x
Boundaries & upsampling
6.02
24,62
10.97
Upsampling mn
4.50
66.65
40.43
1.00x
1.00x
F Filtered nonlincarities
6.35
30,60
10.81
Upsampling m
4.57
74.21
40.97
2,31x
Non-critical sampling
Transformed Fourier Teatures
4.64
4.78
45.20
10.61
Stophand fie 2
4.62
51.10
29. 14
0.86
0,90
T + Flexible layers (SN)
4.62
63.01
13.12
Stopband fio 2
4.50
66.65
40.48
1.00x
1.00x
Rotation equiv. (S1yleGAN3-R)
4.50
66.65
40.48
Stopband 2
4.68
73.13
41.63
1.36x
1.25x
Figure 3: Results for FFHQ-U (unaligned FFHQ) at 256 Left: Training contigurations. FID is computed between 50k generated images and all training images [8]; lower is better. EQ-T and EQ-R are our equivariance metries in decibels (dB); higher is better. Right: Parameter ablations using our final configuration (R) for the filter's support, magnification around nonlinearities, and the minimum stopband frequency at the first layer. iidiccates our default choices.
Nonlinearity Applying pointwise nonlinearity in the discrete domain does not commute with fractional translation or rotation. However, in the continuous domain, any pointwise function commutes trivially with geometric transformations and is thus equivariant translation and rotation. Fulfilling the bandlimit constraint is another question - applying, e.g., ReLU in the continuous domain may introduce arbitrarily high frequencies that cannot be represented in the output.
natural solution is to eliminate the offending high-frequency content by convolving the continuous result with the ideal low-pass filter v),- Then, the continuous representation of the nonlinearity
becomes (z) = the * (=) s² o. * (=) and the discrete counterpart is F, Fr(2) (2) = s² III, O (ó, ( 2)) (see Figure 2 right). This discrete operation cannot be realized without temporarily entering the continuous representation. We approximate this by upsampling the signal, applying the nonlinearity in the higher resolution, and downsampling afterwards. Even though the nonlinearity is still performed in the discrete domain, we have found that only 2x temporary resolution increase is sufficient for high-quality equivariance. For rotation equivariance, we must use the radially symmetric interpolation filter of in the downsampling step, as discussed above.
Note that nonlinearity is the only operation capable of generating novel frequencies in formulation, and that we can limit the range of these novel frequencies by applying a reconstruction filter with a lower cutoff than s/2 before the final discretization operation. This gives us precise control over how much new information is introduced by each layer of a generator network (Section 3.2
3 Practical application to generator network
We will now apply the theoretical ideas from the previous section in practice, by converting the well-established StyleGAN2 [30] generator to be fully equivariant to translation and rotation. We will introduce the necessary changes step-by-step, evaluating their impact in Figure 3 The discriminator remains unchanged in experiments.
The StyleGAN2 generator consists of two parts. First, a mapping network transforms an initial, normally distributed latent to an intermediate latent code w ~ W. Then, a synthesis network G starts from learned 4x4x512 constant Zo and applies sequence of N layers - consisting of convolutions, nonlinearities, upsampling, and per-pixel noise-tt to produce an output image N = G( w). The intermediate latent code w controls the modulation of the convolution kernels in G. The layers follow rigid 2x upsampling schedule, where two layers are executed at each resolution and the number of fcature maps is halved after each upsampling. Additionally, StyleGAN2 employs skip connections, mixing regularization (29]. and path length regularization.
Our goal is to make every layer of G equivariant w.r.t. the continuous signal, so that all finer details transform together with the coarser features of a local neighborhood. If this succeeds, the entire network becomes similarly equivariant.
other words, we aim to make the continuous operation g of the synthesis network equivariant w.r.t. transformations t (translations and rotations) applied on the continuous input 2p: g(t[=n]: w) = tgg(2n =n; w)]. To evaluate the impact of various architectural changes and practical approximations, we need way measure how well the network implements the equivariances. For translation equivariance, we report the peak signal-to-nois ratio (PSNR) in decibels (dB) between two sets of images, obtained by translating the input and output of the
5

---

synthesis network by a random amount, resembling the definition by Zhang [61]
(3) Each pair of images, corresponding to a different random choice of w, is sampled at integer pixel locations p within their mutually valid region V. Color channels c are processed independently, and the intended dynamic range of generated images -1... ... +1 gives max 2. Operator t, implements spatial translation with 2D offset v; here drawn from distribution X of integer offsets. We define an analogous metric FQ-R for rotations, with the rotation angles drawn from (00,, 360%). Appendix E in the Supplement gives implementation details and our accompanying videos highlight the practical
EQ-T = 10 -o010 (faar/ F..ar//ww..a.xx. p--V.ec
(got>[2o): w)((p) - t (g(z w) w)lo()))
relevance of different dB values.
3.1 Fourier features and baseline simplifications (configs B-D)
To facilitate exact continuous translation and rotation of the input 2o. we replace the learned input constant in StyleGAN2 with Fourier features [48] (88., which also has the advantage of naturally defining a spatially infinite map. We sample the frequencies uniformly within the circular frequency band fo = 2. matching the original 4x4 input resolution, and keep them fixed over the course of training. This change (configs A and B in Figure 3 left) slightly improves FID and, crucially, allows us compute the equivariance metries without having to approximate the operator t. This baseline architecture is far from being equivariant; our accompanying videos show that the output images deteriorate drastically when the input features are translated rotated from their original position.
Next, we remove the per-pixel noise inputs because they are strongly at odds with our goal of natural transformation hierarchy, i.e., that the exact sub-pixel position of each feature is exclusively inherited from the underlying coarse features. While this change (config C) is approximately FID--eutral, fails to improve the equivariance metrics when considered in isolation.
To further simplify the setup, we decrease the mapping network depth as recommended by Kar- ras et al. [28] and disable mixing regularization and path length regularization [30]. Finally, we also eliminate the output skip connections. We hypothesize that their benefit is mostly related to gradient magnitude dynamics during training and address the underlying issue more directly using a simple normalization before each convolution. We track the exponential moving average 2 = R[xx') over all pixels and feature maps during training, and divide the feature maps by v In practice, we bake the division into the convolution weights to improve efficiency. These changes (config D) bring FID back to the level of original StyleGAN2, while leading to a slight improvement in translation equivariance.
3.2 Step-hy-step redesign motivated by continuous interpretation
Boundaries and upsampling (config F) Our theory assumes infinite spatial extent for the feature maps, which we approximate by maintaining a fixed-size margin around the target canvas, cropping to this extended canvas after each layer. This explicit extension is necessary as border padding is
known to leak absolute image coordinates into the internal representations (25] 31 In practice, we have found a 10-pixel margin to be enough; further increase has no noticeable effect on the results.
Motivated by our theoretical model, we replace the bilinear 2x upsampling filter with a better approximation of the ideal low-pass filter. We use windowed sinc filter with a relatively large Kaiser window (35 of size n 6, meaning that each output pixel is affected by input pixels in upsampling and each input pixel affects 6 output pixels in downsampling. Kaiser window is particularly good choice for our purposes, because it offers explicit control over the transition band and attenuation (Figure 4p. In the remainder of this section, we specify the transition band explicitly and compute the remaining parameters using Kaiser's original formulas (Appendix C). For now, we choose to employ critical sampling and the filter cutoff fe = s/2. i.e., exactly at the bandlimit, and transition band half-width fn = (v2- (s/2). Recall that sampling rate s equals the width of the canvas in pixels, given our definitions in Section
The improved handling of boundaries and upsampling (config E) leads to better translation equiv- ariance. However, FID is compromised by 16%, probably because we started to constrain what the feature maps can contain. In a further ablation (Figure 3 right), smaller resampling filters (n = 4)
hurt translation equivariance, while larger filters (n 8) mainly increase training time.
Filtered nonlinearities (config F) Our theoretical treatment of nonlinearities calls for wrapping each leaky ReLU (or any other commonly used non-linearity) between mx upsampling and mx

---

Inspulse r respouse
Latent
Lug; frequency
Mapping
Fourier feat.
13
3
10
Cony Jel
Demod
Conv
13
Frequency respunse
Half-ardth &
Ureample 2x dx
14
Custom CUDA
Lesky ReLU
Crnot 1.
kormel
Downsample 2+
Pasiban Trunstion bandw Siopband
14
os
is
14
15
Gaia (dB)
Calors
L10
1x
Attennation
Fixed
L11
ToRGB
Lzemal
Affire
L13
EMA
La
Mis stuphand J.
Cunoff f.
Ripple
CUDA
ToRGiB
Sampling nice a
Suphard J. JA
08
K2
M
65
19 16
Li
(a) Filter design concepts
(b) Our alias-free StyleGAN3 generator architecture
(c) Flexible layers
Figure 4: (a) ID example of 2x upsampling filter with 6, 22 fc 1, and fn 0.4 (bluc). Setting fn- = 0.6 makes the transition band wider (green), which reduces the unwanted stophand ripple and thus leads to stronger attenuation. (b) Our alias-free generator, corresponding to configs T and R in Figure 3] The main datapath consists of Fourier features and normalization (Section B.I.. modulated convolutions [30,. and filtered nonlinearities (Section (3.). (c) Flexible layer specifications (config T) with N = 14 and SN = 1024. Cutoff fe (blue) and minimum acceptable stopband frequency St (orange) obey geometric progression over the layers; sampling rate (red) and actual stopband fa + fn. (green) are computed according to our design constraints.
downsampling, for some magnification factor m. We further note that the order of upsampling and convolution can be switched by virtue of the signal being bandlimited, allowing us to fuse the regular 2x upsampling and subsequent mx upsampling related to the nonlinearity into a single 2mx upsampling. In practice, we find m = 2 to be sufficient (Figure 3 right), again improving EQ-T (config F). Implementing the upsample- Lite--LReUU-doonnaee primitives available in current deep learning frameworks (1 [9], and thus we implement a custom CUDA kernel (Appendix D) that combines these operations 4p) leading to 10x faster training and considerable memory savings.
sequence is not efficient using the
Non-critical sampling (config G) The critical sampling scheme- where filter cutoff is set exactly at the bandlimit- ii is ideal for many image processing applications as it strikes a good balance between antialiasing and the retention of high-frequency detail [50]. However, our goals are markedly different because aliasing is highly detrimental for the equivariance of the generator. While high- frequency detail is important in the output image and thus in the highest-resolution layers, it less important in the earlier ones given that their exact resolutions are somewhat arbitrary to begin with.
To suppress aliasing, we can simply lower the cutoff frequency to fe s/2 - ʃn. which ensures that all alias frequencies (above s/2) are in the stopband 3) For example, lowering the cutoff of the blue filter in Figure would move its frequency response left so that the the worst-case attenuation of alias frequencies improves from 6 dB to 40 dB. This oversampling can be seen as a computational cost of better antialiasing, as we now use the same number of samples to express before. In practice, we choose to lower fe on all layers except the highest-resolution ones, because in the end the generator must be able to produce crisp images to match the training data. As the signals now contain less spatial information, we modify the heuristic used for determining the number of feature maps to be inversely proportional to fe instead of the sampling rate s. These changes (config G) further improve translation equivariance and push FID below the original StyleGAN2.
flower-aryying signal than
Transformed Fourier features (config H) Equivariant generator layers are well suited for modeling unaligned and arbitrarily oriented datasets, because any geometric transformation introduced the intermediate features 2; will directly carry over to the final image øn_ Due to the limited capability of the layers themselves to introduce global transformations however, the input features 2g play crucial role in defining the global orientation of 2x. To let the orientation vary on a per-image basis,
'Here, fo and fü correspond to the output (downsampling) filter of each layer. The input (upsampling) filters are based the properties of the incoming signal, i.e., the output filter parameters of the previous layer.
7

---

the generator should have the ability to transform 2u based on w. This motivates us to introduce a learned affine layer that oulputs global translation and rotation parameters for the input Fourier features (FigureF and Appendix F). The layer is initialized to perform an identity transformation, but learns to use the mechanism over time when beneficial; in config H this improves the FID slightly.
Flexible layer specifications (config T) Our changes have improved the equivariance quality considerably, but some visible artifacts still remain as our accompanying videos demonstrate. On closer inspection, it turns out that the attenuation of our filters (as defined for config G) is still insufficient for the lowest-resolution layers. These layers tend to have rich frequency content near their bandlimit, which calls for extremely strong attenuation to completely eliminate aliasing.
So far, we have used the rigid sampling rate progression from StyleGAN2, coupled with simplistic choices for filter cutoff fc and half-widtt fh. but this need not be the case; we are free specialize these parameters on a per-layer basis. In particular, we would like fr. to be high in the lowest- resolution layers to maximize attenuation in the stopband, but low in the highest-resolution layers to allow matching high-frequency details of the training data.
Figure 4C illustrates an example progression of filter parameters in a 14-layer generator with two critically sampled full-resolution layers at the end. The cutoff frequency grows geometrically from fe 2 in the first layer to fc SN /2 in the first critically sampled layer. We choose the minimum acceptable stopband frequency to start at fro = 22.1, and grows geometrically but slower than the cutoff frequency. In our tests, the stopband target at the last layer is ft fe 20.3 but the progression is halted at the first critically sampled layer. Next, we set the sampling rate s for each layer so that it accommodates frequencies up to fi, rounding up the next power of two without exceeding the output resolution. Finally, to maximize the attenuation of aliasing frequencies, we set the transition band half-width to fn- - max(s/2, ft) Sce i.e., making it as wide as possible within the limits of the sampling rate, but at least wide enough to reach ft- The resulting improvement depends on how much slack is left between fa and s/2; as an extreme example, the first layer stopband attenuation improves from 42 dB to 480 dB using this scheme.
The new layer specifications again improve translation equivariance (config T), eliminating the remaining artifacts. A further ablation (Figure right) shows that fr. provides an effective way to trade training speed for equivariance quality. Note that the number of layers is now free parameter that does not directly depend on the output resolution. In fact, we have found that a fixed choice of N works consistently across multiple output resolutions and makes other hyperparameters such as learning rate behave more predictably. We use N = 14 in the remainder of this paper.
Rotation equivariance (config R) We obtain a rotation equivariant version of the network with two changes. First, we replace the x convolutions with 1x 1 on all layers and compensate for the reduced capacity by doubling the number of feature maps. Only the upsampling and downsampling operations spread information between pixels in this config. Second, we replace the sinc- downsampling filter with radially symmetric jinc-hased one that we construct using the same Kaiser scheme (Appendix C). We do this for all layers except the two critically sampled ones, where it is important to match the potentially non-radial spectrum of the training data. These changes (config R) improve EQ-R without harming FID, even though each layer has 56% fewer trainable parameters.
We also employ an additional stabilization trick in this configuration. Early on in the training, we blur all images the discriminator sees using a Gaussian filter. We start with a = 10 pixels, which we ramp to zero over the first 200k images. This prevents the discriminator from focusing too heavily on high frequencies early on. Without this trick, config R is prone to early collapses because the generator sometimes learns to produce high frequencies with small delay, trivializing the discriminator's 's task.
4 Results
Figure gives results for six datasets using StyleGAN2 30 as well as our alias-free StyleGAN3-T and StyleGAN3-R generators. In addition to the standard FFHQ [29] and METFACES
28 we created unaligned versions of them. We also created a properly resampled version of AFHQ
14 and
collected new BEACHES dataset. Appendix B describes the datasets in detail. The results show that our FID remains competitive with StyleGAN2. StyleGAN3-T and StyleGAN3-R perform equally well in terms of FID, and both show a very high level of translation equivariance. As expected, only the latter provides rotation equivariance. In FFHQ (1024x 1024) the three generators had 30.0M, 22.3M and 15.8M parameters, while the training times were 1106, 1576 (+42%) and 2248
8

---

Dataset
Config
FID
EQ-T1
BQ-RT
Ablation
Translation
Rotation eq.
FFRQ-U
StykGAN2
3.79
15.89
10.79
FID4
EQ-TT
FID L
EQ-TT
EQ-R1
nastiing 1034
StyleGAN3-T [ours)
3.67
61.69
13.95
Main configuration
4.62
63.01
4.50
66.65
40.48
souch
SLyeGANN-R (ouns)
3,66
64.78
47.64
With mixing reg.
4.60
63.48
4.67
63.59
40.90
FFHQ
StyleGAN2
2.70
13.58
10.22
With noise inpuls
4,96
24.46
5.79
26.71
26.80
i0000ing 1634
StyleGAN3-T (ours)
2.79
61.21
13.82
Without flexible layers
4.64
45.20
4.65
44.74
22.52
Trin from scrach
StyleGAN-R (ours)
64.76
Fixed Fourict features
5.93
64.57
6.48
66.20
41.77
METFACES.UI
StyleGAA2
18.98
18.77
13.19
With length reg.
5.00
68.36
71.64
42.18
ing I0M4
StykGANN-T (ours)
18.75
64.11
16.63
0.5% capacity
7.43
63.14
6.52
63.08
39,89
ADS, Bon FFRQ-U
StyleGAN--R
18.75
66.34
48.57
1.0x capacity
4,62
63.01
4,50
66.65
401.48
MESTFACES
StyleGAN2
15.22
16.39
12.89
2,0% capacity
3.80
66.61
4.18
70.06
42.51
ing. lin
StyleGANN-- [ours)
15.11
65.23
16.82
Kaiser filter, =
4.62
63,01
4.50
66.65
40,448
ADV, Sou FFRQ
SlylGGAN-R (ours)
15.33
64.86
46,81
Lanczos filter a =
4.69
51.93
4.44
57.70
25.25
APHQV2
StyleGAN2
13.83
11.50
Gaussian filter, 0.4
5.91
56.89
5.73
59.53
39,43
15805 5125
StyleGAN3-T [ours)
4,04
60.15
13.51
StykGAN--R (ours)
4,40
64.89
40.34
G-CNN comparison
F
EQ-T1
EQ-R
Params
Time
BRACHRS
StyleGAN2
15.73
12.69
StyleGAN3-T (ours)
4.62
63.01
13.12
23.3M
1,00x
20155 ing, 5125 ADA, Boss scradh
StycGGA3-T -A (ours)
4.32
59.33
15.88
p4 symmetry [i6]
4.69
61.90
17.07
21.8M
x.48x
StylGGAN-R (ours)
63.66
37.42
StyleGiAN-- (ours)
66.65
40.48
1.37%
Figure 5: Left: Results for six datasets. We use adaptive discriminator augmentation (ADA) (28] for the smaller datasets. "StyleGAN2" corresponds to our baseline config with Fourier features. Right: Ablations and comparisons for FFHQ-U (unaligned FFHQ) at 256 indicates our default choices.
(+103%) GPU hours. Our accompanying videos show bidee comparisons with StyleGAN2. demonstrating visually that the texture sticking problem has been solved. The resulting motion is much more natural, better sustaining an illusion that there a coherent 3D scene being imaged.
Ablations and comparisons In Section 3.1 we disabled number of StyleGAN2 features. We can now turn them on one by one to gauge their effect on our generators (Figure 5 right). While mixing regularization can be re-enabled without any ill effects, we also find that styles can be mixed quite reliably even without this explicit regularization (Appendix A). Re-enabling noise inputs or relying on StyleGAN2's original layer specifications compromises equivariances significantly, and using fixed Fourier features or re-enabling path length regularization harms FID. Path length regularization is in principle at odds with translation equivariance, as it penalizes image changes upon latent space walk and thus encourages texture sticking. We suspect that the counterintuitive improvement in equivariance may come from slightly blurrier generated images, at cost of poor FID.
In scaling test we tried changing the number of feature maps, observing that equivariances remain at high level, but FID suffers considerably when the capacity is halved. Doubling the capacity improves result quality in terms of FID, at the cost of almost 4x training time. Finally, we consider alternatives for our windowed Kaiser filter. Lanczos is competitive in terms of FID, but as a separable filter it compromises rotation equivariance in particular. Gaussian leads to clearly worse FIDs
We compare StyleGAN3-R R to an alternative where the rotation part is implemented using p4 sym- metric G-CNN 16 17 on top of our StyleGAN3-T. This approach provides only modest rotation equivariance while being slower to train. Steerable filters [55] could theoretically provide competitive EQ-R, but the memory and training time requirements proved infeasible with generator networks of this size.
Appendix A demonstrates that the spectral properties of generated images closely match training data, comparing favorably to several earlier architectures.
Internal representations Figure 6 visualizes typical internal representations from the networks. While in StyleGAN2 all feature maps seem to encode signal magnitudes, in our networks some of the maps take different role and encode phase information instead. Clearly this is something that is needed when the network synthesizes detail on the surfaces; it needs to invent a coordinate system. In StyleGAN3-R, the emergent positional encoding patterns appear to be somewhat more well-defined. We believe that the existence of a coordinate system that allows precise localization on the surfaces of objects will prove useful in various applications, including advanced image and video editing.
5 Limitations, discussion, and future work
In this work we modified only the generator, but it seems likely that further benefits would be available by making the discriminator equivariant as well. For example, in our FFHQ results the teeth do not move correctly when the head turns, and we suspect that this is caused by the discriminator

---

Generated image
Input 2g
Internal representations
Latent interpolation
StyleGAN2
StylGANN3R StyleGAN3-T
Figure 6: Example internal representations (3 feature maps as RGB) in StyleGAN2 and our generators.
accidentally preferring to see the front teeth at certain pixel locations. Concurrent work has identified that aliasing is detrimental for such generalization [5
Our alias-free generator architecture contains implicit assumptions about the nature of the training data, and violating these may cause training difficulties. Let consider an example. Suppose we have black-and-white cartoons as training data that we (incorrectly) pre-process using point sampling 38), leading to training images where almost all pixels jagged. This kind of badly aliased training data is difficult for GANs in general, but it is especially at odds with equivariance: on the one hand, are asking the generator be able translate the output smoothly by subpixel amounts, but on the other hand, edges must still remain jagged and pixels only black/white, to remain faithful to the training data. The same issue also arise with letterboxing of training images, low-quality JPEGs, or retro pixel graphics, where the jagged stair-step edges are a defining feature of the aesthetic. In such cases it may be beneficial for the generator to be aware of the pixel grid.
either black white and the edges
In future, it might be interesting to re-introduce noise inputs (stochastic variation) in way that consistent with hicrarchical synthesis. better path length regularization would encourage neighboring features move together, not discourage them from moving at all. It might be beneficial to try to extend our approach to equivariance w.r.t. scaling, anisotropic scaling, or even arbitrary homeomorphisms. Finally, it is well known that antialiasing should be done before tone mapping. So far, all GANs including ours have operated in the sRGB color space (after tone mapping).
Attention layers in the middle of generator [60 could likely be dealt with similarly to non- non- lineaiirssss by temporarily switching to higher resolution although the time complexity of attention layers may make this somewhat challenging in practice. Recent attention-based GANs that start with tokenizing transformer (e-g., VQGAN [8] may at odds with equivariance. Whether is possible to make them equivariant is important open question.
GANs include many forms of disinfor- mation, from fake portraits in social media [24] propaganda videos of world leaders [43]. Our contribution eliminates certain characteristic artifacts from videos, potentially making them more convincing deceiving, depending the application. Viable solutions include model watermarking [59 along with large-scale authenticity assessment in major social media sites. This entire project in-house cluster of NVIDIA V100s. The new StyleGAN3 generator is only marginally costlier to train or use than that of StyleGAN2.
Potential negative societal impacts of
consumed 92 GPU years and 225 MWh of electricity
Acknowledgments
We thank David Luebke, Ming- Yu Liu, Koki Nagano, Tuomas Kynkäääniemi,, and Timo Viitanen for reviewing early drafts and helpful suggestions. Frédo Durand for early discussions. Tero Kuosmanen for maintaining our compute infrastructure. AFHQ authors for updated version of their dataset. Getty Images for the training images in the BEACHES dataset. We did not receive external funding or additional revenues for this project.
10

---

References
[1] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard,
M. Kudlur, J. Levenberg. R. Monga, S. Moore, D. G. Murray, B. Steiner, P. Tucker, V. Vasudevan, P. Warden,
M. Wicke, Y. Yu, and X. Zheng. TensorFlow: system for large-scale machine learning. In Proc. 12th USENIX Conference on Operating Systems Design and Implementation, OSDI 16, pages 265-283, 2016.
[2] R. Abdal, P. Zhu, N. J. Mitra, and P. Wonka. StyleFlow: Attribate-conditionee exploration of StyleGAN-
generated images using conditional continuous normalizing flows. ACM Trans. Graph., 40(3), 2021.
[3] Y. Alaluf, O. Patashnik, and D. Cohen-Or. Only matter of style: Age transformation using a style-based
regression model. CoRR, abs/2102.02754, 2021.
[4] I. Anokhin, K. Demochkin, T. Khakhulin, G. Sterkin, V. Lempitsky, and D. Korzhenkov. Image generators with conditionall-inddeendeent pixel synthesis. In Proc. CVPR, 2021.
[5] A. Azulay and Y. Weiss. Why do deep convolutional networks generalize so poorly small image transformations? Journal of Machine Learning Research, 20( (0(184):122))) 2019.
[6] D. Bau, A. Andomian, A. Cui, Y. Park, A. Jahanian, A. Oliva, and A. Torralba. Paint by word. CoRR, 0bs/2103.1095/1 2021.
[7] D. Bau, S. Liu, T. Wang, J.-Y. Zhu, and A. Torralba. Rewriting deep generative model. In Proc. ECCV, 2020.
[8] D. Bau, J. Zhu, H. Strobelt, B. Zhou, J. B. Tenenbaum, W. T. Freeman, and A. Torralba. GAN dissection:
Visualizing and understanding generative adversarial networks. In Proc. ICLR, 2019.
191 R. E. Blahut. Theory of remote image formation. Cambridge University Press, 2004.
[10] T. Broad, F. F. Leymarie, and M. Grierson. Network bending: Expressive manipulation of deep generative models. In Proc. EvoMUSART, pages 20-36, 2021.
[11] A. Brock, J. Donahuc, and K. Simonyan. Large scale GAN training for high fidelity natural image synthesis. In Proc. ICLR, 2019.
[12] A. Chaman and Dokmanic. Truly shift-invariant convolutional neural networks. In Proc. CVPR, 2021.
[13] Y. Chen, S. Liu, and X. Wang. Learning continuous image representation with local implicit image function.
In Proc. CVPR, 2021.
[14] Y. Choi, Y. Uh, J. Yoo, and J.-W. Ha. StarGAN v2: Diverse image synthesis for multiple domains. In Pmc.
CVPR, 2020.
[15] M. Chu, Y. Xie, J. Mayer, I.. Teax-Taxxé, and N. Thuerey. Learning temporal coherence via self-supervisin for GAN-based video generation. ACM Trons. Graph., 39(4), 2020.
[16] T. S. Cohen and M. Welling. Group equivariant convolutional networks. In Proc. ICMI., 2016.
[17] N. Dey, A. Chen, and S. Ghafurian. Group equivariant generative adversarial networks. In Proc. ICLR, 2021.
[18] P. Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. In Proc. CVPR, 2021.
[19] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. In Proc. NIPS, 2014.
[20] Gu, Y. Shen, and B. Zhou. Image processing using multi-code GAN prior. In Proc. CVPR, 2020.
[21] Z. Hao, A. Mallya, S. J. Belongie, and M. Liu. GANeraft: Unsupervised 3D neural rendering of minecraft
worlds. CoRR, abs/2104.07659, 2021.
[22] E. Härkönen, A. Hertzmann, J. Lehtinen, and S. Paris. GANSpace: Discovering interpretable GAN
controls. In Proc. NeurlPS, 2020.
[23] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and Hochreiter. GANs trained by a two time-scale update rule converge to local Nash equilibrium. In Proc. NIPS, 2017.
[24] K. Hill and J. White. Designed to deceive: Do these people look real to you? The New York Times, 11 2020.
[25] M. A. Islam, S. Jia, and N. D. B. Bruce. How much position information do convolutional neural networks encode? In Proc. ICLR, 2020.
[26] A. Jahanian, L. Chai, and P. Isola. On the "steerability" of generative adversarial networks. In Proc. ICLR, 2020.
[27] T. Karras, T. Aila, S. Laine, and J. Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In Proc. ICLR, 2018.
[28] T. Karras, M. Aittala, J. Hellsten, Laine, J. Lehtinen, and Aila. Training generative adversarial networks with limited data. In Proc. NewrlPS, 2020.
[29] T. Karras, S. Laine, and T. Aila. A style-hased generator architecture for generative adversarial networks.
In Proc. CVPR, 2018.
[30] T. Karras. S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of StyleGAN. In Proc. CVPR, 2020.
[31] O. S. Kayhan and J. C. van Gemert. On translation invariance in CNNs: Convolutional layers can exploit absolute spatial location. In Proc. CVPR, 2020.

---

[32] M. Liu, T. Breuel, and J. Kautz. Unsupervised image-to-image translation networks. In Proc. NIPS. 2017.
[33] M. Manfredi and Wang. Shift equivariance object detection. In Proc. ECCV 2020 Workshops, 2020.
[34] S. Menon, A. Damian, S. Hu, N. Ravi, and C. Rudin. PUI SE: Self-supervised photo upsampling via latent
space exploration of generative models. In Proc. CVPR, 2020.
[35] A. V. Oppenheim and R. W. Schafer. Discrete-Time Signal Processing. Prentice Hall Press, USA, 3rd
edition, 2009.
[36] T. Park, M. Liu, T. Wang, and J. Zhu. Semantic image synthesis with spatially-daaptive normalization. In Proc. CVPR, 2019.
[37] T. Park, 7hu, O. Wang, J. Lu, E. Shechtman, A. A. Efros, and Zhang. Swapping autoencoder for deep image manipulation. In Proc. NeurlPS, 2020.
[38] G. Parmar, R. Zhang, and J. Zhu. On buggy resizing libraries and surprising subtleties in FID calculation.
CoRR, abs/2104.1 112222 2021.
[39] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury. G. Chanan, T. Killeen, Z. Lin, N. Gimelshein,
L. Antiga, A. Desmaison, A. Kopf, F. Yang. 7. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner,
I.. Fang. J. Bai, and S. Chintala. PyTorch: An imperative style, high-performance deep learning library. In
Proc. NeurIPS, 2019.
[40] P. Ramachandran, B. Zoph, and Q. Lee Swish: self-gated activation function. CoRR, ahs/1 hidd/799009941, 2017.
[41] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot
text-to-image generation. CoRR, abs/2102. 12002 2021.
[42] E. Richardson, Alaluf, O. Patashnik, Y. Nitzan, Y. Azar, S. Shapiro, and D. Cohen-Or. Encoding in style:
A StyleGAN encoder for image-to-image translation. In Proc. CVPR, 2021.
[43] M. Seymour. Canny AI: Imagine world leaders singing- fxguide, 4 2019.
[44] C. E. Shannon. Communication in the presence of noise. Proc. Institute of Radio Engineers, 37(1):10 (11):10-21.) 21,
1949.
[45] V. Sitzmann, J. N. Martel, A. W. Bergman, B. Lindell, and G. Wetzstein. Implicit neural representations with periodic activation functions. In Proc. NewrIPS, 2020.
[46]
Skorokhodov, S. Ignatyev, and M. Elhoseiny, Adversarial generation of continuous images. Proc.
CVPR, 2021.
[47] R. Suzuki, M. Koyama, T. Miyato, T. Yonetsuji, and H. Zhu. Spatially controllable image synthesis with
internal representation collaging. CoRR, abs/1811.10153, 2019.
[48] M. Tancik, P. P. Srinivasan, B. Mildenhall, S. Fridovich- Keil, N. Raghavan, U. Singhal, R. Ramamoorth,,
J. T. Barron, and R. Ng. Fourier features let networks Icarn high frequency functions in low dimensional domains. Proc. NeurlPS, 2020.
[49] S. Tulyakov, M. Liu, X. Yang, and J. Kautz. MoCoGAN: Decomposing motion and content for video
generation. In Proc. CVPR, 2018.
[50] K. Turkowski. Filters for Common Resampling Tasks, pages 147-165. Academic Press Profcssional, Inc., USA, 1990.
[51] C. Vasconeelos, H. Larochelle, V. Dumoulin, R. Romijnders, L. Roux, and R. Goroshin. Impact of aliasing on generalization deep convolutional networks. In /CCV, 2021.
[52] C. von der Malsburg. Self-organization of orientation sensitive cells in striate cortex. Biological Cybernetics, 14(2):85-100, 100, 1973.
[53] T. Wang. M. Liu. J. Zhu, A. Tao, J. Kautz, and B. Catanzaro. High-resolution image synthesis and semantic manipulation with conditional GANs. In Proc. CVPR, 2018
[54] M. Weiler and G. Cesa. General B(2) -qqiivrauva steerable CNNs. In Proc. NeurIPS, 2019.
[55] M. Weiler, F. A. Hamprecht, and M. Storath. Learning steerable filters for rotation equivariant CNNs. In
Proc. CVPR, 2018
[56] D. Worrall and M. Welling. Deep scale-spaces: Equivariance over scale. In Proc. NeurIPS, 2019.
[57] D. E. Worrall, S. Garbin, D. Turmukhambetov, and J. Brostow. Harmonic networks: Deep translation
and rotation equivariance. In Proc. CVPR, 2017.
[58] R. Xu, X. Wang, K. Chen, B. Zhou, and C. C. Loy. Positional encoding as spatial inductive hias in GANs.
In Proc. CVPR, 2021.
[59] N. Yu, V. -Skripniuk, S. Abdelnabi, and M. Fritz. Artificial fingerprinting for generative models: Rooting
deepfake attribution in training data. CoRR, abs/2007 .08457, 2021.
[60] H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena. Self-attention generative adversarial networks. In
Proc. ICMI,, 2019.
[61] R. Zhang. Making convolutional networks shift-invariant again. In Proc. ICMI., 2019.
[62] J. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired saage-to-image translation using cycle-consisten adversarial networks. In Proc.
[63] x. Zou, F. Xiao, Z. Yu, and Y. J. Lee. Delving deeper into anti-aliasing in ConvNets. In Proc. BMVC, 2020.
12


Reviews:
[{"review_id": "y3Qp4fTZlir", "reviewer": "Reviewer_6RJP", "summary": "This paper analyzed the aliasing and equivariance issues of standard GANs components through the lens of signal processing and proposed several corresponding techniques to modify the StyleGAN2 generator to make it better suited to video and animation tasks. The visual results are impressive and will benefit a variety of following works.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this is a high-quality paper with solid contributions to be published at NeurIPS.\n\n**Originality**: This paper makes the first step in solving aliasing and equivariance issues in GANs using a series of newly proposed methods. Both the tasks and methods are new to the GAN community.\n\n**Quality**: This paper is of high quality. Both the theoretical reasoning and the experimental results are convincing. However, it could be better to include experiments or some discussions on the following points:\n1. The method is built on the assumption of limited bandwidth signals, whose valid bandwidth is restricted by the sampling rate or image resolution. So how does the proposed method work on low-resolution images where the sampling rate is more challenging?\n2. What exactly happens when the training dataset contains very high-frequency signals? Will it simply fail (as indicated by the authors) or will the generated images look like the ones that have been passed through a low-pass filter? This may help better understand the limitation of the proposed method.\n\n**Clarity**: This paper is well-organized but some parts of it need more explanation:\n1. There seems to be a gap in motivation. The “texture sticking” phenomenon, positional references and equivariance are easy to understand, but how are they related to aliasing? I do not see strong evidence showing that “texture sticking” is caused by aliasing. So why is the paper named “Aliasing-free GANs” but not something like “Equivariant GANs”? Equivariance seems to be a more straightforward motivation.\n2. How do the modulation and demodulation influence the equivariance?\n3. The visualization in Figure 6 looks impressive. However, the explanation of how it is generated and how to interpret it is not enough (e.g. why the proposed method yield the grid/wave-like patterns?).\n\n**Significance**: The results are impressive and the contributions are solid. I believe it will attract many follow-up works in GAN-based video and animation synthesis and processing.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "m2u5mcZ7exv", "reviewer": "Reviewer_1gcw", "summary": "This paper explores in detail the intrinsic properties of StyleGAN generators that lead to a variety of aliasing artifacts. The source of these artifacts is traced to several distinct phenomena, which are pointwise addressed via a lens of analysis in the continuous signal space, relying heavily on fundamental signal processing techniques. The resulting solutions are applied to a StyleGAN generator, resulting in an “alias-free” model that is better able to handle effects like pose variation while still maintaining the sample quality of the baseline.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an excellent paper based around a solid premise, with strong empirical results against relevant and up-to-date baselines. The analysis is easy to follow and the proposed methods are sensible. This paper is a clear accept without need for revision; I feel that (barring actual mistakes, which I am not able to spot after reading through in detail) seeking out minor points to trip the authors up on would be a waste of time for all involved.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. It's important to note that StyleGAN models in particular have been used to fuel bot armies and disinformation campaigns through the use of generated faces, and the authors acknowledge this.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "htKAHwukVwA", "reviewer": "Reviewer_B84m", "summary": "The paper notes that the current GAN architectures do not synthesize images in a natural hierarchical manner. Finer details in the images are dependent on pixel coordinates, which is undesirable. This is evident from the internal representations learned by the network. The authors attribute this problem to aliasing and propose changes to get rid of it. Specifically, they propose changes to upsampling, downsampling, and non-linearity to make the features rotation and translation invariant. Finally, the authors apply those changes to StyleGAN2 architecture and show comparable FID scores and improved translation and rotation invariance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is well motivated and does a good job of explaining the problem. Authors introduce \"non-critical sampling\" and the \"flexible layers\" which clearly show to increase desirable invariances for GANs and increase FID scores. The paper is moderately well written, clarifications.\n\nHere are some questions that need discussion:\nWhy would the handling of boundaries and upsampling hurt FID? Why would filtered non-linearities hurt FID?  Does the change from D to E keep the filter size to be the same? Wondering if the increase in translation invariance with E is causes just by using a larger filter size! Why does adding noise is contrary to the goal of this paper? Can they be treated as inputs, and hence, they should be translated (or rotated) to compute translation (or rotation) invariance metrics?\n\nThe biggest contributor to the performance seems to come from “non-critical sampling” and the “flexible layers”. It will be valuable for readers to know how does the performance look with just these two changes.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Even if the proposed changes apply to any convolutional architecture, the tests have been done only on StyleGAN2, which limits the empirical soundness of the results. The analysis is limited to GANs without attention layers. Either this should be specified in the limitations or experiments related to attention layers should be added.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "NVC8VX0TVz", "reviewer": "Reviewer_gQig", "summary": "Starting from the observation that current generative models in fact do not synthesize images in a natural hierarchical manner (which they call “texture sticking”), this paper defines the problem systematically, analyze the root of it (unintentional information leaking and aliasing), and proposes a set of simple architectural solutions that are motivated from the signal processing tools. \nThe key component of the approach are: \n- pointing out the presence of “texture sticking” phenomenon that much of the fine details of synthesized images appear to be fixed in the absolute pixel coordinates, not the relative position of the underlying objects being drawn. \n- analyzing signals that flow through a network in a continuous domain and interpreting the actual, discrete features as merely a convenient sampled encoding of it\n- fixing the operations (up/down sampling and nonlinearity) that are the root of the aliasing by applying a proper low-pass filtering. \nAfter applying these fixes, the generator becomes translation- and/or rotation- invariant and provides authentic 2D images as if there is actual 3D model and it is capturing pictures in different views. \n", "questions": "", "limitations": "", "rating": 10, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a very nice work with impressive results, a great progress achievement in the field of image generation. Very well written.\nIn particular, I am very impressed with how deeply the authors investigate the deficiencies of the current state-of-the-art, which could have been easily overlooked. The way that the authors analyze the problem is very convincing, and not only do the theoretical analyses provide deep insights but the empirical results shows big advances by fixing the issues at a fundamental level. I believe these insights will be very useful for the future developments of the field. \nExcept a few minor things for clarification, I recommend the paper to be accepted as is. \n\n1) If the ReLU-type of nonlinearity is the problem, would using other smooth type of nonliearity (like swish) function give better result? \n2) In config R, every 3x3 convolution is now replaced to 1x1. Then, is it correct to understand that the network is now entirely relying on the interpolation function (upsamlpling part) for learning the relationship between the neighboring pixels? \n\n2) Line 135, “convolving it with a discretized downsampling filter …” I think this should be changed to low-pass filter. \n3) Line 34, “faint after-images of the pixel grid …”: what does this mean? It was not clear for me even after reading the entire paper. Could you please clarify this a bit more in the paper?\n4) Line 93, “In the remainder of this paper we shall interpret …” In the remainder of this paper, we shall interpret … (comma is missing)\n5) Line 153, “This discrete operation cannot be realized without temporarily entering the continuous\nrepresentation.” This was not obvious to me. Isn’t the downsampling operation has the same issue (since it needs to be convolved with low-pass filter in continuous domain)? Can you elaborate a bit more? \n6) Line 158 to Line 161, Does this mean that if we limit the range of novel frequencies very strictly, the generator won't be able to add any fine details? \n\nOverall, as stated above, the paper is well written. However, considering the impact of the paper to the field, it would nice if the authors elaborate the details (that I pointed here) a bit more for the ones who are not familiar with signal processing context. \n\n---\n\nupdate after the rebuttal: I am satisfied with the authors' response and I keep my score. \n\nSome questions I actually forgot to ask: \n1) \"Recall that sampling rate s equals the size of the canvas in pixels, given our definitions in...\" Here, what is the (physical) size of the canvas then? How do you set it? \n2) \"but as a separable filter it compromises rotation equivariance in particular.\": why? \n3) \"they contain strong overlays.\": what does this \"overlays\" mean? \n\n---", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No comment.", "review_text": "", "extra_scores": {"ethical_concerns": "No comment.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper identifies the "texture sticking" problem in image synthesis GANs, argues that this is caused by aliasing in the convolutions, proposes architectural modifications to fix the aliasing, and demonstrates strong improvements in equivariance / smooth transitions. All reviewers agreed that the results were impressive. Separately and equally importantly, all reviewers agreed that the problem was clearly motivated and multiple reviewers (gQig, 1gcw) also agreed that the analysis was insightful. Papers which succeed on both of these fronts are rare; I recommend acceptance.
Decision: Accept (Oral)