Learning Modulated Transformation in GANs
Authors: Ceyuan Yang
Qihang Zhang
Yinghao Xu
Jiapeng Zhu
Yujun Shen
Bo Dai
Venue: NeurIPS 2023 poster (2023)
Keywords: generative adversarial network
image synthesis
video synthesis
TL;DR: GAN, image generation, video generation

Abstract: The success of style-based generators largely benefits from style modulation,
which helps take care of the cross-instance variation within data. However, the
instance-wise stochasticity is typically introduced via regular convolution, where
kernels interact with features at some fixed locations, limiting its capacity for
modeling geometric variation. To alleviate this problem, we equip the generator
in generative adversarial networks (GANs) with a plug-and-play module, termed
as modulated transformation module (MTM). This module predicts spatial offsets
under the control of latent codes, based on which the convolution operation can
be applied at variable locations for different instances, and hence offers the model
an additional degree of freedom to handle geometry deformation. Extensive
experiments suggest that our approach can be faithfully generalized to various
generative tasks, including image generation, 3D-aware image synthesis, and
video generation, and get compatible with state-of-the-art frameworks without
any hyper-parameter tuning. It is noteworthy that, towards human generation on
the challenging TaiChi dataset, we improve the FID of StyleGAN3 from 21.36 to
13.60, demonstrating the efficacy of learning modulated geometry transformation.
Code and models are available at https://github.com/limbo0000/mtm.

Learning Modulated Transformation in GANs
Ceyuan Yang'
Qihang Zhang²
Yinghao Xu²
Jiapeng Zhu
Yujun Shen'
Bo Dai
SSanggaai Al Laboratory
CCUHK
*HKUST *Ant Group
Abstract
The success of style-based generators largely benefits from style modulation, which helps take care of the cross-instance variation within data. However, the instance-wise stochasticity is typically introduced via regular convolution, where kernels interact with features at some fixed locations, limiting its capacity for modeling geometric variation. To alleviate this problem, we equip the generator in generative adversarial networks (GANs) with a plug-and-play module, termed modulated transformation module (MTM). This module predicts spatial offsets under the control of latent codes, based on which the convolution operation can be applied at variable locations for different instances, and hence offers the model an additional degree of freedom to handle geometry deformation. Extensive experiments suggest that approach be faithfully generalized to various generative tasks, including image generation, 3D-aware image synthesis, and video generation, and get compatible with state-of-the-art frameworks without any hyper-parameter tuning. It is noteworthy that, towards human generation on the challenging TaiChi dataset, we improve the FID of StyleGAN3 from 21.36 13.60 30,6, demonstrating the efficacy of learning modulated geometry transformation. Code and models are available https://github. http:/hhww..oommmoooo/////////o/
Introduction
Generative models are distinguished from other vision models by creating new data. The crux of learning a generative model is to mimic the observed data distribution through modeling as many data variations as possible. For example, good generative model learned on collection of cat images is expected to produce cats with diverse breeds, postures, hair colors, etc. Among all kinds of generative models [27, 43, 18, 48, 47], generative adversarial networks (GAN) [13] have received wide attention thanks to their flexibility and fast inference speed.
The recent success of GANs lies in the design of style-based generator [23]. Inspired by adaptive instance normalization (AdaIN) [19], StyleGAN [23] introduces layer style codes modulate the per-layer feature map from the generator. As these style codes are instance-specifue the stochasticity manifests itself in different ways of feature modulation, which finally contributes to diverse generation. Nevertheless, it is observed that GANs usually achieve good performance on single-object and aligned datasets, where all instances are with similar geometry, yet struggle in handling complex data distribution, such as ImageNet [10] that consists of various categories of objects, or TaiChi players [41] that make irregular movements.
We argue that such an issue may get caused by style modulation failing to model large geometry variations within data. Concretely, the operation of AdaIN equally acts on the feature map every spatial location, leaving the problem of geometry modeling to the convolution operation together with its receptive field. Unfortunately, regular convolution samples the input feature map at fixed locations, as shown in Fig. la. In this way, the model applies a fixed receptive field for all instances, limiting its capacity for learning geometry variations.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Latent Code Modulation
Spatial Temporal Latent Code Modulation
Feature
Kernel
(a) Regular Convolution
(b) Explicit Transform with Learnable Offsets
(c) Explicit Transform with Learnable Offsets
with Fixed Locations
Figure 1: Illustration of our MTK. (a) Regular convolution interacts with features at fixed locations. (b) Our module predicts the offsets under the latent code modulation, based on which convolution can operate at variable locations for different instances. (c) When generating videos, the offsets with respect to each frame jointly modulated by the spatial and temporal latent codes. With these learnable offsets, this module offers additional degree of freedom to handle geometry transformation.
To tackle this obstacle, propose modulated transformation module (MTM) that allows the convolutional kernels to interact input features at variable locations, as shown in Fig. 1b. In particular, given the feature map from the previous stage, first predict offset for each spatial point and then use the predicted offsets to warp the input feature before performing convolution. Here, the offset prediction conditioned on the latent code and hence varies across instances. That way, the model is supported with an additional degree of freedom to learn geometry variation explicitly. It is noteworthy that, our MTM also applicable to video generation, as shown in Fig. Ic, to help model the geometry transformation over time.
We evaluate approach
range of generative tasks, including image generation, 3D-aware image synthesis, and video generation. For each task, we choose the state-of-the the art frameworks, i.e., StyleGAN2 [24], StyleGAN3 [26). EG3D [7], and StyleSV [61] our strong baselines. Extensive results on ImageNet [10], LSUN [58], You'ubbe Driving [60]. SkyTimelapse [53], and TaiChi [41] suggest that MTM is capable of consistently boosting the performance, demonstrating its efficacy and generalizability. More importantly, MTM works as plug-and-plat module such that it can be effortlessly deployed popular GAN-based frameworks even without any syper-parameter tuning. Hopefully, our efficient MTM could serve as basic operation in the future design of generative models.
2 Related work
Image generation with GANs. Generative adversarial network (GAN) [13] has significantly advanced image generation. Typically, two-player game is played between generator and discriminator. The generator aims to reproduce the real data distribution, while the discriminator aims to distinguish fake samples generated by the generator from the real ones. Many efforts have been made to design better architectures for both generators and discriminators [34, 15, 4, 22, 23, 38, 39, 21], to regularize discriminators for training stability [1, 15, 30, 57, 29, 2]. and to bring various inductive biases [32, 6, 14, 54, 40, 7, 62, 55, 33, 50, 16, 63] to support explicit control. With the rapid development of generative models, a family of the style-based generators [23] stands out due to their photo-realistic synthesis and editing applications. In this work, start with this popular generator and introduce new module that targets at explicitly modeling the large geometric variation, leading to better synthesis quality on various benchmarks.
Video generation with GANs. Many attempts have been made leverage GANs synthesize videos from latent space. Video generation [44, 36, 45, 11, 42, 59, 49, 37, 8, 52, 61, 5] also grows up by incorporating the latest designs of image generators with temporal modeling. For example, TGAN [36] and MoCoGAN [45] explicitly disentangle the motion from the content, which is widely adopted in many years. Considering the dominant advantage of the style-based generator image synthesis, multiple successful designs are further developed into video generation [42, 5, 61]. In this

---

work, we generalize the proposed module to video generation and demonstrate that it could also lead to consistent gains of synthesized video quality.
Spatial transformation in neural networks. In the field of visual perception, considering the inherent limitation of regular convolutions, spatial transformer network (STN) [20] first proposed to warp the whole intermediate representations according to learned global affine transformation. Deformable ConvNets (DCN) [9] further leveraged learnable offsets sample intermediate features in local and dense manner. Therefore, the spatial representations could be arbitrarily rearranged to represent the large-scale geometric variation. Its second version DCNv2 [64] enhanced the capacity and simplified the training strategy. The latest version DCNv3 [51] has been vital component for the scaee vision foundation models. Our work is very closed the family of these spatial networks. Differently, the predicted offsets are modulated by the latent code which does not exist prior perception tasks. Moreover, to the best of our knowledge, it the first time to introduce such explicit transformation into visual generation tasks.
3 Method
To facilitate generative adversarial networks in synthesizing data with large geometric transformations such as unaligned ImageNet images and TaiChi videos, we propose the modulated transformation module (MTM) that could explicitly deform the spatial geometry of given intermediate representations. Sec. 3.1 first presents how a regular convolution is performed in detail and shows simple yet effective improvement that could handle large geometric variation. Sec. 3.2 further illustrates how to incorporate such module into prior generative models for content generation (i.e., images and videos). At last, Sec. 3.3 describes the training details including learning objectives and implementations.
3.1 Explicit spatial transformations
Regular convolution. Regular convolutional operations are inherently limited to model large geometric Transfomation/eeformatooo originating from their fixed shape of kernels. Specifically, given the intermediate representation x and the corresponding output y after a regular convolution, we let (p) and (p) denote the feature at location p respectively. Taking convolution with a 3 x 3 kernel as an example, the regular convolution could be applied by sampling features at 9 locations:
(p)
Σ
Ws x(p + P:),
where w; is the i-th weight of the convoultional kernel and p; 1-1,0, 1 2 denotes the pre-defined spatial offset. In particular, current GANs usually adopt the modulated convolutions [23] whose weights would be modulated by the latent codes. Here the modulation is omitted for brevity. Such an operation aggregates the local information of rectangular window (e.g., 3 x 3) centered at location p. For visual concepts whose information is scattered at irregular locations, it's thus hard for generative adversarial networks with regular convolutions to handle them effectively. Although ideally this limitation can be alleviated by stacking several layers and accumulating a sufficiently large receptive field, we empirically found in practice such bypass struggles draw non-rigid visual concepts. We therefore introduce the learnable offsets to GANs and enhance their ability with negligible computational overhead.
Geometric transformations with learnable offsets. Akin to the family of spatial transformer networks [20, 9, 64, 51], we introduce the learnable offsets Ap that explicitly rearrange the spatial locations of intermediate representations first. The regular convolution could modified as
(p)) Σ x.-x(pP + Pi + Apr)
Here Ap; stands for the learnable offset for the i-th location. In particular, representation at location p + p; + Ay; would be obtained through the bilinear interpolation since the final offset would be fractional. Accordingly, the original fixed shape of the regular convolution would be arbitrarily reshaped to enable a larger receptive field, facilitating the geometric variation modeling simultaneously. To make the additional offsets data-dependent, prior literature regarding visual
3

---

perception [20, 9, 64, 51] usually predict offsets through an extra convolution (Conv) as
Ap (xnv
(3)
Thus, for each location of representation x, 18 scalars are predicted form 9 pairs of spatial offsets. This could be easily implemented by setting the number of output dimensions as 18 regarding the extra convolution.
Latent code modulation. Different from perception tasks that often take diverse images as inputs, generation tends to produce realistic images from latent codes that are randomly sampled from pre-defined distribution (e.g., normal distribution). Therefore, we further leverage the latent code and improve the offset prediction through the latent code modulation:
Åp- ModConv(x, z).
Here, ModConv denotes the modulated convolution proposed by Karras et. al. [24]. Concretely, ModConv allows the input latent code z to modify the weight w of convolutions first by multiplication (i.e., w = w z). Such an improvement further makes the predicted offsets much more instance- specific since the corresponding latent code indeed contains all semantics of the synthesized output. Meanwhile, the optimization of training becomes more stable.
To this end, combining Eq. (2) and Eq. (4) results in our modulated transformation module (MTM) with the negligible computational overhead. Importantly, no additional explicit supervisions are required for the learning of MTM, making it as plug-and-play module for various generators and tasks.
3.2 Content generation framework
Considering that generative adversarial networks (GANs) have advanced 2D/3D-aware image and video generation, we incorporate the proposed module into multiple popular generators for various synthesis tasks. Briefly, GANs formulate content generation as a two-player game where a generator G(.) learns the mapping from a pre-defined distribution to the data distribution while a discriminator D(-) aims at distinguishing the synthesized from the real data distribution. As no modifications are made on the discriminator, we thus neglect the relevant discription in the following context for brevity.
Synthesizing 2D images. A style-based generator is proposed in [23], which injecting the mapped latent codes into the generator layer by layer. Such layer-wise design of the architecture results in the semantic hierarchy [23, 56] where various layers usually encoder different visual concepts, making become a standard generator for image synthesis. We thus choose the StyleGAN2 [25] and its alias-free version StyleGAN3 [26] as the image generator. To incorporate the proposed MTM, we simply replace one convolution various resolutions with our module. In this way, the improved generators are able to handle the geometric variation from small to the large scales.
Synthesizing 3D-aware images. With the development of 2D image synthesis, researchers pour their attention to incorporate the inductive bias into the original generator such that more explicit control could be empowered. For instance, EG3D [7] makes the best of 2D image generators to produce three-views drawing of a 3D representation space, leading to a effective and efficient tri tri-plane representations for 3D-aware image synthesis. Accordingly, could still insert the proposed MTM into the 2D generator of EG3D [7] (as before), indirectly rearranging the tri-plane representations.
Synthesizing videos. When producing videos, large geometric variation does not only exist across video instances but also occur over time. MTM is thus employed into the video generator. In particular, MTM is equpped with the previous state-of-the-a manages to lift a 2D generator to synthesize infinite and smooth video frames. Particularly. regular convolutional layers would be also replaced by module. Furthermore, following the similar philosophy of [61], ]. the latent code that drives the learnable offsets in Eq. (4) consists of the motion and content information simultaneously for video generation.
video generation approach StyleSV [61] that
3.3 Training
Learning objectives. As mentioned before, MTM makes no additional modifications on the discrininators and requires no explicit supervisions. Therefore, learning objectives strictly follow the corresponding approaches. For example, non-saturating logistic loss [13] together with R regularization [30] is adopted for the adversarial training of the generator and discriminator.

---

Table 1: Comparisons on 2D image generation. FID, CLIP-FD and sFID are reported for the
quantitative measurement.
(a) Evaluation on ImageNet-1288 [10].
(b) Evaluation on TaiChi-256 [41].
Methods
FID,
CLIP-FD,
sFID,
Methods
FID,
CLIP-FD,
sFID
StyleGAN2 [24]
21.14
35.60
4.52
StyleGAN3 [26]
21.36
25.97
4.79
MTM
19.16
33.92
4.39
w/ MTM
13.60
19.27
3.57
(c) Evaluation on LSUN Church-256 [58].
(d) Evaluation LSUN Cat-256 [58].
Methods
F
CLIP-FD,
sFID4
Methods
FID,
CLIP-FD,
sFID,
StyleGAN2 [24]
4.04
33.34
8.15
StyleGAN2 [24]
6.87
24.31
6.06
w/ MTM
2.32
25.98
6.93
w/ MTM
5.92
19.89
5.70
Parameter efficiency. Compared to the regular convolution, the newly-introduced parameters by MTM are mainly those of the extra convolution that aims at predicting the offsets. Importantly, as mentioned in Sec. 3.1, the number of output dimension is quite limited (18 in practice). Therefore, negligible parameters would be caused even counting extra parameters of multiple MTM in total.
Computing efficiency. Although all convolutions could be replaced by the proposed MTK (still causing negligible extra parameters), we observe that the training would be further slowed down due to large number of new operations. Intuitively. the high resolution intermediate representations might have already determined the geometric structure. Inspired by this, we further investigate which layers are supposed be enhanced by the proposed MTM. Our experiments suggest that replacing regular convolutions at few low-resolution layers is sufficient, leading to a comparable training efficiency. This also matches the finding in [56] that the low-resolution intermediate representations mainly control the spatial layout'structure.
Experiments
We evaluate the proposed MTM on various generative models for both image and video generation tasks. In Sec. 4.1, we outline the benchmarks, evaluation metries, and corresponding baseline approaches involved in the experiment. Sec. 4.2 showcases the primary synthesis outcomes for 2D and 3D-aware image generation. We further integrate MTM into video generators Sec. 4.3, Finally, we present the ablation study in Sec. 4.4 to demonstrate the impact of each design.
4.1 Settings
Datasets. We employ several challenging benchmarks to evaluate the efficacy our module. Firstly, use ImageNet [10], which contains approximately 1.2 million images covering 1000 object categories, showcasing significant geometric variation. Due to limited training resources, we resize all images 128 x 128 pixels. Additionally, benchmarks for image generation that feature outdoor scenes and non-rigid animals respectively. Furthermore, employ TaiChi [41], which consists of around 3000 videos of people performing TaiChi, as an image benchmark. We treat each frame as an independent image for image generation since diverse gestures among individuals exist. All images resized 256 256 pixels.
utilize LSUN Church[58] and Cat [58], two typical
Regarding video generation, we use SkyTimelapse [53], which captures moving clouds, and TaiChi [41], which features large non-rigid motion. Moreover, use YouTube Driving[60], which collects 134 driving videos with varying weather conditions, regions, and cities. According to[61], these driving videos have relatively strict geometric constraints, making them appropriate benchmark for our evaluation.
Metrics. We use Fréchet Inception Distance (FID) [17] as the quantitative metric for measuring the quality image synthesis, as it can reflect human perception to some extent. We employ the official pre-trained Inception feature extractor, which can be downloaded from here. Additionally, we follow prior literature[28] and replace the Inception extractor with CLIP [35], which we refer to as CLIP-FD. Furthermore, to assess the synthesis quality from the perspective of spatial structure, we also report
5

---

ImageNet-128
TaiChi-256
LSUN Church-255
LSUN Cat-256
FFHQ-256
Figure 2: Qualitative results on various datasets for image generation. Note that synthesis FFHQ-256 [23] is produced by a 3D-aware generator i.e., EG3D [7].
sFID [31]. For video generation, utilize Frechet Video Distance (FVD) [46] the metric, with two temporal spans: consecutive 16 and 128 frames, respectively.
Baselines. For 2D and 3D-aware image generation, we select StyleGAN2 [24], StyleGAN3 [26], and EG3D [7] as the baseline approaches. We also employ StyleSV [61], the recent state-of-the-ar video generator, as a baseline for video synthesis. Importantly, when incorporating our MTM with these baselines, we make no additional modifications to them. The training scheme and hyper-parameters remain untouched.
4.2 Towards image generation
Main results. Tab. presents the experimental results on multiple datasets for 2D image generation. Notably, treat all frames in TaiChi [41] independent images. We keep all training hyper-

---

Table 3: Comparisons on video generation. FID, and FVD, are reported for the quantitative
measurement.
(a) Evaluation on Sky SkyTieeapss [53].
(b) Evaluation on TaiChi-256 [41].
Methods
FVD1s
2VD128
FID
Methods
FVD16
FVD128
FID
MoCoGAN [45]
85.9
272.8
MoCoGAN-HD [44]
144.7
DIGAN [59]
83.1
196.7
DIGAN [59]
128.1
Long Lin [5]
116.5
152.7
TATS-basc [12]
94.6
StyleGAN-V [42]
73.9
248.3
40.8
StyleGAN-V [42]
152.0
267.3
33.8
StyleSV [61]
49.0
135.9
49.9
StyleSV [61]
97.4
188.9
26.6
MTM
42.3
124.6
44.5
w/ MTM
89.5
180.6
25.2
(c) Evaluation on YouTube Driving [60].
Methods
FVD16
FVD128
FID
StyleGAN-V [42]
449.8
460.6
28.3
StyleSV [61]
207.2
221.5
19.2
w/
194.8
198.4
10.3
parameters (e.g., batch size, learning rates, number of iterations, and coefficients of RS penalty) identical for both baseline generators and the improved ones (i.e., w/ MTM).
Our module introduces consistent gains from all measurement perspectives. The differences in FID and CLIP-FD metries suggest significant improvement in overall synthesis quality. Specifically, without increasing the generators' capacity, FID on ImageNet can be boosted from 21.14 4 to 19.16. Previous approaches [38, 3] suggest that capacity matters for large-scale datasets. However, our negligible computations also result in noticeable synthesis improvement, strongly demonstrating our module's s effectiveness to some extent. Moreover, sFID indicates that generators can produce images with more reasonable spatial structure when equipped with our MTM. Fig. Al showcases the qualitative results for 2D and 3D-aware image generation.
Tab. 2 presents the experimental results for 3D-aware image generation. It is important to note that we strictly follow the data pre-processing of the original EG3D [7]. Although EG3D runs on well-aligned facial datasets, such as FFHQ [23], which hardly contain large-scale geometric variation, we still observed an improvement in quality.
Table 2: 3D-aware image generation.
FFHQ-256
FID,
StyleGAN2 [24]
3.78
EG3D [7]
4.32
w/
4.07
This further narrows the gap between pure 2D image generation (i.e., StyleGAN2 [24]) and 3D-aware image generation (i.e., EG3D [7]).
4.3 Towards video generation
Main results. Tab. 3 presents the comparisons for video generation. Additionally, Fig. A2 showeases the qualitative results. Notably, StyleSV [61] serves our baseline. Specifically, StyleSV trains video generator in two stages, first pre-training an image generator on video datasets where each frame is considered an individual image. As expected, generators with our module outperform the baselines again. After pre-training StyleSV finetunes the generator on video datasets, incorporating temporal motion representations and explicit temporal modeling the generator and discriminator, respectively, Our MTM is always trainable during these two stages. Finally, we compare the improved StyleSV [61] against prior literature on multiple benchmarks.
Our MTK significantly improves the generator's performance on video synthesis benchmarks, leading to new state-of of-the-art results. The improvement is evident from both FVD16 and FVD128 metrics, indicating enhanced short-term and long-term video synthesis. Additionally, our module improves the single-frame quality, as evidenced by lower FID scores. In summary, our proposed MTM enables the original generator to handle large geometric variation across instances and over time, resulting in significant improvements in vidco synthesis performance.
7

---

SkyTimelapse
TaiChi
YouTube Driving
Figure 3: Qualitative results various datasets for video generation. Due the limited space, subsample frames from the generated videos here.
4.4 Ablation studies
Which layer requires the proposed MTM? is possible to replace all convolutions in given generator with our module, but doing so would significantly slow down the training process. Therefore, we conducted an ablation study to investigate the best trade-off between speed and performance. Prior literature [23, 56] suggests that synthesized image is represented hicrarchically. Specifically, the lower-resolution layers (closer to the latent codes) tend to determine the spatial structure of generated image, while the higher-resolution layers focus on fine- fine-grained details like texture. This inspired us to question whether replacing all convolutions is necessary. Instead, we hypothesized that empowering the layers most relevant to the spatial structure might be sufficient. To test this hypothesis, we split the layers of the generator into three groups based on the resolution of intermediate representations: low, mid, and high.
Tab. presents the comparisons ImageNet [10] when replacing convolutions of different layers with our MTM. The results indicate that applying our module at the low-resolution layers achieved appealing performance with acceptable training time. Incorporating more modules into the generator

---

Table 4:
Ablation studies on effect of various layers. All layers of the generator are split into three groups: Low, Mid and High which we replace with the proposed module. Obviously, introducing it into the low-resolution layers results in the best trade-off between performances and training efficiency.
Low
Mid
High
FID_
CLIP-FD,
sFID
Training time
Inference time
Param. (MB)
21.14
35.60
4.52
1.0x
1.0x
27.78
19.16
33,92
4.39
1.2x
1.0x
28.55
19.35
34.53
4.42
2.1x
1.0x
28.94
v
20.02
34.76
4.46
3.2x
1.0x
29.32
Figure 4: Visualization of disabling offsets after training. We set the learnable offsets zero at the second row. Interestingly, the shape of the main objects in the synthesis becomes quite similar, demonstrating the potential of MTM in performing explicit deformation to deliver various shapes from the blob-like.
did not lead to further improvements but significantly slowed down the training process. We also observed that training became somewhat unstable when using multiple modules on high-resolution layers in practice. Therefore, we decided to use MTM only in the low-resolution layers, which is also how implemented all image and video generation experiments presented above.
What do the learnable offsets exactly capture? Visualizing the learnable offsets our MTM is challenging and non-intuitive because it operates directly on intermediate representations with the generator. Therefore, we disabled the learnable offsets to analyze the behavior of MTM. Specifically, after training, we set the offsets to zero. Importantly, even when the learnable offsets are set to zero, the regular convolutions still function normally.
Fig. presents the corresponding results. The first row shows the original synthesis for image generation on TaiChi [41]. When we roughly disabled the learnable offsets, the synthesis seemed to suffer from collapse. Interestingly, the shape of the main objects in the synthesis became blob-like. deliver
This demonstrates that our proposed MTM performs explicit deformationn//raasfommaaooo variety of gestures that all derive from a blob-like shape.
5
Conclusion
In this work, we investigate how GANs handle data collection that involves multiple samples with large-scale geometric variation. Since regular convolution is inherently limited, we propose new method called the Modulated Transformation Module (MTM) that enables explicit and flexible transformations of intermediate representations. Through careful design and comprehensive study, our method significantly improves the quality of image and video generation without additional supervision computational overhead. We evaluate architectures, demonstrating its effectiveness in various datasets and generative tasks. We believe that MTM could become fundamental operation in future generative foundation models.
method across multiple benchmarks and
Discussion. While our proposed MTM consistently leads to gains for both image and video generation, there are still some limitations that we would like to discuss. First, we have not explored whether our method could result in further improvements for other generative models, such as rego-segreseive models and bcore- diffusion models. Fortunately, recent work [39, 21] has already demonstrated

---

that GANs are comparable to other generative models with the proper scaling scheme. Second, the effectiveness of our MTM on large-scale generation tasks such as text-to-image generation. which often requires larger GAN, is unknown. Given the current limited resources, we leave this study for future work.
References
[1] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In Inf. Conf. Mach. Learn., pages 21 214-223, 2017.
[2] Q. Bai. C. Yang. Y. Xu, X. Liu, Y. Yang, and Y. Shen. GLeaD: Improving GANs with a generator- task. In IEEE Conf. Comput. Vis. Puttern Recog., 2023.
[3] A. Brock, J. Donahue, and K. Simonyan. Large scale GAN training for high fidelity natural image synthesis. In Int. Conf. Learn. Represent. 2019.
[4] A. Brock, Donahue,
Simonyan. Large scale GAN training for high fidelity natural image synthesis. In Int. Conf. Learn. Represent, 2019.
[5] T. Brooks, Hellsten, M. Aittala, T.-C. Wang. T. Aila, Lehtinen, M.-Y. Liu, A. A. Efros, and Karras. Generating long vidcos of dynamic scenes. In Adv. Newral Inform. Process. Syst., 2022.
[6] E. R. Chan, M. Monteiro, P. Kellnhofer, J. Wu, and G. Wetzstein. T-GAN: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
[7] E. R. Chan, C. Z. Lin. M. A. Chan, K. Nagano, B. Pan, S. De Mello, O. Gallo, L. J. Guibas, J. Tremblay, S. Khamis, al. Efficient geomelry-aware 3d generative adversarial networks, In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
[8] A. Clark, J. Donahue, and Simonyan. Adversarial video generation on complex datasets. arXiv preprint arXiv:1907.06571, 2019.
[9] J. Dai, H. Qi Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei. Deformable convolutional networks. In Int. Conf. Comput. Vis., 2017.
[10] J. Deng, W. Dong, R. Sacher, Li, K. Li, and I., Fei-Fei. ImageNet: A large-scale hicrarchical image datahase. In IEEE Conf. Comput. Vis. Pattern Recog., 2009.
[11] G. Fox, A. Tewari, M. Elgharib, and C. Theobalt. StyleVideoGAN: A temporal generative model using pretrained StyleGAN. In Brüt. Mach. Vis. Conf., 2021.
[12] S. Ge, T. Hayes, H. Yang, X. Yin, G. Pang. D. Jacobs, J.-B. Huang, and D. Parikh. Long video generation with agnostii VQGAN and time-sensitive transformer. Eur. Conf. Comput. Vis., 2022.
[13] Goodfellow, J. Pouget-Abadie, M. Mirza, Xu. D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Ady, Neural Inform. Process. Syst., 2014.
[14] J. Gu, L. Liu, P. Wang, and C. Theobalt. Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis. In Int. Conf. Learn. Represent. 2022.
[15] I. Guirajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville. Improved training of wasserstein GANs. In Adv. Neural Inform. Process. Syst., 2017.
[16] 7. He, M. Kan, and S. Shan. FigenGAN: Layer-wise eigen-learning for GANs. In Int. Conf. Comput. Vis., 2021.
[17] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs trained by time-scale update rule converge to local nash equilibrium. Adv. Neural Inform. Process. Syst., 2017.
[18] Ho, A. Jain, and Abbeel. Denoising diffusion probabilistic models. Ady. Neural Inform. Process. Syst., 33:6840-6857 2020.
[19] X. Huang and S. Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Inf. Conf. Comput. Vis., 2017.
[20] M. Jaderberg. K. Simonyan, A. Zisserman, et al. Spatial transformer networks. In Adv. Neural Inform. Process. Syst., 2015.
10

---

[21] M. Kang, J.-Y. Zhu, R. Zhang, J. Park, E. Shechtman, S. Paris, and T. Park. Scaling up GANs for text-to-image synthesis. In IEEE Conf. Compul. Vis. Pattern Recog, 2023.
[22] T. Karras, T. Aila, S. Laine, and J. Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In Int. Conf. Learn. Represent., 2018
[23] T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
[24] T. Karras, S. Laine, M. Aittala, Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of StyleGAN. In JEEE Conf. Comput. Vis. Pattern Recog., 2020.
[25] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of stylegan. In IEEE Conf. Comput. Vis. Pattern Recog., 2020.
[26] T. Karras, M. Aittala, S. Laine, E. Härkönen, J. Hellsten, J. Lehtinen, and T. Aila. Alias-free generative adversarial networks. In Adv. Neural Inform. Process. Syst., 2021.
[27] D. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013.
[28] T. Kynkäänniemi, T. Karras, M. Aittala, T. Aila, and J. Lehtinen. The role of ImageNet classes in fr), 'echet inception distance. In Int. Conf. Learn, Represent. 2023.
[29] G. Lee, Kim, J. Kim, S. Kim, J.W. Ha, and Choi. Generator knows what discriminator should learn in unconditional GANs. In Eur. Conf. Comput. Vis., 2022.
[30] L. Mescheder, A. Geiger, and S. Nowozin. Which training methods for GANs do actually converge? In Int. Conf. Mach. Learn., 2018.
[31] C. Nash, J. Menick, S. Dieleman, and P. W. Battaglia. Generating images with sparse representations. In Int. Conf. Mach. Learn., 2021.
[32] T. Nguyen-Phuoc, C. Li, L. Theis, C. Richardt, and Y.-.. Yang. HoloGAN: Unsupervised learning of 3d representations from natural images. Int. Conf. Comput. Vis., 2019.
[33] Peebles, Peebles, J.-Y. Zhu A. Efros, and A. Torralba. The hessian penalty: A weak prior for unsupervised disentanglement. In Eur. Conf. Comput. Vis., 2020.
[34] A. Radforrd L. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In Int. Conf. Learn. Represent., 2016.
[35] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In Int. Conf. Mach. Learn., 2021.
[36] M. Saito, Matsumoto, and Saito. Temporal generative adversarial nets with singular value clipping. In Int. Conf. Comput. Vis., 2017.
[37] M. Saito, S. Saito, M. Koyama, and S. Kobayashi. Train sparsely, generate densely: Memory-efficient unsupervised training of high-resolution temporal gan. Int. J. Comput. Vis., 2020.
[38] A. Sauer, K. Schwarz, and A. Geiger. StyleGAN-XL: Scaling StyleGAN to large diverse datasets. In SIGGRAPHI, 2022.
[39] A. Sauer, T. Karras, S. Laine, A. Geiger, and T. Aila. StyleGAN-T: Unlocking the power of GANs for fast large-scale text-to-image synthesis. In Int. Conf. Mach. Learn., 2023.
[40] Z. Shi, Y. Shen, Zhu, D.-Y. Yeung, and Chen. 3d-aware indoor scene synthesis with depth priors. In Eur. Conf. Comput. Vis., 2022.
[41] A. Siarohin, Lathuilière. S. Tulyakov, E. Ricci, and N. Sebe. First order motion model for image animation. In Adv. Neural Inform. Process. Syst., 2019.
[42] I. Skorokhodov, S. Tulyakov, and M. Elhoseiny. Stylegan-V: continuous video generator with the price, image quality and perks of StyleGAN2. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
[43] J. Sohl-Dicksteer Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Int. Conf. Mach. Learn., pages
PMLR, 2015.
11

---

[44] Y. Tian, J. Ren, M. Chai, K. Olszewski. X. Peng. N. Metaxas, and S. Tulyakov. A good image generator is what you need for high-resolution video synthesis. In Int. Conf. Learn. Represent. 2021.
[45] S. Tulyakov, M.-Y. Liu, X. Yang, and J. Kautz. MoCoGAN: Decomposing motion and content for video generation. In IEEE Conf. Comput. Vis. Pattern Recog., 2018
[46] T. Unterthiner, S. van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:18/2.01777, 2018.
[47] A. Van den Oord, N. Kalchbrenner, I.. Espeholt, O. Vinyals, A. Graves, et al. Conditional image generation with pixelenn decoders. Adv. Neural Inform. Process. Syst., 29, 2016.
[48] A. Van Den Oord, N. Kalchbrenner, and K. Kavukcuoglu. Pixel recurrent neural networks. In Int. Conf. Mach. Learn., pages 1747-1756. PMLR, 2016.
[49] C. Vondrick, H. Pirsiavash, and A. Torralba. Generating videos with scene dynamies. Adv. Neural Inform. Process. Syst., 2016.
[50] J. Wang, C. Yang. Y. Xu, Y. Shen, H. Li, and B. Zhou. Improving GAN equilibrium by raising spatial awareness. In IEEE Conf. Comput. Vis. Pattern Recog.. 2022.
[51] W. Wang, J. Dai, Z. Chen, Z. Huang. Z. Li, X. Zhu, X. Hu, Lu, L. Lu, H. Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In IEEE Conf. Comput. Vis. Pattern Recog., 2023.
[52] Y. Wang, D. Yang, Bremond, and A. Dantcheva. Latent image animator: Learning to animate images via latent space navigation. Int. Conf. Learn. Represent., 2022.
[53] W. Xiong, W. Luo, L. Ma, W. Liu, and J. Luo. Learning generate time-lapse videos using multi-stage dynamic generative adversarial networks. In IEEE Conf. Comput. Vis. Pattern Recog., 2018.
[54] Y. Xu, Peng, Yang, Y. Shen, and B. Zhou, 3d-aware image synthesis via learning structural and textural representations. In IEEE Conf. Comput. Vis. Pattern Recog. 2022.
[55] Y. Xu, M. Chai, 7. Shi, S. Peng, Skorokhodov, A. Siarohin, C. Yang, Y. Shen, H.-Y. Lee, B. Zhou, al. DisCoScene: Spatially disentangled generative radiance fields for controllable 3d-aware scene synthesis. In IEEE Conf. Comput. Vis. Pattern Recog,, 2023.
[56] C. Yang. Y. Shen, and B. Zhou, Semantic hierarchy emerges in deep generative representations for scene synthesis. Int. J. Comput. Vis., 2021.
[57] C. Yang, Y. Shen, Xu, D. Zhao, B. Dai, and B. Zhou. Improving GANs with dynamic discriminator. In Adv. Neural Inform. Process. Syst. 2022.
[58] F. Yu, A. Seff, Y. Zhang. S. Song, T. Funkhouser, and Xiao. LSUU:: Construction a large-scale image dataset using deep learning with humans in the loop. arXiv preprint 150.03355, 2015.
[59] S. Yu, J. Tack, S. Mo, H. Kim, J. Kim, J.-W. Ha, and J. Shin. Generating videos with dynamics-awar implicit generative adversarial networks. In Int. Conf. Learn. Represent., 2022.
[60] Q Zhang, Z. Peng, and B. Zhou. Learning to drive by watching youtube vidcos: Action-conditioned contrastive policy pretraining. In Eur. Conf. Comput. Vis., 2022.
[61] Q Zhang, C. Yang. Y. Shen, Y. Xu, and B. Zhou. Towards smooth video composition. In Int. Conf. Learn. Represent., 2023.
[62] X. Zhao, F. Ma, D. Güera, Z. Ren, A. G. Schwing, and A. Colburn. Generative multiplane images: Making 2d GAN 3d-aware. In Eur. Conf. Comput. Vis., 2022.
[63] J. Zhu, C. Yang, Y. Shen, Z. Shi, D. Zhao, and Q Chen. LinkGAN: Linking GAN latents to pixels for controllable image synthesis. arXiv preprint ar Xii::3301.04600 2023.
[64] X. Zhu, H. Hu, S. Lin, and J. Dai. Deformable convnets v2: More deformable, better results. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
12

---

Appendix
This supplementary material organized follows. We first discuss the broader impact of the proposed method in appendix A. Besides, appendix B provides the implementation details including required resources and codebases we used. Fig. Al and Fig. A2 shows more synthesis. At last, appendix C presents more comparisons and experimental results.
A Broader impact
The proposed approach represents a significant advancement the field of image and video synthesis, which is both fundamental and practical problem. The introduction of the modulated transformation module has greatly enhanced the capacity for handling geometric variation, making generative models more accessible the general public. However, this technique may also facilitate the creation of fake content, such as Deepfake, which could have negative consequences. We want to emphasize that strongly oppose the misuse of this approach to violate security and privacy issues. The negative impact of such misuse can be mitigated by the development deep fake detection technology.
B Implementation details
MTM is developed based on the official implementation of StyleGAN3 and StyleGAN-V for 2D image and video generation. Besides, regarding the 3D-aware image generation, we use the official implementation EG3D. That is, we strictly follow the data-prep prap-prrpooesssing and hyper-parametert of the original ones. Although we train each model on a server with A100 GPUs, it requires no large GPU memory and allows other researchers reproduce easily.
C More Results
Quantitative results for image generation. Tab. Al provides more results for the image synthesis. In particular, frame in the video datasets used for video generation is regarded as an individual image. FID is reported for the comparison. Obviously, our method could still improve the baseline generator.
Longer training for the baseline. To investigate the performances introduced by our approach, we train the baseline methods for double training iterations. Tab. A2 presents the comparison where our method remains to outperform baselines, even with longer training. Interestingly, the synthesis of the baseline ImageNet becomes worse with longer training while module could significantly improve the FID with the same training iterations and negligible computational overhead, demonstrating the effectiveness of our method.
Table A1: Quantitative results on StyleGAN3. FID (the lower the better) is reported for the comparison.
Methods
SkyTimelapse
TaiChi
YouTube Driving
StyleGAN3 w/ MTM
9.37
21.36 13.60
8.40
8.72
6.16
Table A2: Comparison against longer-tuned baseline. FID (the lower the better) is reported for comparison.
Methods
Iterations
ImageNet
LSUN Church
LSUN Cat
StyleGAN2
1x
21.14
4.04
6.87
StyleGAN2
2x
22.35
2,82
6,38
MTM
1x
19.16
2.32
5,92
13

---

ImageNet-
ODDO
TaiChi-256
LSUN Church-255
LSUN Cat-256
FFHQ-256
Figure A1: Qualitative results various datasets for image generation.
14

---

SkyTimelapse
TaiChi
YouTube Driving
Figure A2: Qualitative results on various datasets for video generation.
15


Reviews:
[{"review_id": "0BBej7ykM4", "reviewer": "Reviewer_cDZK", "summary": "The paper proposes a modulated transformation for GANs. Specifically, they propose learning the offset of each convolutional layer by learning additional convolution layers to predict offsets where the inputs are the latent code and the current intermediate layer features. The proposed method improves the expressive power of overall GAN framework and shows notable improvements in various situations, including GAN-based image, video, and 3D-aware generation. ", "questions": "- How much memory is increased during training (and inference) with the proposed method, compared with baselines?", "limitations": "The paper appropriately addresses the limitation. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is generally well-written and easy to follow.\n- The proposed method is simple yet effective while showing quite a strong performance in a variety of tasks.\n- The analysis (e.g., which layer should use this concept, the role of this learned transformation) is interesting. ", "weaknesses": "- The paper argues the motivation of this paper is the limitation of AdaIN in handling complex data distribution (e.g., ImageNet). But for me, the motivation seems a bit unclear as that how this learned modulated transformation can mitigate this issue. It seems like the paper claims the proposed method helps to mitigate the limitation of AdaIN that acts at every spatial location equality by learning (possibly) non-local offsets for convolutions. Then one might expect the performance gain would be dramatic in complex datasets such as ImageNet, but it seems the gain is a bit marginal here and rather larger in other datasets or situations. Thus I suspect the improvement of the performance is simply from the increased expressive power than prior GAN architecture, not directly related to AdaIN. \n\n- As mentioned in the discussion section, I think showing whether this type of architecture can improve the performance of other types of generative models, as the proposed method itself does not modify the AdaIN itself and thus can be used in any type of convolutional networks. For instance, Can we expect the better performance of 2D UNet-based diffusion models if we replace some convolutional layers with this method? \n\n- For 3D-aware generation, I expect the paper should show EG3D+MTM also maintains great 3D consistency, rather than just showing the improvement in  FID score.  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. About the motivation of mitigating the limitation of AdaIN.**\n\nThis should be a misunderstanding. AdaIN has become a standard operation in GANs, which helps model the cross-instance variation. This work inherits the AdaIN design and does *not* intend to improve this operation. Instead, our point is that \"the conventional convolution interacts with the feature map at *fixed* locations, resulting in limited expressive power of handling *spatial* variations\". We suspect that this might be the reason why GANs usually work well on datasets where all instances are with a similar shape (*e.g.*, human face), but perform poorly on datasets where instances are with large shape variations (*e.g.*, human body movements in TaiChi). This motivates us to propose MTM, which allows the convolutional kernels to act on the feature map at *variable* locations. It is noteworthy that MTM is also controlled by the latent code (similar to AdaIN) such that different samples will be convoluted at different locations. Experiments suggest that MTM indeed helps improve the performance of GANs on datasets with large shape variations, such as TaiChi (FID from 21.36 to 13.6) and LSUN churches (FID from 4.04 to 2.32). We also included some qualitative comparisons in Fig. R2 (see **the newly uploaded one-page PDF**).\n\n**Q2. About “performance gain on ImageNet is marginal.”**\n\nWith all due respect, we do *not* think improving the FID of StyleGAN2 on ImageNet-128 from 21.14 to 19.16 is marginal. Considering the fact that ImageNet is with complex data distribution, improving the generative performance on ImageNet is not an easy thing, let alone we only introduce a lightweight plug-in module (*e.g.*, MTM) without any other modifications or hyper-parameter tuning. Meanwhile, StyleGAN2 only employs around 30 million parameters, which could be the major bottleneck under such a challenging setting. To verify this hypothesis, we evaluate our approach on a more capable GAN model (*i.e.*, GiganGAN [a]), and confirm that our MTM can also improve the performance of such a strong baseline (with 210 million parameters). We hope that the additional results could help address your concern.\n\n| ImageNet-64 | FID |\n| :- | :-: |\n| GigaGAN  | 7.62 |\n| *w/* MTM | 6.73 |\n\nNOTE: The GigaGAN results are reproduced by ourselves due to the fact that the official implementation is not open-sourced.\n\n[a] Scaling up GANs for Text-to-Image Synthesis. Kang *et al.* CVPR'23.\n\n**Q3. About “the improvement of the performance is simply from the increased expressive power, not directly related to AdaIN”.**\n\nYou are correct that our MTM indeed offers a promising way to enhance the expressive power of generators in GANs, with only a few more parameters and computation overheads (see the table below). Meanwhile, as stated in Q1, this work does *not* target improving the AdaIN operation. In fact, AdaIN has inspired style modulation, which provides a good solution to modeling the cross-instance variation. Its core idea is to use the latent code to control the feature map modulation. However, style modulation usually works with conventional convolution, where the convolution position is shared across instances. Our key motivation is to offer the generator an additional degree of freedom (*i.e.*, the spatial convolutional locations), which is also controlled by the latent code. We are sorry for the misunderstanding we have caused, and we will revise the presentation to make our motivation clearer.\n\n| ImageNet-128 | FID | Training time | Inference time | # Param. (MB) |\n| :- | :-: | :-: | :-: | :-: |\n| StyleGAN2 | 21.14 | 1.0× | 1.0× | 27.78 |\n| *w/* MTM  | 19.16 | 1.2× | 1.0× | 28.55 |\n\n**Q4. About applying MTM to 2D UNet-based diffusion models.**\n\nDiffusion models adopt a different philosophy from GANs to model the data variation. More concretely, there is no concept of \"latent code\" in diffusion models. Recall that, the rationale behind MTM is to use latent code to modulate the convolutional positions of each instance. From this perspective, it is not straightforward to test MTM on diffusion models.\n\n**Q5. About the 3D consistency evaluation on the task of 3D-aware generation.**\n\nThanks. Following EG3D, we calculate the depth error for 3D consistency evaluation. The results are listed in the table below, where a smaller depth error means better 3D consistency. We can tell that our MTM improves the synthesis performance (*i.e.*, measured by FID) without sacrificing the 3D consistency. We also include a new figure (Fig. R1) in **the newly uploaded one-page PDF** to visualize the geometry of synthesized samples from various viewing points. All these results will be included in the revision. \n\n| FFHQ-256 | FID | Depth Error |\n| :- | :-: | :-: |\n| EG3D     | 4.32 | 0.328 |\n| *w/* MTM | 4.07 | 0.336 |\n\n\n**Q6. About the training/inference memory comparison against baselines.**\n\nFollowing the suggestion, we list the GPU memory cost of both training and inference stages in the table below. Thanks to the lightweight design of our MTM, the memory cost barely increases.\n\n| ImageNet-128 | Training (batch size 128) | Inference (batch size 1) |\n| :- | :-: | :-: |\n| StyleGAN2 | 32.17G | 2731M |\n| *w/* MTM  | 32.34G | 2751M |\n"}, {"review_id": "A340OD8CPY", "reviewer": "Reviewer_Sr9W", "summary": "In this paper, the authors equip the generator in generative adversarial networks (GANs) with a plug-and-play module, termed as modulated transformation module (MTM). This module predicts spatial offsets under the control of latent codes, based on which the convolution operation can be applied at variable locations for different instances, and hence offers the model an additional degree of freedom to handle geometry deformation. Extensive experiments suggest that this approach can be faithfully generalized to various generative tasks, including image generation, 3D-aware image synthesis, and video generation, and get compatible with state-of-the-art frameworks without any hyper-parameter tuning. ", "questions": "See Weaknesses", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well written and easy to understand.", "weaknesses": "1. The proposed Spatial Temporal Latent Code Modulation is somewhat similar to deformable convolution.\n2. The comparison methods in Table 1 are too old. StyleGAN2 and StyleGAN3 were published in 2020 and 2021, respectively, and need to be compared with more recent methods published in 2022 and 2023.\n3. In Table 2, the authors also need to compare with more recent methods in 2023 such as [1,2].\n4. The model complexity should be compared with the SOTA methods, such as training time, inference time, model parameters, etc.\n\n[1]Xie, Jiaxin, Hao Ouyang, Jingtan Piao, Chenyang Lei, and Qifeng Chen. \"High-fidelity 3D GAN Inversion by Pseudo-multi-view Optimization.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 321-331. 2023.\n[2]Shi, Zifan, Yujun Shen, Yinghao Xu, Sida Peng, Yiyi Liao, Sheng Guo, Qifeng Chen, and Dit-Yan Yeung. \"Learning 3d-aware image synthesis with unknown pose distribution.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13062-13071. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. About \"the proposed Spatial Temporal Latent Code Modulation is somewhat similar to deformable convolution\".**\n\nAbout the rationale of performing convolution at variable locations, our philosophy is indeed similar to the family of spatial transformation networks, to which deformable convolution belongs. However, different from discriminative tasks, generative tasks require the model to learn data variations. That is the reason why we introduce latent code modulation into our MTM, which is able to link the sampling stochasticity to the generation process. The table below suggests that, without such a modulation, directly applying deformable convolution to GANs drastically harms the synthesis performance. To our best knowledge, we are the first to make the idea of spatial transformation work on generative models, and we also demonstrate its effectiveness and generalizability across various generation tasks, including image generation, video generation, and 3D-aware image synthesis. We hope our discovery could make MTM a basic operation in the future design of GANs.\n\n| TaiChi-256 | FID | \n| :- | :-: |\n| StyleGAN3                  |  21.36 | \n| *w/* deformale convolution | 192.65 | \n| *w/* MTM                   |  13.60 | \n\n**Q2. About \"the comparison against StyleGAN2 and StyleGAN3 are too old\".**\n\nDisagree. Even till now, StyleGAN2 and StyleGAN3 still serve as strong baselines for GAN-related studies, especially for the works that target architecture design. For example, the very recent text-to-image generation works [a][b] are developed from StyleGAN2 and StyleGAN3 as well. As in this work we would like to propose a general and fundamental operation for GANs, using StyleGAN2 and StyleGAN3 as our baselines is fair. Still, following your suggestion, we evaluate our MTM on GigaGAN [b], which is currently the most cutting edge algorithm in this field. The results are shown below, where we can observe that MTM even manages to improve the performance of such a powerful model. We hope that the additional results could help address your concern.\n\n| ImageNet-64 | FID |\n| :- | :-: |\n| GigaGAN  | 7.62 |\n| *w/* MTM | 6.73 |\n\nNOTE: The GigaGAN results are reproduced by ourselves due to the fact that the official implementation is not open-sourced.\n\n[a] StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis. Sauer *et al.* ICML'23.\n\n[b] Scaling up GANs for Text-to-Image Synthesis. Kang *et al.* CVPR'23.\n\n**Q3. About the comparison with more recent methods in 2023.**\n\nWith all due respect, [c] solves a different task (*i.e.*, 3D GAN inversion) from this work, making it hard to compare our MTM (which aims to improve the GAN model itself) with it. The table below includes our comparison with PoF3D [d], where our approach achieves better performance.\n\n| FFHQ-256 | FID |\n| :- | :-: |\n| PoF3D [d]  | 4.99 |\n| MTM (ours) | 4.07 |\n\n[c] High-fidelity 3D GAN Inversion by Pseudo-multi-view Optimization. Xie *et al.* CVPR'23.\n\n[d] Learning 3d-aware image synthesis with unknown pose distribution. Shi *et al.* CVPR'23.\n\n**Q4. \"The model complexity should be compared with the SOTA methods, such as training time, inference time, model parameters, etc.\"**\n\nWe transcribe Tab. 4 of our manuscript below, which indicates that our MTM works as a lightweight plug-in module.\n\n| ImageNet-128 | FID | Training time | Inference time | # Param. (MB) |\n| :- | :-: | :-: | :-: | :-: |\n| StyleGAN2 | 21.14 | 1.0× | 1.0× | 27.78 |\n| *w/* MTM  | 19.16 | 1.2× | 1.0× | 28.55 |"}, {"review_id": "9R8AwOUeqc", "reviewer": "Reviewer_zzjJ", "summary": "The paper introduces learnable modulated convolutions to the literature on GANs. Instead of using standard convolutions with fixed 3x3 kernels, the paper proposes to learn the kernel spatial offsets to allow flexibility in the receptive field of generators. The proposed module can be added to any GAN based on convolutions, and experiments demonstrate it for StyleGANv2, StyleGANv3, EG3D, StyleSV. The module introduces only marginal computational overhead but improves the synthesis of models substantially. ", "questions": "Please comment on the Weaknesses. \n\nIn addition, why are the baselines in Table 1 non-uniform? Would it be possible to show results for all the datasets for both StyleGANv2 and StyleGANv3?", "limitations": "Limitations and Societal Impacts are addressed. ", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- To my knowledge, this paper brings deformable convolutions to the GAN literature for the first time. This method can be easily applied to almost any CNN-based GAN. It introduces only marginal computational overhead but sometimes gives big gains if FID. It can play a role in the further development of GANs, esecially in video synthesis.\n- The paper is easy-to-follow, the ideas are intuitive, and the experiments are extensive. ", "weaknesses": "[Novelty] - I do not understand what is the technical novelty introduced in Sec. 3. Learnable offsets, as shown in Sec. 3.1., have already been introduced in cited prior work (e.g., DCN). The Style Block used for latent modulation in Sec 3.2. has already been introduced in Karras et al. It seems that the method naively applies existing technology (DCN) into existing GANs without any modifications or novel analysis.  It is therefore not clear where is the technical challenge the paper solves.\n\n[Comparison] - The presented motivation for introducing learnable offsets is the fact that GANs are too \"local\" and cannot handle more global dependencies. This problem has been also investigated in prior work (U-Net based discriminator, SAGAN, etc), and some solutions exist. With this motivation, I would expect a discussion and comparison of how well global coherency is preserved in generated images thanks to the introduced method, as well as comparisons to alternatives.\n\n[Comparison] - I would also expect more visualizations of what the module actually learns. For example, how big are the learned offsets usually in comparison to whole feature dimensions?\n\n[Comparison] - As a thought, would a simpler strategy without learnable parameters that increases the receptive field of convolutions (e.g., 5x5, 7x7 convolutions, strides) also help?\n\n[Experiment] - At the moment, there are no visual results to see the qualitative effect of the presented method on studied models (e.g., StyleGANv2 w/o ours vs with ours). I think this is an interesting yet missing analysis. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. About the technical novelty introduced in Sec.3, and \"It seems that the method naively applies DCN into existing GANs without any modifications of novel analysis. It is therefore not clear where the technical challenge the paper solves\".**\n\nDisagree. Unlike DCN, which is originally evaluated on discriminative tasks, MTM is particularly designed for generative tasks, which require to decode the sampling stochasticity to diverse realistic images. To this end, we condition the learning of the transformation on the sampled latent code. As the table shown below, directly applying deformable convolution to GANs drastically harms the synthesis performance. By contrast, applying MTM to GANs brings substantial performance gain. From this perspective, even though DCN and MTM share a similar philosophy, MTM surpasses DCN and works as an general and efficient module in GANs, which is further supported by the experiments on a range of generation tasks (including image generation, video generation, and 3D-aware image synthesis) and datasets. To our best knowledge, we are the first to *make the idea of spatial transformation work on generative models*, which is not that straightforward according to the table below. We hope our discovery could make MTM a basic operation in the future design of GANs. We will add the additional results and the discussion in the revision.\n\n| TaiChi-256 | FID | \n| :- | :-: |\n| StyleGAN3                  |  21.36 | \n| *w/* deformale convolution | 192.65 | \n| *w/* MTM                   |  13.60 |\n\n**Q2. \"I would expect a discussion and comparison of how well global coherency is preserved in generated images thanks to the introduced method, as well as comparisons to alternatives.\"**\n\nIn generation tasks, global coherency should be well reflected by the overall synthesis quality, which is usually measured by FID and FVD. The consistent FID/FVD improvement on various tasks and datasets provides strong support for the effectiveness of our proposed MTM. Following your suggestion, we compare our approach with SAGAN, which introduces self-attention to the GAN generator. For a fair comparison, we use StyleGAN3 as the same backbone and enhance the conventional convolution operation with self-attention or our MTM. The table below demonstrates the superiority of MTM over SAGAN.\n\n| TaiChi-256 | FID | \n| :- | :-: |\n| StyleGAN3  | 21.36 | \n| *w/* SAGAN | 26.83 | \n| *w/* MTM   | 13.60 | \n\n**Q3. \"I would also expect more visualizations of what the module actually learns. For example, how big are the learned offsets usually in comparison to whole feature dimensions?.\"**\n\nDue to the lack of spatial correspondence between feature maps and the final synthesis, it is hard to directly visualize the offsets learned by our MTM, which are at early layers (*i.e.*, small resolution), on the generated image. We have included Fig. 4 in the manuscript, where the offsets are disabled during inference, to give a rough overview of what the module learns. Following your suggestion, we summarize the stats of the learned offsets at the resolution of 36x36 and 52x52. As shown below, compared to the conventional convolution which adopts a fixed receptive field, our MTM offers the model an additional degree of freedom to decode the sampling stochasticity.\n\n| TaiChi-256 | 36x36 | 52x52 |\n| :- | :-: | :-: |\n| StyleGAN3 |   3 ± 0    |   3 ± 0    |\n| *w/* MTM  | 4.7 ± 0.22 | 5.4 ± 0.36 |\n\n**Q4. \"As a thought, would a simpler strategy without learnable parameters that increases the receptive field of convolutions (e.g., 5x5, 7x7 convolutions, strides) also help?\"**\n\nFollowing your suggestion, we conduct an experiment by replacing the 3x3 convolutional kernels in StyleGAN3 with 5x5 kernels. The results are listed below, where we can tell that larger kernels do not always lead to better performance.\n\n| TaiChi-256 | FID | \n| :- | :-: |\n| StyleGAN3            | 21.36 |\n| *w/* 5x5 convolution | 45.94 |\n| *w/* MTM             | 13.60 |\n\n**Q5. \"There are no visual results to see the qualitative effect of the presented method on studied models.\"**\n\nThanks. We include some qualitative comparisons in Fig. R2 (see **the newly uploaded one-page PDF**). We will also add the results in revision.\n\n**Q6. \"Why are the baselines in Table 1 non-uniform?\"**\n\nCompared to StyleGAN2, StyleGAN3 targets the anti-alias property of the generator instead of improving the synthesis performance. In fact, StyleGAN2 has already achieved satisfying performances on many object-centric datasets, such as LSUN church and LSUN cat. But for the challenging TaiChi dataset, where the human body is far from aligned, StyleGAN2 struggles in learning such a complex distribution. Hence, we choose StyleGAN3 as the baseline on TaiChi dataset considering its effectiveness in learning from unaligned data. We also evaluate the generator with respect to its equivariance property to translation (measured by EQ-T, where higher number is better). The table below suggests that our MTM significantly improves the synthesis performance without sacrificing much equivariance property. Due to the limited time, however, we cannot conduct experiments on all datasets with both StyleGAN2 and StyleGAN3.\n\n| TaiChi-256 | FID | EQ-T |\n| :- | :-: | :-: |\n| StyleGAN2 + Fourier features | 23.89 |  9.36 |\n| StyleGAN3                    | 21.36 | 45.87 |\n| *w/* MTM                     | 13.60 | 41.52 |"}, {"review_id": "RcBCcXvewV", "reviewer": "Reviewer_A6mm", "summary": "The paper applies ideas from Spatial Transformer Networks in the context of generative models, specifically GANs, introduces a new module that gives the GAN generator a more natural way to generate content at spatially varying locations. The resulting models are better able to generate content with smooth geometric changes, such as videos, but more traditional non-video datasets also see an improvement. The method seems easy to implement and shows promise when applied to various network architectures and datasets.", "questions": "Line 42 talks about warping the input features, whereas line 112 talks about bilinear interpolation (presumably of the unwrapped input feature map). Figure 1 also seems to indicate that the feature map remains unwarped. How is the operation implemented in practice? The text and figures need to be unambiguous.\n\nSG3 takes great care to use very high-quality resampling operations within the network, in order to minimize aliasing. Does the proposed bilinear interpolation negatively affect the equivariance of SG3? Is the interpolation operation otherwise compatible with the strict signal-processing requirements of SG3? A comment on this would be valuable.", "limitations": "The limitations are not discussed much and should be expanded upon. The paper makes it seem like the proposed method lead to improvements in every single setting that was tried, which is unlikely.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The changes introduced are self-contained and minimally disruptive, and notably do not require changes to hyper parameters or training schemes. The numerical results look good across the board, and the technique seems compatible with various network architectures.\n\nEven though the proposed method seems straight-forward to implement, the promised code release is still a plus.", "weaknesses": "The paper does not impart much intuition to the reader: what exactly are the new capabilities of these models? Are there smooth spatial movements upon latent-space interpolation etc? Are there any situations where the proposed method performs worse and should not be used? In a similar vein, visual comparisons to baselines are missing. Seeing FIDs decrease is nice, but understanding the properties of the new model is even better. Figure 4 is a step in this direction, but more is needed.\n\nFigure 2 is quite uninformative on its own - without comparisons to baseline models, the reader doesn't really learn anything (we already know vanilla SG2 works great for cats, for example).\n\nFailure cases are not discussed in detail - were there datasets where the proposed method did not help, or even hurt performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. \"What exactly are the new capabilities of these models?\"**\n\nCompared to existing GANs that perform convolution at *fixed* locations (*i.e.*, shared by all samples), we offer the generator an addition degree of freedom though performing convolution at *variable* locations. For this purpose, we apply learned transformation to feature maps before feeding them to the convolution operation, where the learned transformation is controlled by the latent code. Such a newly introduced lightweight plug-in alleviates the difficulty of modeling geometric variations within the dataset. Here, we would like to clarify that our design does not intend to enable new functionalities, but it is indeed possible to study whether MTM enables some functional byproducts during inference. Thanks for your suggestion and we will leave it as future work.\n\n**Q2. \"Figure 2 is quite uninformative on its own - without comparisons to baseline models.\"**\n\nThanks. We include some qualitative comparisons in Fig. R2 (see **the newly uploaded one-page PDF**). We will also add the results in revision.\n\n**Q3. \"Failure cases are not discussed in detail - were there datasets where the proposed method did not help, or even hurt performance?\"**\n\nThanks. So far, we evaluate our approach on multiple datasets for image generation, video generation, and 3D-aware image synthesis, and observe consistent performance gain. Still, it is interesting to mention that, on the commonly used FFHQ dataset, applying our MTM seems to have neither positive nor negative effect. One possible guess is that FFHQ contains human faces that are with similar shape and already well-aligned, containing limited geometric variations. Under such a case, using MTM would bring additional computation overheads during training yet achieve on-par performance. We will add the discussion.\n\n| FFHQ-256 | FID |\n| :- | :-: |\n| StyleGAN2 | 3.72 |\n| *w/* MTM  | 3.74 |\n\n**Q4. \"How is the operation implemented in practice?\"**\n\nIn practice, feature warping is incorporated into the convolutional operation. Namely, when a kernel performs convolution, the desired features are obtained through bilinear interpolation, which is equivariant to warping them first. We will revise the text and the figures to clarify this. Also, the core implementation (PyTorch code) is already submitted as the supplementary material, and we will release the entire code to facilitate reproduction.\n\n**Q5. \"Does the proposed bilinear interpolation negatively affect the equivariance of SG3?\"**\n\nThanks. Following the suggestion, we evaluate the generator with respect to its equivariance property to translation (measured by EQ-T, where higher number is better). The table below suggests that our MTM significantly improves the synthesis performance without sacrificing much equivariance property. We will add the results.\n\n| TaiChi-256 | FID | EQ-T |\n| :- | :-: | :-: |\n| StyleGAN2 + Fourier features | 23.89 |  9.36 |\n| StyleGAN3                    | 21.36 | 45.87 |\n| *w/* MTM                     | 13.60 | 41.52 |\n"}]

Author rebuttal:
Thank all reviewers for their valuable comments and suggestions. We additionally included some geometry of synthesized samples from various viewing points and qualitative comparisons in **the newly uploaded one-page PDF**.

Meta-review: 1x A, 1x BA, and 2x BR. This paper proposes the modulated transformation modules that learn the kernel spatial offsets in CNN-based GAN generators, so as to enable increased degree of freedom and handle geometry deformation better. Most reviewers recognize its (1) clear presentation, (2) simple yet effective method, and (3) strong experimental results. The major concerns of the borderline reviewers lie in its modest novelty compared to the previous work DCN. Other concerns including the comparisons to addition baselines have been addressed by the rebuttal. The AC leans to accept the paper due to its solid implementation and empirical studies.
Decision: Accept (poster)