SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our MTM significantly improves the generator's performance on video synthesis benchmarks, leading to new state-of-the-art results. The improvement is evident from both FVD16 and FVD128 metrics, indicating enhanced short-term and long-term video synthesis. Additionally, our module improves the single-frame quality, as evidenced by lower FID scores. In summary, our proposed MTM enables the original generator to handle large geometric variation across instances and over time, resulting in significant improvements in video synthesis performance.
SkyTimelapse
TaiChi
YouTube Driving
Figure 3: Qualitative results on various datasets for video generation. Due to the limited space, we subsample frames from the generated videos here.
Which layer requires the proposed MTM? It is possible to replace all convolutions in a given generator with our module, but doing so would significantly slow down the training process. Therefore, we conducted an ablation study to investigate the best trade-off between speed and performance. Prior literature [23, 56] suggests that a synthesized image is represented hierarchically. Specifically, the lower-resolution layers (closer to the latent codes) tend to determine the spatial structure of a generated image, while the higher-resolution layers focus on fine-grained details like texture. This inspired us to question whether replacing all convolutions is necessary. Instead, we hypothesized that empowering the layers most relevant to the spatial structure might be sufficient. To test this hypothesis, we split the layers of the generator into three groups based on the resolution of intermediate representations: low, mid, and high.
Tab. 4 presents the comparisons on ImageNet [10] when replacing convolutions of different layers with our MTM. The results indicate that applying our module at the low-resolution layers achieved appealing performance with acceptable training time. Incorporating more modules into the generator
Table 4: Ablation studies on effect of various layers. All layers of the generator are split into three groups: Low, Mid and High which we replace with the proposed module. Obviously, introducing it into the low-resolution layers results in the best trade-off between performances and training efficiency.
Low	Mid	High	FID↓	CLIP-FD↓	sFID↓	Training time	Inference time	#Param. (MB)
21.14	35.60	4.52	1.0×	1.0×	27.78
✓	19.16	33.92	4.39	1.2×	1.0×	28.55
✓	✓	19.35	34.53	4.42	2.1×	1.0×	28.94
✓	✓	✓	20.02	34.76	4.46	3.2×	1.0×	29.32
Figure 4: Visualization of disabling offsets after training. We set the learnable offsets as zero at the second row. Interestingly, the shape of the main objects in the synthesis becomes quite similar, demonstrating the potential of MTM in performing explicit deformation to deliver various shapes from the blob-like.
did not lead to further improvements but significantly slowed down the training process. We also observed that training became somewhat unstable when using multiple modules on high-resolution layers in practice. Therefore, we decided to use MTM only in the low-resolution layers, which is also how we implemented all image and video generation experiments presented above.
What do the learnable offsets exactly capture? Visualizing the learnable offsets of our MTM is challenging and non-intuitive because it operates directly on intermediate representations with the generator. Therefore, we disabled the learnable offsets to analyze the behavior of MTM. Specifically, after training, we set the offsets to zero. Importantly, even when the learnable offsets are set to zero, the regular convolutions still function normally.
Fig. 4 presents the corresponding results. The first row shows the original synthesis for image generation on TaiChi [41]. When we roughly disabled the learnable offsets, the synthesis seemed to suffer from collapse. Interestingly, the shape of the main objects in the synthesis became blob-like. This demonstrates that our proposed MTM performs explicit deformations/transformations to deliver a variety of gestures that all derive from a blob-like shape.
In this work, we investigate how GANs handle data collection that involves multiple samples with large-scale geometric variation. Since regular convolution is inherently limited, we propose a new method called the Modulated Transformation Module (MTM) that enables explicit and flexible transformations of intermediate representations. Through careful design and comprehensive study, our method significantly improves the quality of image and video generation without additional supervision or computational overhead. We evaluate our method across multiple benchmarks and architectures, demonstrating its effectiveness in various datasets and generative tasks. We believe that MTM could become a fundamental operation in future generative foundation models.
Discussion. While our proposed MTM consistently leads to gains for both image and video generation, there are still some limitations that we would like to discuss. First, we have not explored whether our method could result in further improvements for other generative models, such as auto-regressive models and score-based diffusion models. Fortunately, recent work [39, 21] has already demonstrated
that GANs are comparable to other generative models with the proper scaling scheme. Second, the effectiveness of our MTM on large-scale generation tasks such as text-to-image generation, which often requires a larger GAN, is unknown. Given the current limited resources, we leave this study for future work.
This supplementary material is organized as follows. We first discuss the broader impact of the proposed method in appendix A. Besides, appendix B provides the implementation details including required resources and codebases we used. Fig. A1 and Fig. A2 shows more synthesis. At last, appendix C presents more comparisons and experimental results.
The proposed approach represents a significant advancement in the field of image and video synthesis, which is both a fundamental and practical problem. The introduction of the modulated transformation module has greatly enhanced the capacity for handling geometric variation, making generative models more accessible to the general public. However, this technique may also facilitate the creation of fake content, such as Deepfake, which could have negative consequences. We want to emphasize that we strongly oppose the misuse of this approach to violate security and privacy issues. The negative impact of such misuse can be mitigated by the development of deep fake detection technology.
MTM is developed based on the official implementation of StyleGAN3 and StyleGAN-V for 2D image and video generation. Besides, regarding the 3D-aware image generation, we use the official implementation of EG3D. That is, we strictly follow the data-preprocessing and hyper-parameters of the original ones. Although we train each model on a server with 8 A100 GPUs, it requires no large GPU memory and allows other researchers to reproduce easily.
C More Results
Quantitative results for image generation. Tab. A1 provides more results for the image synthesis. In particular, frame in the video datasets used for video generation is regarded as an individual image. FID is reported for the comparison. Obviously, our method could still improve the baseline generator.
Longer training for the baseline. To investigate the performances introduced by our approach, we train the baseline methods for double training iterations. Tab. A2 presents the comparison where our method remains to outperform baselines, even with longer training. Interestingly, the synthesis of the baseline on ImageNet becomes worse with longer training while our module could significantly improve the FID with the same training iterations and negligible computational overhead, demonstrating the effectiveness of our method.
Table A1: Quantitative results on StyleGAN3. FID (the lower the better) is reported for the comparison.
Methods	SkyTimelapse	TaiChi	YouTube Driving
StyleGAN3	9.37	21.36	8.40
w/ MTM	8.72	13.60	6.16
Table A2: Comparison against longer-tuned baseline. FID (the lower the better) is reported for the comparison.
Methods	Iterations	ImageNet	LSUN Church	LSUN Cat
StyleGAN2	1×	21.14	4.04	6.87
StyleGAN2	2×	22.35	2.82	6.38
MTM	1×	19.16	2.32	5.92
ImageNet-128
TaiChi-256
LSUN Church-256
LSUN Cat-256
FFHQ-256
Figure A1: Qualitative results on various datasets for image generation.
YouTube Driving
Figure A2: Qualitative results on various datasets for video generation.