WaGI: Wavelet-based GAN Inversion for Preserving High-Frequency Image Details
Authors: SeungJun Moon
Chaewon Kim
Gyeong-Moon Park
Venue: Submitted to ICLR 2023 (2023)
Keywords: GAN inversion
wavelet transform

Abstract: Recent GAN inversion models focus on preserving image-specific details through various methods, e.g., generator tuning or feature mixing.
While those are helpful for preserving details compared to naive low-rate latent inversion, they still fail to maintain high-frequency features precisely. In this paper, we point out that existing GAN inversion models have inherent limitations in both structural and training aspects, which preclude the delicate reconstruction of high-frequency features. Especially, we prove that the widely-used loss term in GAN inversion, i.e., is biased to mainly reconstructing low-frequency features. To overcome this problem, we propose a novel GAN inversion model, coined WaGI, which enables handling high-frequency features explicitly, by using a novel wavelet-based loss term and a newly proposed wavelet fusion scheme. To the best of our knowledge, WaGI is the first approach to interpret GAN inversion in the frequency domain. We demonstrate that WaGI shows outstanding results on both inversion and editing, compared to existing state-of-the-art GAN inversion models. Especially, WaGI robustly preserves high-frequency features of images even in the editing scenario. We will release our code with the pre-trained model after the review.

Under review as a conference paper at ICLR 2023
WAGI: WAVELET-BASED GAN INVERSION FOR PRESERVING HIGH- FREHUEREQUNNCY IMAGE DETAILS
Anonymous authors
Paper under double-blind review
ABSTRACT
Recent GAN inversion models focus on preserving image-specific details through various methods, e-g., generator tuning or feature mixing. While those are helpful for preserving details compared to a naïve low-rate latent inversion, they still fail features precisely. In this paper, we point out that the existing GAN inversion models have inherent limitations in both structural and training aspects, which preclude the delicate reconstruction of high-frey features. Especially, we prove that the widely-used loss term in GAN inversion, i.e. Lg, is biased to reconstruct low- fwwfereuency features mainly. To overcome this problem, we propose novel GAN inversion model, coined WaGI, which enables to handle high-frequency features explicitly, by using a novel wavelet-based loss term and newly proposed wavelet fusion scheme. To the best of our knowledge, WaGI is the first attempt to interpret GAN inversion in the frequency domain. We demonstrate that WaGI shows outstanding results on both inversion and editing, compared to the existing state-of-ne the-art GAN inversion models. Especially, WaGI robustly preserves high-frequency features of images even in the editing scenario. We will release our code with the pre-trained model after the review.
maintain
INTRODUCTION
Recently, the inversion of Generative Adversarial Networks (GANs) (Goodfellow et al. 2020) has
dramatically improved by using the prior knowledge of powerful unconditional generators Karras
al. 2019 2020 202T) for the robust and disentangled image attribute editing (Abdal et al. Richardson et al. 2021 Tov al.) 2021 Alaluf et al. 202 12 b. Wang et al. 2022a The carly GAN
2019
inversion models mostly rely on per-image optimization (Abdal et al. 2019 2020 Zhu et al.
2020).
which is extremely time-consuming. For real-time inference, the encoder-
GAN inversion
method becomes prevalent (Richardson et al.) 2021 Tov et al. 2021] Allalu et al.j 2021a Moon & Park 2022), which trains encoder that returns the corresponding GAN latent of an input image. The acquired latent from the encoder is desired to reproduce the input image as closely as possible.
However, the encoder needs to compress the image into small dimension, i.e, low-rate inversion. For instance, in the case of StyleGAN2 (Karras et al. 2020). for encoding an image with the size 10242 3 the encoders return the corresponding latent with the size 18 512, which extremely smaller than the original image dimension (about 0.3%). Due to the Information Bottleneck theory Tishby & Zaslavsky 2015 Wang et al. 2022a). an attempt to encode information into small tensor occurs severe information loss, and it deteriorates the image details, i.e., high- figh-frequency features.
To overcome this shortage, recent GAN inversion models propose new directions, such as fine- tuning the generator (Roich et al. 2021 Alaluf et al. 2021b) or directly manipulating the interme- diate feature of the generative model (Wang et al. (202a)) to deliver more information using higher dimensional fcatures than latents, i.e., high-rate inversion. However, results of high-rate inversion models are still imperfect. Figure [ showw the inversion results of high-rate inversion models, Hyper- Style (Alaluf et al. (021b), and HFGI (Wang al. 2022a). Though both models generally preserve coarse features, the details are distorted, e.g., boundaries of the letter and the shape of accessories.
The aforementioned high-rate inversion models remarkably decrease distortion compared to the State-of-the-as low-rate inversion models, i.e., Restyle (Alaluf et al., (221a)). However, this does not mean that distortion every frequency spectrum is evenly decreased. To explicitly check distortion

---

Under review as a conference paper at ICLR 2023
oma HI
om Homa
Original
HyperStyle
HFGI
WaGI
Figure 1: Preserving details at the image inversion. Comparison of inversion results for the noisy image. The zoomed parts are regions that require delicate preservation of details. The existing GAN inversion models including recent high-rate inversion methods, such as generator tuning, egg., HyperStyle, and feature mixing, e.g., HFGI, still struggles to restore segh-freuuency details.
100
Restyle
LH
Hyperstyle
HFGI
_ 10
WaGl
HH 10-2
HL
HH
(a) Wavelet transform of X
(b) Comparison of La of the wavelet coefficients
Figure 2: Wavelet transform and L2 loss from each filter. (a) We plot the wavelet coefficients by each filter at 1st wavelet decomposition. The gray color denotes the zero value. Coefficients from LH, HL, and HH, have significantly high sparsity than the coefficient from LL. (b) We plot the average T-2 of each wavelet coefficient between CelebA-HQ test images and corresponding inverted images by various state-of-the-art inversion models. Due the significant gap between I.2,LL L. and the rest (about 30 times), we display the losses with the logarithmic scale for better visualization.
per frequency sub-bands, we adopt a wavelet transform, which enables to use of both frequency and spatial information. The wavelet transform yields total of four coefficients by passing filters, which are low-pass filter set F; = {T.L.] and high-pass filter set IF, { (L.H, HI., HH). Figure 2a we visualize the coefficients obtained by each filter. In Figure 2b we compare T2 between the coefficients of ground truth images and inverted images, yielded by filter f, i.e., L2.5. While the high-rate inversion models apparently decrease L2.5 for E F,, they marginally decrease or increase L2. for f E F,,, compared to Restyle. the light of this observation, we can argue that the existing methods of increasing the information rate can decrease distortion the low www-frquency sub-band, but are not effective for decreasing distortion on the high-frequency sub-band.
Contributions
First, we prove that the widely used loss term in GAN inversions, i.e., Tz,, is biased on low-frequency by using the wavelet transform. Then, we propose a simple wavelet-based GAN inversion model, named WaGI, which effectively lowers distortions on both the low-frequency and high-frequency sub-bands. To the best of our knowledge, WaGI is the first attempt to interpret GAN inversion the frequency domain. Especially, we propose (i) wavelet loss and (ii) wavelet fusion. First, (i) amplifies the loss of the high-frequency sub-band by using the wavelet coefficients from f Ey. By using (i) training, WaGI is proficient in reconstructing high-frequency details. Second, (ii) transfers the high-frequency features directly the wavelet coefficients of the reconstructed image. Due to the wavelet upsampling structure of SWAGAN, can explicitly manipulate the wavelet coefficients during the hierarchical upsampling.
novel terms for our model:
We demonstrate that WaGI shows outstanding results, compared to the existing state-of-the-art GAN inversion models(Alaluf et al. 2021b Wang al. 2022a). ). We achieve the lowest distortion among the existing GAN inversion models on the inversion scenario. Moreover, qualitative results show the robust preservation of image- mase details of our model, both on the inversion and editing scenarios via FaterFaceGAN (Shen et al. (2200) and StyleCLIP (Patashnik et al. 202T). Finally, we elaborately show the ablation results and prove that each of proposed methods is indeed effective.
2

---

Under review as a conference paper at ICLR 2023
2
RELATED WORK
2.1 WAVELET TRANSFORM
Wavelet transform provides information on both frequency and spatiality (Daubechies) 1990). which crucial in the image domain. The most widely-used wavelet transform in deep learning-bas based image processing is the Haar wavelet transform, which contains the following simple four filters:
1 LL 1 LH [1 -1 HL -HL -1 1 -HHH- 1
1 1
[-1 1 HH
-]
(1)
Since the Haar wavelet transform enables reconstruction without information loss via inverse wavelet transform (Yoo al. 2019), it is widely used in image reconstruction-reltee tasks, e-g-, super-resolution (Huang et al. 2019 In GAN inversion, image- wise details that cannot be generated via unconditional generator, e.g. StyleGAN (Karras et al. 2019, 2020), should be transferred without information loss. Conse- quently, we argue that the Haar wavelet transform is appropriate for GAN inversion. To the best of our knowledge, our method is the first approach to combine the wavelet transform in GAN inversion.
2017 Liu et al. (0188) and photo-realistic style transfer (Yoo et al.)
2.2 FREQUENCY BIAS OF GENERATIVE MODELS
Recent works (Dzanic al., 2019 Liu et al. 2020 Wang et al. 2020) tackle the spectral bias of GANs, in that GAN training is biased to learn the low-frequency distribution whilst struggling to learn the high-frequency counterpart. GANs suffer from generating high- fegh-freuuecc features due to incompatible upsampling operations in the pixel domain (Schwarz et al.) 202T]] Frank et al., 2020) Khayatkhoei & Elgammal| 22022). In order to alleviate the spectral distortions, prior works (Schwarz) al. 102 Durall et al. (2200] Jiang et al. 2021) propose spectral regularization loss term to match the high-freyuency distribution. However, we empirically find that images generated via spectral loss contain undesirable high-frequency noises to coercively match the spectrum density, eventually degrading the image quality (see Appendix I)..
Currently, SWAGAN (Gal et al. 2021) is the first and only StyleGAN2-basee architecture that gen- erates images directly in the frequency domain to create fine-grained contents in the high-frequency range. The hierarchical growth of predicted wavelet coefficients contributes the accurate approxi- mation of spectral density, especially in the spatial domain for realistic visual quality. We adopt this wavelet-based architecture to preserve high-frequency information and mitigate spectral bias.
2.3 HIGH--AATE GAN INVERSION
The most popular high-rate GAN inversion methods are generator tuning and intermediate feature fusion. First, generator tuning first proposed by Pivotal Tuning (Roich et al. (00I)) which fine- tunes the generator to lower distortion for input images. Since Pivotal Tuning needs extra training for every new input, it is extremely time-consuming. Recently, HyperStyle (Alaluf et al. 202Tb) enables generator tuning without training, by using HyperNetwork (Ha et al. 2016). Though Hy- perStyle effectively lowers distortion compared to low-rate inversion methods, still encounters frequency bias. Second, intermediate feature fusion is proposed by HFGI (Wang et al.) 2022a).
HFGI calculates the missing information of low-rate inversion and extracts feature vectors via consultant encoder. The extracted feature is fused with the original StyleGAN feature, to reflect image-specific details into the decoding process. However, HFGI relies on the low-frequency biased loss term even when training the image-specific delivery module, which leads frequency bias.
3 METHOD
In this section, we propose an effective Wavelet-based GAN Inversion method, named WaGL. In Section 3.1 we first introduce the notation and the architecture of WaGI. In Section 3.2 we prove the low-frequene frequencccc bias in L2, using the Haar wavelet transform. In addition, to alleviate frequency bias, we propose a novel loss term, named wavelet loss. Lastly, we point out the limitation of existing feature fusion and propose wavelet fusion, which robustly transfers high-frequency features.
3

---

Under review as a conference paper at ICLR 2023
Trainable module
Discrete Wavelet Transform (DWT)
Pixed module
pated
coscode En
SWAGAN
Go
Updasted layer
Wavelet
Source
Inversion
X
Ffoar
(8 Ayeat)
G2
ManCarr
G
(ancAAggnemt()))
inversion
Fexture fusion
Wavelet loss
Wavelet fusion
DWT
(Wasclett
fnn &
Figure 3: Training scheme of Wavelet-based GAN Inversion (WaGI). Given a pre-trained encoder Eg and generator Gg. we can obtain initial inverted image No- The residual A contains high- fidelity details that Xo misses. The model leverages trainable ADA module to align the residual, which should ultimately be in alignment with Xo or the edited image Xgdit inference. From the aligned Å, we can replenish the missing high-fidelln fideliidlitty information with the two fusion modules Ffeat and Fwave- Fusion with each output is operated in the feature and spatial frequency domains in separate intermediate layers between lower layers G and higher layers G of Go, respectively. The final inversion result X contains rich information without the loss of high-frequency components. Note that ADA, Ffeat, and F Farave are all jointly trained, while Eg and Go are frozen.
3.1 NOTATION AND ARCHITECTURE
Figure 3 shows our overall architecture. Since the goal of WaGl is to retain high-frequency details of an image X, we design our model to two-stage; naïve inversion and image- wise detail addition.
First, the pre-trained SWAGAN generator Go and its pre-trained encoder Eg can obtain low-rate latent, w = Fo (X), and the corresponding naïve inversion X, (w)) Due to the inherent limitation of low-rate inversion, X, misses image-win wise details, denoted by A X Xo. Since the edited image, Xgddtt might have distorted alignment, e-g., a varied face angle, eye location, etc., A should adapt to the alignment of Xedit the editing scenario. Hence, we train the Adaptive Distortion Alignment (Wang et al., (2022a) (ADA) module that re-aligns A to fit the alignment of Xgdst, At the training. due to the absence of edited images preserving image-wiss details, impose self-supervised task by deliberately making a misalignment via a random distorted map to A, Á = RandAugment( (A) (Wang et al. 2022a). The purpose ADA is to minimize the discrepancy between output = ADA( (X,, A) and A.
WaGI combines Å using the low-rate latent w with feature fusion, Freat (Wang et al. 2022a and our proposed method, wavelet fusion, F Fwave. Both Fjeat and Fwove follow the linear gated scheme to filter out redundant information, which return the pairs (scale, shift), i.e., (gfeat: feat) = Fyeat ()) and (gware: (lavee F Fwave (Å), respectively. The pairs of and h adaptively merge desired information from w and Å. Refer to Section 3.3 for the detailed processes of fusions.
The overall loss merges self supervud alignment loss LADA wavelet loss I,K Laae between X and the final image X. and reconstruction loss Limage (AL, Aid, ALPIPS) consisting of the weighted sum of L2, LPIPS (Zhang et al. 2018), and Lia, with weights At.g Aia, and ALPIPS, respectively:
K Liotal = LADA(A, Å) + Tasaage (AL Via, ALrIPs) X) + Awave wave X)X). (2)
We used ede al. 2021 for Fo.
4

---

Under review as a conference paper at ICLR 2023
3.2 WAVELET LOSS
Majority of GAN inversion models use Limage(AAts Xid, AIPPIS) with weights A
Ap > Xid-
for comparing the ground truth and the reconstructed image (Richardson et al. 2021 Tov et al.)
2021 Alaluf et al. 202 la Roich et al., [02T]] Alaluf et al., 2021b; Wang et al.] 2022a). Combining the following theorem and our observation, we demonstrate that L2, which has the highest weight
in L Image, is biased on the low-frequency sub- band in terms of the Haar wavelet transform:
Theorem 1. The following equation holds when >S L, Vf E Ei F.
L2((1,12)= =
Σ
X112,)(41,12)
(3)
PE FUUF
where I and I2 are arbitrary image tensors. Proof is in Appendix A.1
We can derive following simple lemma using Theorem !
Lemma 1.1. When the distributions of pixel-wise differences between I and Ig are i.i.d, and follow N(,, o with μ = O, the following equation holds when X₁ 1.2 Vf E Fj UFF
log [2)
Σ FU
Bl (11. 12)1,
(4)
where C is a constant. Proof in Appendix A.I
In Theorem since L2 reflects T. F, U,F,F with the same weight, it seems fair loss without empirically find that L2, LL of the existing GAN inversion models, e.g., Restyle (Alaluf et al. 2021a), HyperStyle (Alaluf et al. 2021b), and HFGI (Wang et 2022a)). have around 30 times larger value than 2 fe F3 on average. The same logic can be applied to L1, which shows the extreme low-frequency bias while using the same ' for the equation, as shown in Lemma I.I (Please refer to Appendix [A.] for the detailed descriptions). Consequently, we can argue that Afe R3 should be higher than ALL to avoid the oowffequency bias.
frequency bias. However, as shown in Figure 2b
In the light of this observation, we design novel loss term, named wavelet loss, to focus on the high-frequency details. The wavelet loss Lewave between I1 and T2 is defined as below:
I Tawave ( [1, T2) = Σ (2,,((1, se IF,
(5)
In and Tz in equation 5 only pass f E Fn, which discards the sub-bands with the frequency below f2 where Sagg is the Nyquist frequency of the image. However, we empirically find that substantial amount of image details are placed below fawg/2. In other words, Lwave should cover broader range of frequency bands. To this end, we improve Liwave combining with multi-level wavelet decomposition (Liu et al. 2018 Yoo et al. 2019). which can subdivide the frequency ranges by iteratively applying four filters to the LL sub-bands. We improve Lwove with the K -eleel wavelet decomposition, named L.K Laave:
K ΣΣ 12,( LL (,2 + L)),), JE
K wave (1,,
(6)
where (t) stands for iteratively applying LL for times for multi-level wavelet decomposition. L wuve can cover the image sub-bands with the frequency ranges between fnys/2K* +1 and fnyg.
Now we propose that K is especially helpful for training modules that address high-frequency details, e.g., ADA. The existing ADA in HFGI uses L, ( (,, A), but we find that this loss term is not enough to focus on high-frequency features. Since ADA targets to re-align high-frequency details to fit the alignment of the edited image, high-frequency details should be regarded for both purposes; fitting aligument and preserving details. Consequently, we add the wavelet loss to train ADA:
LADA(UI, /2) = Li(I,, 12) + >waae, wave (1,, I2).
We set Awave, ADA 0.1 and K 2 for training.
5

---

Under review as a conference paper at ICLR 2023
3.3 WAVELET FUSION
To prevent the generator from relying on the low-rate latent, i.e,, w E W+. we should delicately transfer information from Å to the generator. For instance, HFGI (Wang et al. 2022a) extracts scale frent and shift feat using A, and fuse them with Pe,, the original StyleGAN intermediate feature layer f,, and the latent at layer ef. we, follows:
F = Ifeat (oF,, we,) + hffat-
(8)
Though feature fusion is helpful for preserving the image-specific details, the majority of image details, e.g., exact boundaries, are still collapsed (see Section 3 for more details). We attribute this to the low-resolution of feature fusion. In the case of HFGI, the feature fusion only applied to the resolution of 64, i.e., E 7. According to the Shannon-Nyquist theorem, the square image I E ## where H W I, cannot store the information with the frequency range higher than Snya VH W 1v2. Consequently, the upper bound of information frequency by the feature fusion of HFGI is fnzg 64v2, which relatively lower than the image size, 1024. To solve this, we modify the feature fusion to be done on both 64 and 128 resolutions, i.c., Ey 7 and 9, respectively, which means fnyg is doubled.
However, a simple resolution increment cannot address the problem totally (see Section 4.3 for more details). Since feature fusion goes through the pre-trained convolution of the generator, the degra- dation of ipage-specific details is inevitable. To avoid degradation, we propose a novel method, named wavelet fusion. Wavelet fusion directly handles the wavelet coefficients instead the gener- ator feature.
Using the hierarchical upsampling structure of SWAGAN which explicitly constructs the wavelet coefficients, wavelet fusion transfer high-frequency knowledge without degradation. Similar with feature fusion, model obtains scale Gwove and shift hwave using Å, and fuse them with the wavelet coefficients obtained by the layer, (Wavelets as follows:
We.. = Swave tWavelets(FF Fe..) + hwave:
where Wi., are the wavelet coefficients of the -w layer.
We empirically find that feature fusion is helpful for reconstructing the coarse shape, while wavelet fusion helpful for reconstructing the fine-grate fine-grained details. Consequently, we apply feature fusion for the earlier layers (f, and 9) than wavelet fusion (fw. 11).
EXPERIMENTS
In this section, we compared the results of WaGI with various GAN inversion methods. We used the widely-used low-rate inversion models, egg., pSp (Richardson et al. (021), e4e (Tov et al., 2021 and Restyle (Alaluf et al., 2021a), together with the state-of-the the art high-rate inversion mod- els, egg., HyperStyle (Alaluf et al. 2021b) and HFGI (Wang et al., 2022a) as baselines. First, we compared the quantitative performance of inversion results. Next, we compared the qualitative re- sults of inversion and editing scenarios. For editing, we used InterFaceGAN (Shen et al. 2020) and StyleCLII (Patashnik et al. 202T) to manipulate the latents. Finally, we analyzed the effec- tiveness of our wavelet loss and wavelet fusion. We conducted all the experiments in the human face domain: we used the Flickr-Faces-HC (FFHQ) dataset (Karras et al., 2019) for training and the CelebA-HQ dataset (Karras al., 2017) for evaluation, with all images generated the high- resolution 1024 x 1024. We trained our own e4c encoder, InterFaceGAN boundaries, and StyleCLIP for each attribute exploit the latent space of SWAGAN (Gal al. 2021).
4.1 QUANTITATIVE EVALUATION
We first evaluated our inversion qualities with the existing baseline inversion methods. Table I shows the quantitative results of each method. We used all 3k images in the test split of the CelebA- HQ dataset and evaluated them with (i) the standard objectives including L2, LPIPS, SSIM, and ID similarity, and (ii) the wavelet loss (equation 6) to measure the spatial frequency distortion in the

---

Under review as a conference paper at ICLR 2023
Ground Truth
pSp
ede
Restyle
Hyperstyle
HFGI
WaGI
Figure 4: Qualitative comparison between inversion results of baselines. The baseline models
including state-of-the-arr high-rate inversion models failed to robustly preserve details, such as accessories and complex backgrounds. In contrast, the inverted images through WaGI showed robust reconstruction of image-wise details, egg., legible letters on the second and third rows.
Method
L2 J
Lwave $
LPIPS
SSIM f
ID sim
pSp |Rihhardson al. e4e Tovv et al. 2021)
2021
0.039
0.369
0.369
0.644
0.810
0.053
0.390
0.419
0.591
0.786
Restyle Alalul el 2021a
0.049
0.389
0.383
0.638
0.783
HyperStyle AAlauu et al.
20216
0.027
0.354
0.332
0.652
0.831
HFGI (Wang 2022b)
0.023
0.351
0.323
0.661
0.864
WaGI
0.011
0.230
0.277
0.753
0.906
Table Quantitative comparison between inversion results baselines. WaGI outperformed all of the baselines consistently with a large margin in the various metries, related to both reconstruction quality and fidelity. Quantitative results showed that our model learned the ground truth frequency distribution most accurately, without loss of identity and perceptual quality.
high-frequency sub-band. Our model consistently outperformed all of the baselines with a large mar- gin. The results empirically prove that our wavelet loss and wavelet fusion the spatial frequency domain are capable of minimizing the spectral distortion and thus improving the reconstruction quality. We note, interestingly, that our model recorded the lowest LPIPS loss despite the explicit manipulation of intermediate wavelet coefficients of the generator.
4.2 QUALITATIVE EVALUATION
We show the qualitative comparisons of inversion and editing results in Figure and Figure 5 respectively. We observed that model produced more realistic quality of inversion results than baselines, especially when images required more reconstructions of fine-grained details or complex backgrounds. For instance, Hyperstyle (Alaluf al. 20216). which refined the weights of the generator image, failed to reconstruct the out-of-distributis HFGI (Wang et al.] (022b) could generate most of the lost details from the initial inversion feature fusion, but restored details were close to artifacts, e,g., letters in the background. In contrast, our method was solely capable of reconstructing the details with minimum distortion consistently.
objects, e.g., earrings, and camera.
Figure 5 shows the editing results for seven attributes, manipulated by InterFaceGAN directions (heen et al. 2020) and StyleCLIP (Patashnik et al.) 2021). Our model consistently showed the most robust inversion results with high editability, while the baselines failed to edit images and restore details simultancously. For instance, in the case InterFaceGAN, the baselines struggled preserve details, e.g., the hat in the second row showed low editability, e.g., HFGI in the third

---

Under review as a conference paper at ICLR 2023
Pose
Old
Young
Smile
Eyeglass
Lipstick
Beard
EM
EM
Input
pSp
e4c
Restyle
Hyperstyle
HFGI
WaGI
Figure 5: Qualitative comparison between editing results of baselines. From the first to the fifth rows, we show edited images via InterFaceGAN directions, and editing results for StyleCLIP from the six e.g., letters, backgrounds, and hats. HFGI. which relatively restores details among baselines, fails edit the image a disentangled way. e.g., distortion eye shapes for editing with "beard". Our proposed method efficiently restores high-fidelity details with satisfactory editability with highly disentangled editing performance, throughout all various scenarios.
rows. Both low-rate and high-rate inversion baselines suffer from preserving details,
row. In the case of StyleCLIP, similar InterFaceGAN, the baselines failed preserve details, e-g., eyeglass at the sixth row, or showed undesirable entanglement, e.g., identity shift of HFGI at the sixth and seventh rows. Overall, both InterFaceGAN and StyleCLIP editing results showed that our model is the most capable of handling the trade-off between reconstruction quality and editability.
4.3 ABLATION STUDY
In this section, analyzed the effecttivnees of each component of WaGI, especially for wavelet
loss and wavelet fusion. In Table 2 we quantitatively compared the performance while adding each component we proposed, to the state-of-the-art GAN inversion model, HFGI (Wang et al. 2022ab. We compared T, through Å for the evaluation of ADA, and Ta and SSIM through X
8

---

Under review as a conference paper at ICLR 2023
A
Á
X
Ground truth
HFGI
+ SWAGAN
Wavelet loss
+ Wavelet fusion
Figure 6: Qualitative ablation of WaGL We compared the performance of ADA and the final inversion from the visualization of Å and X, together with A and X. While the existing state-of- the-art model introduced severe artifacts for computing Å, our proposed methods reduced artifacts effectively and showed better inversion quality.
Configuration
L (1 A)
L2(X, (S)
A HFGI (Wang et al. 2022a)
0.124
0,023
0.653
B + SWAAAA (Gal et al. 2021)
0.124
0.024
0.659
C Wavelet loss
0.100
0.023
0.660
D + Wavelet fusion
0.089
0.011
0.753
Table 2: Quantitative ablation of WaGI. We compared the performances of ADA and the final inversion by adding each component we proposed. Except for altering generators from StyleGAN SWAGAN, our proposed methods, ie., wavelet loss and wavelet fusion, showed remarkable gains in the performance of ADA and the final inversion, respectively.
for the evaluation of the final inversion. Firstly, a simple variation of altering the generator from StyleGAN2 (Karras al.) 2020 to SWAGAN (Gal et 2021) (Config B) made only marginal gains for both ADA and the final inversion. We attribute this gain to the characteristics of SWAGAN, which delicately generates the high-frequency more than StyleGAN. In Config C. training with the wavelet loss achieved remarkably lower Lı (A, Å) (about 19%) than training solely with L1. This compelling that training together with the wavelet loss achieved lower L₁ than training solely with L1 From this observation, we can argue that the wavelet loss is not only helpful for preserving the high-frequency sub-band, but also for reducing distortion the low-frequency sub-band. Moreover, as shown in Figure 66 the wavelet loss indeed helpful for preserving details, e.g. nail colors, compared to Config B. Finally, when wavelet fusion is combined to C (Config D). i.e. WaGI, there was the apparent gain on both ADA and the final inversion. Until C, though we elaborately calculated high-frequency features, i.e., Å, the model did not effectively transfer it to the generator. However, wavelet fusion enables information transfer to the generator, resulting in the improvement of the inversion. The gains can be shown apparently in the qualitative ways, as in Figure
5 CONCLUSION
Recent high-rate GAN inversion methods focus on preserving image-wise details but still suffer from the low-frequency bias. We point that the existing methods biased low-frequee www-frequency sub- band, in both structural and training aspects. To overcome this, we proposed a novel GAN inversion, named WaGI, which explicitly handles the wavelet coefficients of the high-ffreuency sub-band via wavelet loss and wavelet fusion. We demonstrated that WaGl achieved the best performances among the waae-of-the--ad GAN inversion methods. Moreover, we explored the effectiveness of each of the proposed methods through the elaborate ablation study. Since our framework is simple and can be simply reproduced with our released code, we look forward to its wide usage future work.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4432 4441, 2019.
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan++: How edit the embedded images? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8296-8305, 2020.
Yuval Alaluf, Or Patashnik, and Daniel Cohen-Or. Restyle: A residual-based stylegan encoder via iterative refinement. In Proceedings the IEEE/CVF International Conference Computer Vision, pp. 6711-6720, 2021a.
Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit H Bermano. Hyperstyle: Stylegan inversion with hypernetworks for real image editing. arXiv preprint arXiv:21 I. 1.1666, 2021b.
Ingrid Daubechies. The wavelet transform, time-frequency localization and signal analysis. IEEE transactions on information theory, 36(5):961-10M)) 1990.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-2552 2009.
Jiankang Deng, Jia Guo, Jing Yang, Niannan Xue, Irene Cotsia, and Stefanos P 7afeirion. ArcFace: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4690-4699, 2021.
Ricard Durall, Margret Keuper, and Janis Keuper. Watch your up-convolution: Cnn based gener- ative deep neural networks failing to reproduce spectral distributions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7890-7899, 2020.
Tarik Dzanic, Karan Shah, and Freddie Witherden. Fourier spectrum discrepancies in deep network generated images. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Joel Frank, Thorsten Eisenhofer, Lea Schönherr, Asja Fischer, Dorothea Kolossa, and Thorsten Holz. Leveraging frequency analysis for deep fake image recognition. In Proceedings of the International Conference Machine learning (ICML), 2020.
Rinon Gal, Dana Cohen Hochberg, Amit Bermano, and Daniel Cohen-Or. Swagan: A style-based wavelet-driven generative model. ACM Transactions on Graphics (TOG), 40(4):1-11, 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11): (1(11)):39-144, 2020.
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. arXiv preprint arXiv:1609.09106, 2016.
Huaibo Huang, Ran He, Zhenan Sun, and Tieniu Tan. Wavelet-smet: A wavelet- based enn for multi- scale face super resolution. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1689-1697, 2017.
Liming Jiang, Bo Dai, Wayne Wu, and Chen Change Loy. Focal frequency loss for image recon- struction and synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. In Proceedings of International Conference on Learning Representations (ICLR), 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
10

---

Under review as a conference paper at ICLR 2023
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8110-8119 2020.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Process- ing Systems, 34:852 863, 2021.
Mahyar Khayatkhoei and Ahmed Elgammal. Spatial frequency bias in convolutional generative adversarial networks. In Proceedings of AAAI Conference of Artificial Intelligence, 2022.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference com- puter vision and pattern recognition, pp. 2117-2125, 2017.
Pengju Liu, Hongzhi Zhang, Kai Zhang, Liang Lin, and Wangmeng Zuo. Multi-level wavelet-enn for image restoration. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 773-782, 2018
Zhengzhe Liu, Xiaojuan Qi, and Philip Torr. Global texture enhancement for fake face detection in the wild. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, 2020.
Seungjun Moon and GyeongMoon Park. Interestyle: Encoding an interest region for robust stylegan inversion. arXiv preprint arXiv:2209.10811 2022.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text- driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 2085-2094, 2021.
Elad Richardson, Yuval Alaluf, Or Patashník, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: stylegan encoder for image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 2287-2296, 2021.
Daniel Roich, Ron Mokady, Amit Bermano, and Daniel Cohen-Or. Pivotal tuning for latent-based editing of real images. arXiv preprint arXiv:2106.0577 2021.
Katja Schwarz, Yiyi Liao, and Andreas Geiger. On the frequency bias of generative models. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou. Interfacegan: Interpreting the disentan- gled face representation learned by gans. IEEE transactions on pattern analysis and machine intelligence, 2020.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw), pp. 1-5. IEEE, 2015.
Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. ACM Transactions on Graphics (TOG), 40(4):1-14, 2021.
Sheng-Yu Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A Efros. Cnn-generated images are surprisingly easy to spot...for now. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, 2020.
Tengfei Wang, Yong Zhang, Yanbo Fan, Jue Wang, and Qifeng Chen. High-fidelity gan inversion for image attribute editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11379-11388. 2022a.
Tengfei Wang, Yong Zhang, Yanbo Fan, Jue Wang, and Qifeng Chen. fideliy gan inversion for image attribute editing. htttp:://gttaiii - 2022-09-11.
com www/Teno/teteecee WWang//FGF 2022b. Accessed:
Jaejun Yoo, Youngjung Uh. Sanghyuk Chun, Byeongkyu Kang, and Jung- Woo Ha. Photorealistic style transfer via wavelet transforms. In Proceedings of the IEEE/CVF International Conference
Computer Vision, pp. 9036-9045, 2019.
11

---

Under review as a conference paper at ICLR 2023
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pp. 586-595, 2018.
Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing, In European conference on computer vision, pp. 592-608. Springer, 2020.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A. PROOFS FOR THE THEOREM
In this section, we show the proof for the proposed theorem and lemma in the main paper:
Theorem 1. The following equation holds when >5 = =
(2(11,12)
Σ
(1,22((11122)
(10)
IE FER
Proof. Let 11 = (a.z) E Rmxn and 1 = (bij) E Rmx" And for the simplicity of the notation, we define Ct, Gi,j ba.j.
Then,
(2 12) mn
Σ
E
(cz)²
iE)0,1...mm) .e.
Let memm'= = 1221 1 = -1
Then, we can denote L2 Fy F,, as below;
12, 12)
1
Σ
Σ
(021+125+1 02441,2 021,2)+1 + c21,2))
mnn'
(0. jE (0)
I21
Σ
(-C2+122/+1 C21+122 + +262)+1 + C21,2))
m'n'
iE (0.1....m²) jE (E..I......
12
Σ
(-024112/41 + 021+122 -212)+1 + C21,2))
m'n
1e... JE(0,IL....'')
(I,
1
Σ
Σ
(02++12/+1 C21+125 -2,22+1 + 2
m'n'
iE(O.I....m)) JE(O.I.....
We can rewrite L2(11, as;
L2 12) = m'n
4
Σ Σ
2 + 2. + Gi2j41 + E.e2
XE (0. ,e...),....""
We use the following identical equation, which holds for Vz, 2. 2. w R; (x (++y+++w)) (-x --x--+--- ((
4(x² 2
We can obtain the followings:
Σ
(2, T2)
Σ
Σ
4 + + +
m'n'
Ie FUUF
(E(0,1....m jE(0.....mm))
12) = Σ
1.
SE 17 FF
13

---

Under review as a conference paper at ICLR 2023
Lemma 1.1. When the distributions of pixel-wise differences between I1 and 12 are i.i.d., and follow NH,, o²) with μ = O, the following equation holds when Xy = 1' Vf E
log (2,,
Σ FU
Aylog L 12)1
(11)
where C' is a constant.
Proof. Similar with proving Theorem D we can derive followings;
(2)
4
Σ
Σ
|0241125411 + | + |02,24+1 + |e,,2yl
m'n
(0..
1
(11
102++1225+1 + 02/11.5 + 021.2+1 + 2,,
m'n'
iE(
JannneJeelee (0........
(1,
Σ
Σ
| 1-04122211 021+1.2 + c2
w'n'
ie m!.. } (0.
(1,
Σ
Σ
C21+122411 C2441,25
C21 C2r21
m'n'
ie (0.l...m²) ) jE((.I)...n))
II,
Σ
|21+1,2/+1 C21+1,2 C242j+1 + C2.251
m'n'
(E(0.I....m² (E(0......))
Using Ci. (u, can obtain followings:
(c2++12/441 + C24+122 C2..2++ (21,2))
(4
(-02/+12/4 C21+122 +
2212)+1 + +c22)) ~ 40²),
(-02+1,2/41 C24+1.2 C2,22+1 + c2, 40²,
(02+112/41 - C24122 - 0,5,2++1 c2.22) ~ N(0, 402).
According to the properties of half-normal distribution, for (N,,
Blpe|| = V
2 _ 202 +
2
erf(
1b
where erf(x) = 1¹d e- dt.
(202)
Consequently,
2
n12
Elle24122541 + 10241125 + 102.2/+1| + |c21,2i|]
1
202 + 4p erf(
V(202)
Bl|e2412244 021+122 + C2,2j+1 + C2.2yl = -20
2
-4.
2²
20 + 4/ erf(
2p.
v /202)
2
E|| C2441,2111 C2++1.2 + C2.2/+1 + 02x,211
20
π
2
E|| 221112241 + 021+1,2
02,22/+ + 02x221 = 20
n
E 2211112111111 - C21112 - C2122+1 + C2n2y = =20
14

---

Under review as a conference paper at ICLR 2023
Restyle
Hyperstyle
5
10-1
HFGI
WaGl
LL
LH
HL
HH
Figure 7: Comparison of 11 of the wavelet coeflicients. We plot the average L1 cach wavelet coefficient between CelebA-HQ test images and corresponding inverted images by various state-of- the-art inversion models. Due to the significant gap between LT,LL and the rest (about 30 times), we display the losses with the logarithmic scale for better visualization.
Using the condition that c are iid.d
2 _
2μ
(I,,
12] m'n' m'n' (40
=
202
+
4p
erf(
22
2 #² -160
2p
nnd
+ 16μ erf(
V20²
(11, (11,
m'n'
2 (20
-
202 + 4p
x²
ere(
2p
m'n'
π
((02
2² -
_ 20
4p erf(
((20²)
2
2
EL (T, 12) E[ (1, 12)
E Hn [2)]
m'n' m'n' 20 V
201
Since μ = 0, μ erf( # V203 ~ 0. Consequently,
2
4
log E, 12)] = log 16 + log(o
20
4
log E L1,LLL [1, I2)] log + log(o
4
202
log E 12) = ELL
12) = log E|L, 12)] log 2 + log(
log (T,,
Σ
log ET (T,, C.
FE FEFUR FA
Similar with L2. L1 seems fair loss without the frequency bias, which reflects 1 F, U F. with same weights. However, as shown in Figure 7 we empirically find that LI,LL is around 30 times larger than T. T F,, in case of HyperStyle and HFGI. This leads the biased training, which
results in apparent decrease of LI,LL, but almost no gain, or even increment of L i.fe FF, compared to Restyle. Consequently, we argue that L, contains the low-frequency bias, and needs the wavelet loss avoid it.
15

---

Under review as a conference paper at ICLR 2023
A.2 FREQUENCY BIAS
A.2.1 GENERATOR TRAINING WITH SPECTRAL LOSS
(4) Ground Trush
(6) StyleGAN2
StyfeANN with spectral loes
(40 SWAGAN
(6) SWAGAN with spectral Joss
Figure 8: Regression of single image (top row) and spectral density plot of ground truth im- age and generated images (bottom row) trained with/without additional spectral loss. Here, we used the spectral loss introduced in Schwwrz et al. (202T). For both StyleGAN2 and SWAGAN gen- erators, the additional spectral loss induced artifacts to coercively match the frequency distribution. We recommend you zoom in to carefully observe the reconstructed details.
Previous works (Schwarz et al.) 2021 Diang et al. 2021 Durall et al., (020) each propose an ob- jective function to precisely learn the frequency distribution of the training data, which we com- prehensively named them as spectral loss. Jiang et al. (202T) designed spectral loss function that measures the distance between fake and real images in the frequency domain that captures both amplitude and phase information. Durall et al. (2020) proposed a spectral loss that measures the bi- nary cross entropy between the azimuthal integration over power spectrum of fake and real images. Schwarz et al.] (2021) used a simple L2 loss between the logarithm of the azimuthal average over power spectrum in normalized polar coordinates, L...eduued spectrum, of fake and real images. We adopted the spectral loss term Schwarz et al. (2021) for our experiment:
8/ 2-1
L. H/V2
Σ
||og (Š(G()))) log (K(1))
(12)
R=
where Š is the reduced spectrum, G(z) is the generated image, and I is the ground truth real image.
Here, we conducted a single-image reconstruction task, which is widely done (Gal al., 2021
Schwarz et al. (02T)) to investigate the effectiveness of explicit frequency matching in refining high- fidelity details. For StyleGAN2 (Karras et al.. (020) and SWAGAN (Gal et al., (022]) generator, we used the latent optimization (Karras et al. 2019) method to reconstruct single image, each with and without the spectral loss. All images are generated to resolution 512 x 512, with the weight of spectrum loss x0.1 of the original L2 loss.
Figure g shows the reconstructed images and spectral density plots for each case. As seen in Figure 8 aaa,, the spectrum of natural image follows exponentially decay. Using L2 singularly made both StyleGAN2 and SWAGAN generators overfit to the mostly existing low-frequency distribution. (b) StyleGAN2 struggled learn the high-fidelity details, creating an unrealistic image. (d) SWAGAN was capable of fitting most of the high-frequency parts, except created some excessive high fre- quency noise due checkerboard patterns. Though utilizing the spectral loss for both generators (c.c) exquisitely matched all frequency distribution, qualitative results were degraded. Matching the frequency induced unwanted artifacts to the images, and caused the degradation. Due to the absence of the spatial information, the loss based on the spectral density inherently cannot robustly reconstruct frequerry details. Comparably, our wavelet loss minimizes the L1 distance of high-frequency bands in the spatial frequency domain, restoring meaningful high-fidelity features.
A.2.2 INFORMATION IN SUB-BAND OF IMAGES
In Section 3.2 we designed the luvei-level wavelet loss to cover broader frequency ranges than Sayg/" 2 ~ Ingg- In Figure g we show the results of the inverse wavelet transform by omitting the
16

---

Under review as a conference paper at ICLR 2023
T
T
T
O
O
O
Original
A
B
C
Figure 9: Inverse wavelet transform results by omitting various wavelet sub-bands. To check the qualitative image details in each sub-band, we remove the wavelet coefficients between fnyg/ /2 Saya (Config A). fwgg/ /2a 22 (Config B), and / faza/ /2a (Config C). From A to B, severe degradation of visible image details do not occur. However, for B to C or to D, majority of image details are degraded.
wavelet coefficients between f2 ~ fagg (Config A), fnug/22 ~ faya/ (Config B). and fnug/ /2 ~ Snwg//2 /22 (Config C). Though removes the highest frequency sub-bands, iee, Jnga! /2 Inyg- among all configs, we cannot find visible degradation of image details. In other words, information in the sub-band fuge/2 / ~ Sngg is mostly higher than the visible image details. Since the firstly proposed wavelet loss in Jonyy covers the sub-band fnyg/2 fayg. we should extend the range of bub-bands to effectively preserve the visible details. Consequently, we propose a K-level wavelet loss, which enables covering the sub-band fayg/ /2K+ Sayg.
A.3 EXPERIMENTAL DETAILS
A.3.1 TRAINING DETAILS
In our experiments, we implement our experiments based on the pytorch-version code for SWA- GAN (Gal et al. 2021). We converted the weights of pretrained SWAGAN generator checkpoint from the official tensorflow code to pytorch version. We trained our model single GPU and took only 6 hours for the validation loss to saturate, whereas other StyleGAN2-br based baselines required more than days of training time.
Here, we explain the details of our reconstruction loss terms: L2, Lid, and LLPIPS. We leverage T22, as it is most effective in keeping the generated image similar the original image pixel-wise. Lid is an identity loss function defined as
Lid = 1- < R(Go(w)), R(T) >
1-
(13)
where R is the pre-trained ArcFace (Deng et al. 2021) model, and I is the ground truth image. Lid minimizes the cosine distance between two face images to preserve the identity LPIPS (Zhang eet al. 2018) enhances the perceptual quality of the image via minimizing the feature space of gener- ated images and the feature space of ImageNet (Deng et al. 2009) pre-trained network. For training, used weights =1, 1,==0.1, ArPIPP==0.8 respectively.
A.3.2 DATASET DESCRIPTION
Flickr-Faces-HO (FFHQ) dataset (Karras et al. 2019). Our model and all baselines are trained
with FFHQ, a well-aligned human face dataset with 70,000 images of resolution 1024 x 1024. FFHQ dataset is widely used for training various unconditional generators (Karras et al.) 2019
2020 202T), and GAN inversion models (Richardson et al. 2021 Tov et al. [202]] Alaluf et al.
202 bi Moon & Park 2022 Wang et al.] 2022a). All of the baselines we used in the paper the FFHQ dataset for training, which enables a fair comparison.
CelebA-HQ (FFHQ) dataset CelehA-HQ dataset contains 30,000 human facial images of reso- lution 1024 x 1024, together with the segmentation masks. Among 30,000 images, around 2,800 images are denoted as the test dataset. We use the official split for the test dataset, and evaluate every baseline and our model with all images in the test dataset.
hhttps//github.co hhtts://ittub..cm/rrsinnaat http://www.....m////iiiiiiiinana http:://gtthhh.cm//nnnnnagaa///t
17

---

Under review as a conference paper at ICLR 2023
A.3.3 BASELINE MODELS DESCRIPTION
In this section, we describe the existing GAN inversion baselines, which we used for comparison in Section 4 We exclude the model which needs image-wise optimization, such as Image2StyleGAN Abdal et al. (2019) or Pivotal Tuning Roich et al. (202I).
pSp pixcl2Style2pixel (pSp) adopts pyramid (Lin et al. 2017) network for the encoder-based GAN inversion. pSp achieves the state-of-the-an performance among encoder-based inversion models at the time. Moreover, pSp shows the various adaptation of the encoder model to the various tasks using StyleGAN, such image inpainting. face frontalization, or super-resolution.
e4e encoder4editing (e4e) proposes the existence of the trade-off between distortion and the perception-ediaabliity of the image inversion. In the other words, ede proposes that the existing GAN inversion models which focus on lowering distortion, sacrifice the perceptual quality of in- verted images, and the robustness on the editing scenario. e4e suggests that maintaining the latent close to the original StyleGAN latent space, i.e., W, enables the inverted image to have high per- ceptual quality and editability. To this end, e4e proposes additional training loss terms to keep the latent close to W space. Though distortion of pSp is lower than e4c, e4e shows apparently higher perceptual quality and editabilty than ySp.
Restyle Restyle suggests that single feed-forward operation of existing encoder-based GAN in- version models, i.e., pSp and e4e, is not enough to utilize every detail in the image. To overcome this, Restyle proposes an iterative refinement scheme, which infers the latent with feed-forward- based iterative calculation. The lowest distortion that Restyle achieves among encoder-based GAN inversion models shows the effectiveness of the iterative refinement scheme. Moreover, the iterative refinement scheme can be adapted to both pSp and e4c, which enables constructing models that have strengths in lowering distortion, or high perceptual quality-editability. respectively. To the best of our knowledge, Restylepsp achieves the lowest distortion among encoder-based models which do not use generator-tuning methoq Since we utilize baselines that achieves lower distortion than Restyle,sp i... HyperStyle and HFGI, we only use Restyle, to evaluate its high editability.
HyperStyle To make further improvement from Restyle, Pivotal Tuning (Roich et al. (021) uses the input-wise generator tuning. However, this is extremely time-consuming, and inconvenient in that it requires separate generators per every input image. To overcome this, HyperStyle adopts Hy- perNetwork (Ha et al. 2016), which enables tuning the convolutional weights of pre-traimed Style- GAN only with the forward calculation. Starting from the latent obtained by e4c, HyperStyle iteratively refines the generator to reconstruct the original image with the fixed latent. HyperStyle achieves the lowest distortion among encoder-based GAN inversion models at the time.
HFGI HFGI points out the limitation of the low-rate inversion methods and argues that encoders should adopt larger dimensions of tensors transfer high-fidelity image-wise details. To achieve this, HFGI adapts feature fusion, which enables mixing the original StyleGAN feature with the feature obtained by the image-wise details. To the best of our knowledge, HFGI achieves the lowest distortion among every GAN inversion method, even including Pivotal Tuning, except our model.
A.3.4 ABLATION STUDIES
Choice of fusion layer. We additionally provide both quantitative and qualitative ablation results for the inversion performance of WaGI with fusion in different layers. Note that in our main experiment, we apply feature fusion in layers & = 7 and 9, and wavelet fusion in layer &w = 11. Each layer corresponds to fusion of spatial features with resolution 64 64 and 128 x 128, and wavelet coef- ficients of dimension w E R 12x 288 128. From the quantitative results in Table 3 we observed that the feature fusion on two layers f and 9 showed better reconstruction accuracy than on single layer &y = 7. Additionally, wavelet fusion in lower layers (f.. < 11) was not sufficient enough to preserve the high-fidelity details, especially in the high-frequency region i. Wavelet fusion in the higher layer (f... 13) also degraded the inversion performance, which can be more carefully
observed in Figure 10
'IntereStyle (Moon& & Park 2022 achieves lower distortion on the interest region than Restyle psy: but not for the whole image region.
18

---

Under review as a conference paper at ICLR 2023
Feature fusion
Wavelet fusion
L/2 +
Twaoee
LPIPS +
SSIM t
ID sim f
f = 7
0.028
0.359
0.365
0.667
0.796
E 9
0.026
0.356
0.362
0.701
0.830
l =
Ly = 11
0.026
0.325
0.364
0.727
0.847
ty 13
0.024
0.314
0.366
0.727
0.845
0.020
0.327
0.346
0.711
0.849
t 9
0.016
0.289
0.330
0.724
0.880
f - and 9
L. 11
0.011
0.230
0.277
0.753
0.906
13
0.020
0.307
0.342
0.722
0.861
Ablation of the fusion layers for WaGI. We compared the inversion performance of WaGI with feature and wavelet fusion in different layers. Feature fusion layers & and 9, and wavelet fusion on layer w = 11 consistently showed the highest fidelity and reconstruction quality among all scenarios.
Table 3:
Figure shows the inverted images for each scenario in Table is noticeable that fusion a sin- gle layer (a)-(d) failed retain high-frequency details like the hand and hair texture. Comparably, in the case of multi-layer feature fusion (e)-(h), inverted images reconstructed more high-frequency details. Yet, wavelet fusion in the lower layers (e), (g), and higher layer (h) generated unwanted dis- tortions, which eventually degraded the image fidelity. Overall, our scenario (g) empirically showed the most promising reconstruction quality, generating realistic images with the least distortion.
(h)
(c)
Ground Truth
(o)
(9
(g)
(h)
Figure 10: Quantitative comparison of WaGI inversion with fusion in different layers. Each image represents the inversion results for each scenario in Table 3 The first row (a)-(d) displays inverted images with feature fusion single layer fy = 7, with wavelet fusion in layer ly = 7, Ey fy 11, and ly 13, respectively. The second row (e)-(h) displays inverted images with feature fusion in multi-layers f = 7 and 9, with wavelet fusion in layer &, = 7, e. = 9, by 11, and a. 13, respectively. We recommend you zoom in for careful look into the details.
Design of fusion methods. To prove the effectiveness of the wavelet fusion, we compared the performance of WaGT with the model which uses the feature fusion, proposed in HFGI (Wang al. 2022a), instead of the wavelet fusion in the same resolution layer. In Table 4 we compared the performance of models with the following four settings: The original HFGI which uses feature fusion at 1 = 7, HFGI with additional feature fusion at L 9 and 11, WaGI with the feature fusion at ly = 7, 9, and 11, and the original WaGI which uses the feature fusion at 4 = 7 and 9, and the wavelet fusion L 11. First, simply adding the feature fusion to the higher layer is not helpful for improving the model. we change the WaGI method, i.e., change the generator and add the wavelet loss, the performance significantly improved. And after changing the feature fusion the
19

---

Under review as a conference paper at ICLR 2023
Model
L +
Lypane
SSIM t
ID sim
HFGI (e, 7)
0.023
0.351
0.661
0.864
HFGI (L, = 7,9, 11)
0.036
0.377
0.704
0.795
WaGl (f -.9.
0.017
0.302
0.699
0.873
WaG1 (L3 = 7,9 and Lw 11)
0.011
0.230
0.753
0.906
Table 4: Ablation of the fusion methods for WaGI We compared the inversion performance of WaGI with the model which uses wavelet fusion instead of feature fusion. Though changing all the fusion methods with the feature fusion achieves better results than HFGI, still it shows big performance degradation compared to WaGI.
Original
6
Figure 11: Visualization intermediate features during inversion. We visualize the intermediate output of our model for the inversion scenario. Â indeed preserves image-specific details.
11 layer, ie., WaGI, the performance remarkably improved and achieves state-of-the results on various metrics.
A.3.5 ADDITIONAL EXPERIMENTAL RESULTS.
In this section, we show the additional experimental results of WaGI. First, in Figure I1 we showed the visualization results of intermediate features in the inversion scenario. We indeed find that Á reflects the image-wiis details precisely, which results the robust reconstruction in X. Despite the components, e.g., complex backgrounds,
images in IT containing extreme fingers, and big sears on the face, our model preserves every detail consistently. In Figure [2] we provide extensive comparison between baselines and our model on both inversion and editing scenarios. While baselines omitted or deformed the image-wise details, our model robustly preserve details while maintaining high editability.
20

---

Under review as a conference paper at ICLR 2023
Inversion
Inversion
Inversion
Pose
Smile
Age
De-age
Eyeglass
Input
PSp
e4e
Restylease
Hyperstyle
HFGI
WaGI
Figure 12: Qualitative comparison between inversion and editing results of baselines. From the first to the third rows, we show the inverted images. The inverted image from WaGl preserves the most high-frequency details like the texture of clothes and skin. From the fourth and eighth rows, we show edited images of various attributes. WaGI is the only method capable of editing images without loss of tigh-frequency information.
21


Reviews:
[{"review_id": "vd66q4hBeH", "reviewer": "Reviewer_Xb3x", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents and extension of the HFGI method for GAN-Inversion, named WaGI. The author propose to replace the StyleGAN generator with a SWAGAN generator, and apply Wavelet loss on the HFGI ADA's module. In addition, they propose a Wavelet fusion method to fuse the into the latent \\w. The authors suggest that previous methods fail to preserve high-frequency details where their method   succeed. The authors prove that the widely used loss term in GAN inversions is biased.  ", "summary_of_the_review": "I support the acceptance of the paper due to the importance to the community, the nice implementation details and nice results.", "strength_and_weaknesses": "## Strength\nThe paper is a natural extension of the HFGI and SWAGAN papers. The authors addressed critical points, in the continuations of the ideas presented in the mentioned papers to extended the HFGI method. \nIn qualitative evaluation the method show better results on the inversion task, especially on text, which is usually a high frequency signal. In addition, it seems like this method is better than HFGI for editing tasks (visually).\n\n## Weaknesses\nThe authors suggest wavelet losses for high-frequency details reconstructions. Another option to try is to use the HFGI method while adding additional latent layers. The HFGI method fuses only the earlier layer of G_0 and therefore is unlikely that high-frequency features will be preserved.\n\nThe authors did not show ablation study on which of the G_0 layers they use for fusion.\n\nThe authors write in the abstract that they proved the loss used for others models is biased to the low frequency features. First, the analysis show the loss is unbiased. Second, I'm not sure if the empirical evidence they provide in order to show the loss is indeed biased in practice, is sufficiently convincing. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YR38c67Ow60", "reviewer": "Reviewer_FJXP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the problem that existing GAN inversion models fail to maintain high-frequency features precisely. In this paper, the authors prove that the widely used L2 loss term is biased towards the low-frequency components. To solve this problem, the authors attempt to interpret GAN inversion in the frequency domain and propose a method, WaGI, that contains a new wavelet loss and a wavelet fusion module. The proposed method is proved to be the SOTA GAN inversion model.", "summary_of_the_review": "Generally, this paper notices an important obstacle of the existing methods for the GAN inversion task and proposes a method to solve this problem. However, conditioned upon the issues mentioned in [Weakness], I would give a score 6 for now.", "strength_and_weaknesses": "Strength:\n1. This paper notices an important problem of the existing GAN inversion models that L2 loss is biased to reconstruct low-frequency features.\n2. The motivation of each module in the proposed WaGI is reasonable and well explained. \n\nWeakness:\n1. The Theorem 1 proves that the L2 loss seems a fair loss without frequency bias, but then the authors also claim that they empirically find the existence of this frequency bias. What is the possible reason that causes the controversy between the theorem and the empirically findings? \n2. The authors claim that the wavelet fusion module can transfer high-frequency knowledge without degradation due to the hierarchical upsampling structure of SWAGAN. Does this mean that the wavelet fusion module is specialized to SWAGAN as generator, and the wavelet fusion module cannot be adapted to the generator without a hierarchical upsampling structure? I am really curious about the performance of using other models as generator.\n3. It's better to provide figures or some simple experimental results as evidences to support the claim in page 5 that \"a substantial amount of image details are placed below f_{nyq/2}\".\n4. It seems like the ground truth images in Figures 5 and 10 are actually the input images. Please check these figures again.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Q34THF6INc", "reviewer": "Reviewer_pcW6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper find that the normally used L2 loss is biased on low-frequency by using the wavelet transform. Thus this paper proposed the wavelet-based GAN inversion model to effectively lowers distortions on both the low-frequency and high-frequency sub-bands.", "summary_of_the_review": "My major considersion is the ghosting artifact caused by the proposed method.", "strength_and_weaknesses": "Strength:\n1. The performance of this paper is impressive.\n2. This paper is well written.\n\nWeakness:\n1. In table 4, the performance of WaGI is 0.011(L2). In table 6, why the best performance is only 0.015 for model already adding Wavelet fusion?\n2. Artifact problem.  In Figure 6, we can see that even though the proposed method obtain sharper edge, there exists noticeable artifact in the edge part. The artifact looks like ghosting artifact around edgy. Such phenomenon can be also find in Figure 5. I think this artifact is caused by the proposed method. I suggest authors add more discussion about this part.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BSENEpf6Bo_", "reviewer": "Reviewer_ruHv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work aims to solve the inherent limitations in both structural and training aspects in the existing GAN inversion, which preclude the reconstruction of high-frequency features. They show that L2 loss in GAN inversion is biased to reconstruct low-frequency features. To overcome this problem, they introduce WaGI which enables to handle high-frequency features by using a wavelet-based loss term and a wavelet fusion scheme.", "summary_of_the_review": "As mentioned above, the generalization ability of this method is the main concern. Some findings are also not new.", "strength_and_weaknesses": "Strength:\nThe overall idea is quite clear and easy to follow, the inversion results are impressive.\n\nWeaknesses:\n\n1. The main concern is the generalization ability of this method for different GAN models. The designed module of Wavelet loss and Wavelet fusion rely on 'SWAGAN: A Style-based WAvelet-driven Generative Model'. I cannot see how the method can be applied to general GAN models, which will severely constrain the application of this method. This constraint also makes the contribution & impact of this work to be weak.\n\n2. Some findings claimed in this work is not new, and actually well-known and explored, e.g., image L2 loss is biased to low-frequency, similar claim is also delivered in 'A Wavelet-Based Generation Network for Image Inpainting'.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject