CosAE: Learnable Fourier Series for Image Restoration
Authors: Sifei Liu
Shalini De Mello
Jan Kautz
Venue: NeurIPS 2024 poster (2024)
Keywords: Fourier Series
Image Restoration
Autoencoder
Primary area: deep_learning_architectures

Abstract: In this paper, we introduce Cosine Autoencoder (CosAE), a novel, generic Autoencoder that seamlessly leverages the classic Fourier series with a feed-forward neural network. CosAE represents an input image as a series of 2D Cosine time series, each defined by a tuple of learnable frequency and Fourier coefficients. This method stands in contrast to a conventional Autoencoder that often sacrifices detail in their reduced-resolution bottleneck latent spaces. CosAE, however, encodes frequency coefficients, i.e., the amplitudes and phases, in its bottleneck. This encoding enables extreme spatial compression, e.g., $64\times$ downsampled feature maps in the bottleneck, without losing detail upon decoding. We showcase the advantage of CosAE via extensive experiments on flexible-resolution super-resolution and blind image restoration, two highly challenging tasks that demand the restoration network to effectively generalize to complex and even unknown image degradations. Our method surpasses state-of-the-art approaches, highlighting its capability to learn a generalizable representation for image restoration. The project page is maintained at [https://sifeiliu.net/CosAE-page/](https://sifeiliu.net/CosAE-page/).

CosAE: Learnable Fourier Series for Image Restoration
Sifei Liu, Shalini De Mello, Jan Kautz
NVIDIA {sifeil, shalinig, jkautz)@nvidia.
Abstract
In this paper, we introduce Cosine Autoencoder (CosAE), a novel, generic Au- toencoder that seamlessly leverages the classic Fourier series with feed-forward neural network. CosAE represents an input image as a series of 2D Cosine time series, each defined by tuple of learnable frequency and Fourier coefficients. This method stands in contrast to conventional Autoencoder that often sacri- fices detail in their eduuedd-rsoluttin bottleneck latent spaces. CosAE, however, encodes frequency coefficients, i.e., the amplitudes and phases, in its bottleneck. This encoding enables extreme spatial compression, e.g., 64 downsampled fea- ture maps in the bottleneck, without losing detail upon decoding. We showease the advantage CosAE via extensive experiments resolution and blind image restoration, two highly challenging tasks that demand the restoration network to effectively generalize complex and even unknown image degradations. Our method surpasses state-of-th teee--f--heeeeee art approaches, highlight- ing capability to learn a generalizable representation for image restoration. The project page is maintained at
flexible-resolution super-
Introduction
Training an image Autoencoder by reconstruction is one of the most commonly adopted approaches for image representation learning. At the heart of such a learning process is creating an information bottleneck: the network first downsamples the input image to lower spatial dimensional features, then upsamples them back reconstruct the input image. The key motivation of an information bottleneck is to enhance network's s ability to capture and preserve key data intrinsic patterns and structures, thereby representation and generalization, which is particularly vital in vision tasks where data quality is critical to performance. Introducing narrow bottleneck in Autoencoder provides an effective way not only to learn general-purpose representations for recognition tasks [1, 2, 3, 4], but also to learn disentangled representations for image synthesis and manipulation [5, 6].
To establish an information bottleneck, most existing Autoencoder networks compress input images into a spatially compact latent space [1, 2. 3, 5, 6]. They learn mid- to high-level representation for classification and attributes disentanglement, by extracting the main shared structures while sifting out any noisy components. However, a narrow bottleneck Autoencoder are rarely directly designed for image restoration task, mainly because the use of a downsampled bottleneck in conventional Au- toencoders often results in the loss of spatial details. Thus, these Autoencoders struggle to effectively represent high-frequency details, such as textures, compared to more coarse-grained structures. In other applications where pixel regression quality make key role, e.g., VQVAE [6] or lattent diffu- sion models (LDM)) [7], researchers address this challenge by striking balance between the spatial compactness of the bottleneck and its capability to preserve details. For instance, in training the Autoencoder [6] used for [6, 7], the downsampling stride is typically capped at 16x. Strides higher than this often result in considerable degradation of image quality [7].
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

spatial expand
amplitude
phase
cosine basis
HCM
(a) Conventional Autoencoder
(b) CosAE with extremely narrow bottleneck
Figure 1. Unlike (a) conventional Autoencoder, (b) CosAB encodes frequeney domain coefficients
tremely narrow bottleneck feature maps. Via a basis construction module, faithfully represents both the global structure and fine-grained details of an input image.
In this paper, we propose a novel Autoencoder directly applicable for image restoration not only does it possess an extremely narrow bottleneck to boost representation capability, but also faith- fully preserves high-fidelity details. Our solution, the Cosine Autoencoder (CosAE), draws inspira- tion from Fourier transform principles, which demonstrates that finite signals can be depicted using set of harmonic basis functions. We follow this principle to encode frequency domain coefficients, including amplitudes and phases, and integrate them with a set of Cosine basis functions before being decoded back to the original image (Figure. (b)). This is achieved through Harmonic Construction Module (HCM) that spatially expands the compact feature into series of harmonic functions, following the formulation of the classical Fourier series. Unlike downsampled feature maps from the conventional Autoencoders, we found CosAE, the representation with frequency coefficients can be extremely compressed spatially. Yet, these representations can effectively recon- struet fine- oeee-graied details when combined with high-frequency basis functions, e.g., the cosine basis in the 2nd row in Figure. 1.
As generic Autoencoder, CosAE can be potentially utilized across various applications where tradi- tional Autoencoders are applicable. In this paper, we demonstrate its capability for two challenging pixel reconstruction tasks: (a) flexible-resolution super-resolution, and (b) blind image restoration. We note that for tasks such as super-resolution, image denoising, and image enhancement, is com- monly believed that maintaining as many details as possible in the input without using a bottleneck will lead to better restoration [8, 9, 10]. However, the absence of information bottleneck can potentially limit the capacity for representation. In contrast, the narrow bottleneck CosAE en- courages the alignment of the distributions of image representations under different types and levels of distortions. Unlike most previous work [11, 12] where distinct networks are needed for restoring images of different types of degradation, single CosAE network demonstrates strong generaliza- tion capabilities across a wide range of degradation. We summarize our contributions as: (i) We propose CosAF, novel generic Autoencoder featuring extremely narrow bottleneck while pre- serving high-fidelity image details. (ii) We introduce a basis construction module, a key component of CosAE, that decomposes and represents not only the coarse structure but also the fine details of an image. (iii) CosAE can effectively restore images without the need for degradation types or up- sampling ratios during inference. This capability makes it highly scalable in restoring images with unknown or complex types of degradation.
2 Related Work
Autoencoder Networks. Training Autoencoders [2] is a common approach for representation learn- ing. By creating a bottleneck representation, Autoencoders abstract input data into generic visual representations. This method is used in Denoising Autoencoders (DAE)[1] for image restoration and Variational Autoencoders (3)E)[3] for image synthesis. Recent works have shown that Autoen- coders can learn general representations for downstream tasks [13, 14, 15, 16, 4] and disentangled representations for image synthesis [17, 18, 5, 19]. While bottleneck representations have proven effective, recent works RGB pixel reconstruction tasks [8, 9, 10] typically aim maintain high spatial resolution during auto-encoding to preserve input details for better image reconstruction. In contrast, work revisits the bottleneck concept, making it extremely narrow to achieve both generic representation and high-fidelity reconstruction.
Image Restoration. General image restoration aims to recover and enhance damaged de- graded images to improve their visual quality, clarity, and perceptual information. Within it, super- resolution [20, 21, 22, 23, 24, 8] and image denoising [25, 26, 27, 28, 29, 30] are the two major

---

(2e)XIX]
A
(Ma,Ps)
exXT
spatial expand via Eq.(2)
(A
bottleneck
Figure 2. Overview of CosAE. CosAE contains an encoder that compresses an input image into narrow bottleneck. Each bottleneck vector [An, dn] is translated into a group learnable harmonic functions H of size T x T, according to Eq. (2) (see the box). A decader then reconstructs the input from the learned harmonic functions. Note A. equals to ((.....)) in Eq. (2) for simplicity.
sub-proble problems. Unlike conventional approaches that rely on specific corruption operators or sepa- rate networks for different levels of degradations, recent advancements have focused on learning more generalizable representations to handle complex and unknown degradations. Among them, flexible-resolutionn super-resolution [31, 32, 33] and blind image denoising [31, 34, 35] are widely studied. Our work focuses on these two applications. While most prior methods use encoders with minimal down-sampling, we show that a narrow bottleneck combined with continuous representa- tion modules like LIIF [31] achieves consistent reconstruction across various corruption levels. In blind image restoration, recent networks rely on priors from pretrained models [36] or learnable dictionaries [37, 38], which complicate the training process and require additional measures to bal- ance realism and fidelity. In contrast, CosAE maintains the simplicity of conventional Autoencoder training without such dependencies.
Fourier Space. Recently, methods like SIREN [39] and Fourier feature mapping [39, 40, 41, 42] have been used to preserve high-frequency details in networks. SIREN [39] utilizes the sinusoidal function as an activation layer to reconstruct high-frequency components, while other approaches [40, 41, 42] transform inputs into Fourier coefficients or feature maps to enhance detail retention. Frequency representations are proposed for texture synthesis [43, 19] and image super-resolution [32, 44). Among image super-resolution works, the Fourier module of LTE [32] focuses on high- frequency residuals via a skip-link, limiting its capability to learn compressed frequency domain representations, while OPE-SR [44] employs parameter-free decoder for pixel space decomposi- tion using ID Fourier basis, unlike CosAE, which uses a 2D Fourier basis to enable more compact information bottleneck representation.
3 Approach
Similar to conventional Autoencoders, CosAE has an encoder to produce a bottleneck latent repre- sentation (Sec.3.2) and decoder reconstruct the input image (Sec.3.5). Within the bottleneck, we introduce harmonic construction module (HCM) (Sec.3.3) to spatially expand the latent repre- sentation into Fourier series (Figure.1 (b)). We start by revisiting the classic 2D Fourier transform theory [45] and its harmonic functions (Sec. 3.1). Furthermore, we delve into the details of the proposed HCM, showing how to integrate the classic Fourier transform into latent space in neural networks.
3.1 Preliminary of Fourier Series
Fourier series is an effective tool to represent periodic or finite signals with a group of sinusoidal functions. Formally, the periodic extension of a finite ID signal {x(t), T3 can be represented with Fourier series in the amplitude-phase form:
xx() ~ Ag
Ek=1 Hi
H(k, t) A cos( 2 kt )b)
where & denotes the discrete frequencies, (k: 1) denotes the tth harmonic component the signal. Within H.(k t), we have the scalars Aş and dy as the amplitude and the phase shift. Specifically, Ao

---

denotes the amplitude of the DC component. The Fourier series in Eq. (1) can be easily generalized n-dimensional cases. In the context of a finite 2D signal, e.g., an image, the harmonic function with frequencies (u, v) corresponding to the x and y dimensions is denoted as:
2T (ur + uy) ((a,e)]
H(u, v, A(u,o) 0os
T
where x < T and y < T. We denote cos + (ur + vy)] as a Cosine basis function, and the full term H(u v, z, as harmonic function.
In our 2D domain, the amplitude and phase of an input image can be derived using the Fourier transform (e.g.. 2D FFT) [45]. However, Fourier coefficients are neither learnable nor directly com- pressible for forming an information bottleneck, and thus cannot directly facilitate image restoration or visual representation learning tasks. Instead of applying FFT, we learn the amplitude and phase, denoted as A(u..) and (w,,e)) as bottleneck feature maps via an encoder, allowing flexible dimension configuration. To reconstruct the original image, we use Harmonic Component Modeling (HCM) to generate a set of learned harmonic functions (Eq. (2)), and decoder acts as a learnable summation operator, mimicking the reconstruction process in Eq. (1)-
The key insight is the compactness Fourier coefficients in representing images, even those with intricate details. For example, a complex texture image may have only a few significant frequency components. By making Fourier coefficients a learnable representation, we can design an Autoen- coder with an extremely narrow bottleneck.
3.2 Encoding Fourier Coefficients
The encoder of CosAE is responsible for learning Fourier Coefficients of the bottleneck latent space. Given a 2D square P x P image patch p.. the encoder compresses the input signal into a bottleneck feature vector with the dimension of 2 x c, representing pairs of corresponding amplitudes and phases,
[ E R2 o R = E(4p).
(3)
To generalize to images of any resolutions, in practice, we design the encoder with the downsampling stride setting as P, so that A and á are both channel-dimensiional feature maps with its spatial resolution P/P as that of the input image. In CosAE, relatively large stride P will be adopted so that the bottleneck can be very small: for face images, we maintain a downsampling stride as P = 64 (resulting in a ID bottleneck of R2xe for 64 64 patch, see Eq. (3)), while for natural images, we use a stride as P = 32. Remarkably, CosAE is still able to faithfully reconstruct details through combining these coefficients with the Cosine basis functions as latent feature maps, as introduced in the following.
3.3 Constructing Harmonic via HCM
We introduce a harmonic construction module (Figure. 2, middle box) that operates on the bottleneck to transform the predicted coefficients coming from the encoder into a set of 2D harmonic functions, i.e. H in Eq. (2). Taking the 2D square patch I, as an example. The encoder processes this patch to yield e pairs of amplitudes corresponding Cosine basis functions. Each basis is T x T 2D Cosine waveform, where T can be flexibly adjusted to achieve the desired output resolution. Thus, each pair of Fouricr coefficient A(u,)) and ((,,v) will be spatially expanded to a T T harmonic function, according to Eq. (2) (the spatial expansion is visualized as transitions between the two red blocks, in Figure. 2). The c harmonic functions, formulated as T X x feature map, are then decoded to reconstruct the image patch Ip- Specifically, T the size of the Cosine basis functions can be customized during training, e.g., for the task of
E R and phases & E RS. These are then combined with e
super-resolution. We introduce the details in Sec. 4.1.
Notably, T shouldn't be too small to formulate a valid 2D Cosine waveform. To balance with model efficiency, we set T = P/2, i.e., T 32 for face images and 7 = 16 for natural images, where the decoder has an upsampling stride setting as 2. The e pair of frequencies (u E R v RR), which determine the Cosine basis functions, are also learnable, as detailed in Sec. 3.4.

---

Ground Truth Figure 3. Qualitative evaluation for face FR-SR. While models can super-resolve input image with any ratio between and 8, we show the extreme case: to upsample 32 input a 256 output, i.e., x8. compared to both LII [10] and the CosAF variants stated at Sec. 4.2.
Input
LIIF-4x
LIIF-84x
CosAE-nocos
CosAE-imcos
CosAE
3.4 Learning Frequencies as Network Parameters
Many prior works learn the and also from encoder [19, 32], similar to the Fourier coefficients, ie., the frequencies are conditioned on the input signal. However, the optimal solution? To answer the question, briefly revisit the properties of the harmonic function.
Harmonics. There are two key principles for setting frequencies signal processing. First, fre- quencies (u, v) in Eq. (5) are sampled in increments inversely proportional to the signal's ' length. Second, frequencies must satisfy the sampling theorem avoid aliasing. To determine the range of the frequencies, we adopt the Fourier transform F-)) to the harmonic function (we use 1D case, and eliminate the amplitude and phase for simplicity).
F(H+(W))
1 cos( 2 kl)e 12 dt
(4)
n =|0(2T(( +)) 8(2 41),
ie., the bandwidth of the harmonic B k/T. According to the Nyquíst- theorem, for sampling rate fe. perfect reconstruction is guaranteed for band-limit B = 2 signals, such as images, where the sampling interval is f. = 1 (pixel), can be restricted to < T to prevent aliasing and ensure an accurate harmonic representation.
fa For discrete
Our solution. Based on the design the classic Discrete Fourier Transform (DFT) described above, initialize pairs of (u, v) within the range [0, T/2). However, for more flexible designs, the (u, v) set should adapt to different needs, such as increasing frequency pairs for higher resolution reducing them for efficiency. In such cases, uniform sampling can result in sparse coverage. For example, with T = 64 and the number of basis maps set to 64, both u and v are sparse, sampled as [0, 4, 8,.,8.... 32). Since natural image frequencies are not uniformly distributed. we make (u, v) learnable parameters shared across all input images, enabling effective modeling of diverse frequen- cies during training. Compared prior approaches that adapt frequeeciess based on individual input images [32, 19], our design aligns more closely with classic principles by using globally shared frequencies that span full range. Such design also addresses common limitation in previous models emphasizing lower frequencies that dominate natural images, allowing for a more balanced and comprehensive frequency representation.
3.5 Decoding Harmonic for Reconstruction
Once the basis (H)TxTxe obtained, introduce decoder network D map them RGB pixels:
X DDD((( v,z,y))): (u < T/2. <
(S)
The architecture can be flexibly designed. E.g., our framework allows for either implicit representa- tion networks [10, 39, 32, 33), sampling only small portion of pixels to decode during training, or convlutiionllraasffoneee--b
decoders that use the full latent space, accommodating additional modules like discriminators for full-sized outputs.
5

---

Table I. FR-SR evaluation the val splits of FFHQ+CelebA faces (15k images). We upsample LR images of different input resolution (row-2), all to 256 x 256. The upper block displays methods using only loss, while the lower part show those trained with full objectives, denoted with (G). The gray block in the bottom presents results for ablation studies (Sec. 4.2).
FID-15KL
LPIPS)
PSNR (dby)
SSIM
LR-res
32
64
48
64
128
32
48
128
32
64
128
LIIF-64x
N/A
0.41
0.38
0.37
0.37
26.03
25.23
23.95
26.37
0.68
0.71
0.73
0.74
1.TE-64x
N/A
0.48
0.39
0.33
0.21
22.94
24.80
26.07
29.23
0.62
0.69
0.73
0.84
TTNSR-64K
N/A
0.47
0.39
0.36
0.36
23.10
25.
26.05
26.30
0.63
0.70
0.72
0.73
COSAF.
N/A
0.36
0.32
0.28
0.26
24.34
26.18
27.52
28.51
0.70
0.75
0.79
0.81
LIIF-4x
34,93
11.89
0.27
24.68
27.94
0.75
LIIF-64x (G)
18.91
15,50
15.05
13.94
0.30
0.27
0.26
0.25
22.59
23.24
23.59
23.57
0.62
0.64
0.65
0.65
CosAE
12.81
8.67
8.12
7.86
0.24
0.19
0.17
0.14
23,65
25.37
26.66
27.51
0.67
0.72
0.76
0.78
CosAE-nocos (G)
13.1
13,44
13.54
12.75
0.31
0.30
0.29
0.29
20.92
21.24
21.35
21.34
0.57
0.58
0.59
0.59
CosAE-imcos (G)
10.85
10.81
10.7
10.72
0.31
0.30
0.29
0.29
34
20.64
20:77
20.77
0.56
0.57
0.57
0.57
CosAE-FT (G)
16.55
11.78
0.22
23.55
26.30
0.73
3.6
Network Implementation
FFT for Additional Input. We found that incorporating the 2D FFT of the input image extra channels enhances performance, as it intuitively aligns with the encoder's 's task of capturing Fourier coefficients in the latent space. Specifically, we apply the FFT to generate real and imaginary 2D maps for each RGB channel, which are then concatenated with the original RGB channels, yielding 9-channel input map for the network.
Network architectures. We build our encoder and decoder based on [6], which has been widely utilized for auto-regression and diffusion models [6, 7, 46]. To enable larger downsampling stride in the encoder, we add extra ResNet blocks and position the attention layers near the bottleneck. Correspondingly,
employ relatively lightweight decoder with upsampling factor of 2. We train CosAE on 256 x 256 patches for all the downstream tasks, while setting e = 256 for all the experiments. Please refer to Sec. 3.2 and 3.3 for other hyper-parameters
Notably, we process each bottleneck feature vector independently, allowing inference on images of any size. This is achieved by replacing global attention with window attention [47] within each T x T feature cell, the decoder. Likewise, during inference, for inputs with different size than 256 x 256, we employ window attention as well on each non-overlapping 256 x 256 region.
Objectives. We follow [6] train CosAE with LI loss and a perceptual loss [48] as the reconstruc- tion objectives, denoted as Cree = L11 + Lipips. In addition, we apply patch-based discriminator [49], denoted LGAN, enhance the visual quality of the reconstructed image. The entire network, along with the pairs of learnable frequencies {u, m} E R2xe, is trained with L = Free + ALGAN- We keep the same settings for the adaptive weight À as [6]. More details are presented in the sup- plementary material.
Experimental Results
In this paper, we focus on the applications of dexible-resolution super-resolution and blind image denoising, leaving other potential auto-enccderr-reltte applications for future research. We evaluate both tasks in the domains of face and natural images.
For both face and natural image domains, we train the CosAE with the upscaling factor from 1 to 8. During training, we sample a high-resolution (HR) image IRR R x3 from the training set, and downsample it with uniformly sampled ratio E (1, 8) to obtain the low-resolution (LR) input image LR R x3 To train super-resolution model with flexible output resolutions, sample an upscaling ratio as u E (1, to let CosAE upscale the input image to Lout E R Hx # W x3 This is conducted by setting T = =T manss where Tmaz refers to the T in Sec. 3.3, which upscales the LR image to the original size of the HR image. Note that we always (bilinearly) resize the LR image the original size of the HR image fit the Encoder of CosAE. The size of the HR images fixed to 256x x 256 during training. achieved either by resizing (e.g., for face images) or by random cropping (e.g., for natural images).
then
Our experiments also indicate that strictly limiting frequencies (u, u) to prevent aliasing NOT essential for effective performance. To validate this, we performed an ablation study (see Ap-

---

LIIF-4x
LIIF-32x
CoSAE
BX
8x
Figure 4. FR-SR DIV2K [24]. We showcase 8x results similar to Figure. 3. CosAE faithfully reconstructs fine details, for both regular (buildings) and irregular (skin and food) textures. Zoom in see details.
pendix B.2) where we introduced a channel dropout variant during training, strictly constrained min(u u. v) < T/2, and observed the resulting outcomes. Further frequency analysis and discussions on limitations, including out-of-distribute generalization and common artifacts, provided in the Appendix.
4.1 Flexible-resolution Super-resolution (FR-SR)
FR-SR has been intensively explored via implicit representation [10, 32, 33), in which the output resolution can be flexibly configured by setting the size of the coordinate map.
4.1.1 FR-SR for Face Images.
We train the face model using cropped faces from the training splits of FFHQ [50] and CelebA-HO
[51, 52], utilizing the official training and validation splits. To evaluate the output images, we use
FID [53] and LPIPS [48] metries, along with PSNR and SSIM to compare with methods that use pixel regression objectives [10, 32, 33].
Baselines. We compare CosAE with local implicit representation methods LIIF [10], LTE [32], and ITNSR [33], which generate aubitrar-rresluttion outputs by predicting RGB values at specific coordinates. For a fair comparison, we re-implemented these methods using the same Encoder as CosAE to ensure consistent representation capacity across models. Among those, LIIF is direct counterpart to our approach, approximating the replacement of high-frequency components (Cosine basis functions) with plain coordinate maps. To ensure fair comparison, we align LIIF by replacing its implicit decoder with ours (CNN with self-attention) and adopting the GAN loss.
To compare with LTE [32] and ITNSR [33], we replace their Encoders with ours but keep their original Decoders intact as they are key components. We do not add GAN loss as it does not enhance results. LTE does not have narrow bottleneck due to skipped link before the output. We align the input settings for all methods match ours while keeping other components as per their original versions, denoting the modified networks as "-64x" in Table 1. Notably, during inference, CosAE performs blind super-resolution without needing the upsampling ratio, unlike other methods.
Comparison with the State-of-the-Art Methods. We demonstrate significantly improved per- formance with CosAE, compared with the baselines of LIIF-64x, LTE-64x and ITNSR-64x, both qualitatively (Figure. 3) and quantitatively (Table. 1). In Table. 1, networks trained only with LI loss placed in the upper block and those trained with full objectives (Sec. 3.6) in the bot- tom rows. Our results are significantly better, especially in terms of large-ratio (c.g., x8 and x6) super-resolution, all the matrices.
4.1.2 FR-SR for Natural Images.
To perform FR-SR training on natural images, we initially pretrain CosAE on the ImageNet [54] training split, using the same approach as the face model. However, we encountered distortions in the reconstructed images when applying the 4 x 4 bottleneck same as the face model. To
7

---

Input
GFPGAN
RestorFormer
CodeFormar
CosAE (ours)
Figure 5. Comparisons with the STOA face restoration methods. Results shown for both real- world (WebPhoto- Test, row) and synthetic degraded images (CelebA- Test, row). CosAE restores more high- frequency and realistic facial details compared to other methods.
address this, we adjusted the bottleneck to 8 with Tmar = 16. Subsequently, we fine-tuned the model using the same training settings combination of the DIV-2K [24] training set and all the images from Flicker2K [55] (referred to as the DF2K dataset). We compared the CosAE with LIIF [10] aligned in the same way training the face model, denoted as "I.IIF-32x" as the Encoder for natural images has a x2x downsampling stride. Both models were trained with the same datasets, input settings, and objectives.
To evaluate the performance, we used LPIPS on the DIV-2K validation set, considering the adoption of the discriminator. The FID score [53] is not applicable due to the testing set is small. We keep the same super-resolution up-scaling factor as that of the face (in Table. 2 we present the down-sampling factor instead of the resolution, given images of DIV2K have different sizes between individuals). Similarly, CosAE demonstrates significantly better performance than the others. Qualitative results shown in Figure. 4, and Figure. 9, 10 in the Appendix.
4.2 Ablation Studies
To validate the design choices, we perform the Ablation Studies the task of FR-SR for face images, which focus on the following aspects.
To evaluate if the Cosine basis functions play key role in improving the performance, other than LIIF that replaces them with coordinate maps, we further replace them with plain upsampling layers, while preserving all the other components and training recipes. We denote as nocos.
W or w/o Cosine Basis Functions.
Encoding Input with Cosine Basis. To see if the Cosine basis functions can be placed other than the bottleneck, encode the input image with Cosine basis functions, i.e., similar to Fourier transform to multiply each basis with the signal, but without the summation operator. We keep the bottleneck imcos.
the same design as the nocos model. We denote
Fourier Transform. Instead of using the Decoder to mimic the summation operation in Eq. (l), we directly over the frequencies, i.c., the channel dimension of the bottleneck. before the Decoder. This is equivalent to performing the Fourier transform, instead preserving the Fourier series, in the bottleneck latent space. We denote it as FT.
Results. We compare the ablated models with the others trained with full objectives, in Table. 1. Compared to CosAE, all the variants show degraded performance nocos and imcos exhibit visually pleasant faces with rich details, but keep less fidelity, egg., the eye- lid or lip color may completely changed, see Figure. 3. FT shows lower quality local region details given it does not fully leverage the capabilities of the decoder (see Figure. 12 Supp., due to space limitation).
all the matrices. We observe

---

Input
LDM
CosAE
Ground Truth
Figure 6. Comparisons with LDM on 1x super-resolution on ImageNet. comparison, the CosAE pro- duces more fine-grat one-grained details, in terms of both the structure and the texture. Zoom in to see details.
Table 3. Quantitative comparisons on CelehA-Tert Our CosAE shows better performance on almost all the metrics, except for FID score (see the possible 4.3).
Table FR-SR evaluating of FR-SR on DIV2K,
e.g. 888 LPIPS
validation set
N/A due reference set).
Mathod
FTD.
PSNR
SSIM
LPIPS
IDD,
up-ratio
x8
x5.33
x4
x2
Inpot
132.69
24.96
0.66
0.50
0.93
DFDNet
52.92
24.10
0.61
0.45
0.76
LIIF-4x (G)
0.41
0.33
0.30
0.26
PULSE
1.20
LIIF-32x (G)
0.42
0.34
0.29
0.21
GFP-GAN
42.39
24.46
0.67
0.36
0.60
CosAE (G)
0.33
0.25
0.18
0.11
RestoreFormer
41.45
24.42
0.64
0.36
0.57
CodeFormer
52.43
22.18
0.61
0.30
0.60
CosAE
52.03
25.65
0.70
0.27
0.53
Wide or Narrow Bottleneck? Is narrow bottleneck better? Since CosAE does not support wider bottleneck variant due to certain size T needs to be maintained for valid Cosine functions (Sec. 3.3). Thus, we modify "wide" LIIF [10] by removing the downsampling layers, resulting in the LIIF-4x variant with a stride of We evaluated LIIF-4x on face (Table Figure.3) and natural images (Table2, Figure.4) using full objectives. Notably, the narrow bottleneck provided consistent performance across different ratios. Tablel shows minimal performance drop for super-resolving images by factors of 2 to 8 for LIIF-64x and other methods, compared to LIIF-4x. This consistent performance FR-SR demonstrates the potential benefits of using a narrow bottleneck encoder.
4.3 Blind Face Image Restoration
Due to the space limitation, we present the results for blind face image restoration, while leaving details of blind natural image restoration in the supplementary material.
Blind image restoration restores high-quality images from complex and unknown degradation. For face images, most recent works [36, 37, 38] focus on two aspects: synthesis of degraded images that mimic the divergence and complex type of degradation via math operators [36, 56), and improve network architectures. We focus the latter aspect.
For blind face image restoration, adopt the degradation operators [38] synthesize degraded/- clean paired data during training. Instead of training the network from scratch, we found finetuning the CosAE network trained for the task of FR-SR (Sec. 4.1.1) yields faster convergence. Unlike training a FR-SR task, we instead fix T = Tmaz. and finetune the model with images of 512 x 512.
Comparison to the State-of-the-Art Methods. We compare the CosAE with the state-of-the-art methods, including DFDNet [34], PULSE [57], GFP-GAN [36], RestoreFormer [37] and Code- Former [38]. Following [36, 38), evaluate CosAE in the CelebA-Test. LFW-Test, WebPhoto- Test, and WIDER Test [38] datasets. We provide quantitative evaluations on both the synthetic dataset - the CelebA-Test dataset in Table. and the real datasets including LFW-Test, WebPhoto- Test, and WIDER- Test Table. 7 in Supp. In addition, we show qualitative results on both synthetic and real-world datasets in Figure. 5.
it outperforms the other approaches on almost all the metrics, except for the FID score. However, it is important to note that the FID score may not provide an accurate assessment of realism when dealing with relatively small test set, especially in the aforementioned real datasets. Qualitatively, compared to
Both quantitative comparisons demonstrate the superior performance of our CosAB:

---

Table x4 super-resolution on ImageNet-val CosAE achieves competitive realism scores compared with all the diffusion- and GAN-based methods [7, 58, 59, 60, 61, 62] with much larger capacities. 'N/A' indicates
not reported.
method
U-Nei
SR3
LDM-4
emphLDDM-
LDM-4-G
GigGAN
7*50
DDRM
IRSDE
ResShin
CosAE
LPIPS]
NIA
N/A
VA
NIA
NIA
N/A
0.206
0.471
0.304
0.126
0.167
FID-50KL
15.2
2.8
2.4
4.4
2.8
2.1
Params (M)
625
625
169
552
183
359
552
552
137
121
67
Inference (3)
NA
NA
NA
NIA
N/A
0.13
1.832
1,184
5.927
0.105
0.036
other approaches, CosAE synthesizes much more realistic facial details, e-g., in the hair and skin regions, as well as more wrinkles, freckles, beards, etc., than to generate a smoother one.
4.4 Fixed-ratio Super-resolution on ImageNet
In order to evaluate approach against conventional methods for learning image representation, we conducted super-resolution experiments on the ImageNet dataset using a 4x upsampling factor. To accomplish this, finetuned the CosAE network trained with the FR-SR natural images with a fixed T = maax i.e., no flexible output resolution being configured during training. This setting was applied to ImageNet images that were resized 256 x 256 pixels. We compared CosAE with U-Net regression [7]. LDM [7] series, GigaGAN [58], other diffusion- models, including IB [59], DDRM [60], IRSDE [61] and ResShift [62]. We use FID score with the reference of 50k validation images in ImageNet. and the LPIPS [48] as the evaluation matrices. In addition, model size well as the inference speed (ours are tested
single V100) are reported. As shown in table 4, CosAE outperforms most models in both efficiency (i.e., number of parameters and runtime) and performance. Qualitative visual comparison with L.M-4 [7] are shown Figure. 14.
5 Conclusion
We introduce CosAE, novel auto-encoder architecture for blind image restoration tasks. By in- corporating learnable harmonic functions in the bottleneck, CosAE effectively captures underly- ing structures and patterns of images with extremely compressive representation, achieving curate reconstruction with high-fidelity details. CosAE outperforms state-of-the-ar methods in flexible-reoolution super-resolution and blind image restoration. As generic denoising Autoen- coder, CosAE offers potential for various computer vision tasks.
References
[1] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th inter- national conference on Machine learning, pages 1096-1103, 2008.
[2] Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 31445786) 2006.
[3] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 1326611312.61 14, 2013.
[4] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/VVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[5] Taesung Park, Jun- Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexei A. Efros, and Richard Zhang. Swapping autoencoder for deep image manipulation. Advances in Neural Information Processing Systems, 2020.
[6] Patrick Esser, Robin Rombach, and Björ Ommer. Taming transformers for high-resolution image synthesis, 2020.
[7] Robin Rombach, Andreas Blattannn, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models, 2021.
10

---

[8] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 136-144, 2017.
[9] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. In Proceedings of the IEEE conference on computer vision and pultern recognition, pages 2472-2481, 2018.
[10] Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. arXiv preprint arXiv:2012.09161, 2020.
[11] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep enn for image denoising. IEEE transactions on image processing, 26(7):3142-1155, 2017.
[12] Stamatios Lefkimmiatis. Non-local color image denoising with convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3587-3596, 2017.
[13] Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pitrre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in deep network with a local denoising criterion. Journal of machine learning research, 1 11(1), 2010.
[14] Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi- supervised learning with ladder networks. Advances in neural information processing systems, 28, 2015.
[15] Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1058-1067, 2017.
[16] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learn- ing, pages 1691-1703. PMLR, 2020.
[17] Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convo- lutional inverse graphics network. Advances in neural information processing systems, 28, 2015.
[18] Tian Ye, Xiaolong Wang, James Davidson, and Abhinav Gupta. Interpretable intuitive physics model. In Proceedings of the European Conference on Computer Vision (ECCV), pages 87- 102, 2018.
[19] Xueting Li, Xiaolong Wang, Ming-Hsuan Yang, Alexei Efros, and Sifei Liu. Scraping textures from natural images for synthesis and editing. ECCV, 2022.
[20] Daniel Glasner, Shai Bagon, and Michal Irani. Super-resolution from single image. In 2009 IEEE 12th international conference on computer vision, pages 149-356. IEEE, 2009.
[21] Yochai Blau, Roey Mechrez, Radu Timofte, Tomer Michaeli, and Lihi Zelnik-Manor, The 2018 pirm challenge on perceptual image super-resolution. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0-0, 2018.
[22] Sung Cheol Park, Min Kyu Park, and Moon Gi Kang. Super-resolution image reconstruction: a technical overview. IEEE signal processing magazine, 20(3):21-36, 2003.
[23] Jianrui Cai, Hui Zeng, Hongwei Yong, Zisheng Cao, and Lei Zhang. Toward real-world single image ruper-resolution: A new benchmark and a new model. In Proceedings of the IEEE International Conference on Computer Vision 2019.
[24] Radu Timofte, Shuhang Gu, Jiqing Wu, Lue Van Gool, Lei Zhang, Ming-Hsuan Yang, Muham- mad Haris, et al. Ntire 2018 challenge on single image super-resolution: Methods and results. In The IEFE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2018
11

---

[25] Priyam Chatterjee and Peyman Milanfar. Is denoising dead? IEEE Transactions on Image Processing, 19(4):895 9115,,991,,,1,, 2009.
[26] Javier Portilla, Vasily Strela, Martin J Wainwright, and Eero Simoncelli. Image denoising using scale mixtures of gaussians in the wavelet domain. IEEE Transactions on Image pro- cessing, 12(111) 2003.
[27] Antoni Buades, Bartomeu Coll, and Jean-Michel Morel. Non-local means denoising. Image Processing On Line, 1:208 200 2011.
(28) Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Transactions on image process- ing. 16(8):2080-2088 2095, 2007.
[29] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Bm3d image denoising with shape-adaptive principal component analysis. In (PARS'00--iiee Processing with Adaptive Sparse Structured Representations, 2009.
[30] Kai Zhang, Wangmeng Zuo, and Lei Zhang. Ffdnet: Toward fast and flexible solution for enn-based image denoising. IEEE Transactions on Image Processing, 27(9):4608-466 2018.
[31] Jingwen Chen, Jiawei Chen, Hongyang Chao, and Ming Yang. Image blind denoising with generative adversarial network based noise modeling. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3155-3164, 2018
[32] Jaewon Lee and Kyong Hwan Jin. Local texture estimator for implicit representation function. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1929-1933, 1938, June 2022.
[33] Jingyu Yang, Sheng Shen, Huanjing Yue, and Kun Li. Implicit transformer network for screen content image continuous super-resolution. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 13304-13315. Curran Associates, Inc., 2021.
[34] Xiaoming Li, Chaofeng Chen, Shangchen Zhou, Xianhui Lin, Wangmeng Zuo, and Lei Zhang. Blind face restoration via deep multi-scale component dictionaries. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX 16, pages 399-415. Springer, 2020.
[35] Shi Guo, Zifei Yan, Kai Zhang, Wangmeng Zuo, and Lei Zhang. Toward convolutional blind denoising of real photographs. In Proceedings of the IEEE/CVF conferencc on computer vision and pattern recognition, pages 1712-1722, 2019.
[36] Xintao Wang, Yu Li, Honglun Zhang, and Ying Shan. Towards real-world blind face restoration with generative facial prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9168 91788,918818111188118888111111 2021.
[37] Zhouxia Wang. Jiawei Zhang, Runjian Chen, Wenping Wang, and Ping Luo. Restoreformer: High-quality blind face restoration from undegraded key-value pairs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17512-17521, 2022.
[38] Guangming Liu, Xin Zhou, Jianmin Pang, Feng Yue, Wenfu Liu, and Junchao Wang. Code- former: A gnn-nested transformer model for binary code similarity detection. Electronics, 12(7):1722, 2023.
[39] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In H. Larochelle, M. Ran- zato, R. Hadsell, M.F. Balcan, and Lin, editors, Advances Neural Information Processing Systems, volume 33, pages 7462-7473. Curran Associates, Inc., 2020.
[40] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
12

---

[41] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich- Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let net- works learn high frequency functions in low dimensional domains. Advances in Neural Infor- mation Processing Systems, 33:7537-7547, 2020.
[42] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning, pages 5301-5310. PMLR, 2019.
[43] Morteza Mardani, Guilin Liu, Aysegul Dundar, Shigiu Liu, Andrew Tao, and Bryan Catanzaro. Neural ffts for universal texture image synthesis. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, vol- ume 33, pages 14081- -4092. Curran Associates, Inc., 2020.
[44] Gaochao Song. Qian Sun, Luo Zhang, Ran Su, Jianfeng Shi, and Ying He. Ope-sr: Orthogonal position encoding for designing a parameter-free upsampling module in arbitrary-scale image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10009-10020 June 2023.
[45] Ronald Newbold Bracewell and Ronald N Bracewell. The Fourier transform and its applica- tions, volume 31999. McGraw-Hill New York, 1986.
[46] Andreas Blattmann, Robin Rombach, Kaan Oktay, and Björn Ommer. Retrieval-augmmened diffusion models, 2022.
[47] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceed- ings the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[48] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features perceptual metric. In CVPR, 2018.
[49] Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1 1125-1134, 2017.
[50] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401 1001-4410, 2019.
[51] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[52] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint 1rXiv:1710.10196, 2017.
2017.
[53] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[54] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database.
[55] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
[56] Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In International Conference on Computer Vision Workshops (ICCVW).
[57] Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. Pulse: Self- supervised photo upsampling via latent space exploration of generative models. Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 2437-2445 2020.
13

---

[58] Minguk Kang, Jun- Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR), 2023.
[59] Guan-Horn Liu, Arash Vahdat, De-An Huang, Evangelos A. Theodorou, Weili Nie, and An- ima Anandkumar. sbb Image-to-image schrödinger bridge, 2023.
[60] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restora- tion models. In Advances in Neural Information Processing Systems, 2022.
[61] Ziwei Luo, Fredrik K. Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas Schön. Image restoration with mean-reverting stochastic differential equations. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference Machine Learning, volume 202 of Pro- ceedings of Machine Learning Research, pages 23045 23066.. PMLR, 23 29 Jul 2023.
[62] Zongsheng Yue, Jianyi Wang, and Chen Change Loy. Resshift: Efficient diffusion model for image super-resolution by residual shifting, 2023.
[63] Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. Musiq: Multi-scale im- age quality transformer. In Proceedings of the IEEE/CVF International Conference on Com- puter Vision, pages 5148-5157, 2021.
[64] Google Inc. Open images dataset. web/ index..ee.. html,tmmm.0000 2016.
[65] Kai Zhang, Yawei Li, Jingyun Liang, Jiezhang Cao, Yulun Zhang, Hao Tang, Radu Timofte, and Lue Van Gool. Practical blind denoising via swin-conv-un and data synthesis. arXiv preprint arXiv:2203. 13278, 2022.
[66] Tsung-Yi Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll'a a r, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR, abs/1 his//405.005 00020, 2014.
14

---

Appendix Table of Contents
1
Introduction
2 Related Work
3
Approach
3.1
Preliminary of Fourier Series
3.2
Encoding Fourier Coefficients
3.3
Constructing Harmonic via HCM
3,4
Learning Frequencies as Network Parameters
3.5
Decoding Harmonic for Reconstruction
3.6
Network Implementation
Experimental Results
4.1
Flexible-resolution Super-resolution (FR-SR)
4.1.1
FR-SR for Face Images.
4.1.2
FR-SR for Natural Images.
4.2
Ablation Studies
4.3
Blind Face Image Restoration
4.4
16 Fixed-ratio Super-resolution on ImageNet
10
5
Conclusion
10
Broader Impact
16
B Discussions
Implementation Details
18
D Additional Experimental Results
19
15

---

Appendix Outlines
The Appendix is structured as follows: We present the broader impact of our work, in Sec. A. We provide insights for the theories presented in the main paper, we discuss more on (i) frequency analyses at Sec. B.I; (ii) if the frequencies need to be regularized during training, at Sec. B.2, and (ii) the limitations, results, in Sec. C and D. Most qualitative results are shown in the last few pages.
Sec. B.3. We present additional implementation details and experimental
A Broader Impact
With the presented applications of face and natural image restoration, the societal impact of the CosAE can be far-reaching. Accurate image restoration is crucial in various domains, including forensic analysis, medical imaging, and digital art preservation. CosAE's ability to restore high- quality images while preserving fine-grained details has the potential to enhance visual content in these fields. By enabling the restoration of images with unknown degradation types, CosAE offers a valuable tool for image analysis and enhancement, leading to improved Docision-making enhanced diagnostics, and enhanced cultural preservation. Furthermore, the generalizability of CosAE can contribute advances in computer vision applications. such as image recognition, object detection, and scene understanding, with broader implications in fields like autonomous vehicles, robotics, and augmented reality.
On the potential downsides, one key concern is the bias in the training data used to develop the model. If the training data is not diverse and representative of different demographics, there is risk of perpetuating biases or inequalities the restored images, particularly in the case of face restoration. This lead unintended consequences, such reinforcing societal stereotypes or discrimination. Therefore, we choose to use the training sets on the face-related tasks, e.g., FFHQ and CelebA, that are inclusive, diverse, and carefully curated to mitigate biases and promote fair- ness in image restoration applications. Additionally, transparency and ethical considerations should be prioritized address concerns related to privacy, consent, and the responsible use of restored images, particularly in sensitive domains such as healthcare or criminal investigations. However, important to note that the primary objective of CosAE is pixel-level restoration rather than recognition or synthesis of faces. This distinction implies that there may be relatively fewer con- cerns associated with the use of CosAE compared other models in the face domain, such as face recognition or synthesis models.
B Discussions
While major designs are presented in the main paper, here, we provide more insights and limitations of CosAE.
B.1 Frequency Analyses
We visualize the learned frequencies u, v E Rex2 c = 256 in Figure. 7. While initialized uni- formly between [0, T/2) (see Sec.3.4), the learned frequencies effectively capture both low- and high-frrequeniiss without significant drift from their initial values. We note that unlike [32], where each image has its individual set of frequencies, which can be compared with its ground truth fre- quency values computed via FFT, in our case, all images will share the same set of frequencies. Along with the design where the frequencies are from the latent space, as such, there is such "ground truth" we can compare with. However, our design closer to the classic discrete Fourier transform, where frequencies are often uniformly sampled and are not input-conditioned.
Initializing with low frequencies. To further quantitatively study the potential effects for fre- quency, we conduct another ablation study: we initialize u, e with low frequencies only, i.e., u, U.&&& E (0, 0.5). denoted as -LF, and train the model with LI loss only to exclude any factors (e.g., the discriminator or the LPIPS loss) that may generate high-frequency details. As demonstrated in Table. 6, without eigh-frequency initialization, as proposed in CosAE, CosAF-LF shows degraded performance on all the matrices.
16

---

Initialized learned freqencies
learned
initialized
Figure 7. Initialized frequencies vs. learned frequencies. While initialized uniformly between T/, (see Sec. 3.4), the learned frequencies effectively capture both low- and high-frequencies without significant drift from their initial values. 3 pair of learned frequencies u, v, marked by green A among the points, are visualized as 2D waves on the right.
Table 5. FR-SR evaluation for uniform vs learnable (u, v). We denote the ablation (suggested by Reviewer Gqnq) for using freezed, uniformly sampled frequencies -uuuforf.orm.
FID-15K]
LPIPS
PSNR (dby"
SSIMI
LR-rrs
64
64
48
64
CosAE-uniin undile-miormmmm (G)
13.09
10.80
10.28
10.49
0.28
0.25
0.23
0.22
22.69
23.51
24.05
24.21
0.63
0.65
0.67
0.68
(G)
8.12
0.17
25.37
0.72
0.76
Does non-learnable frequencies lead to performance degradation? We found that fixing fre- quencies (u, v) to uniform grid caused slight performance drop, reinforcing the need for learnable frequencies for more generalizable model. We retrained the model with fixed uniform sampling, finding that CosAE-unifom underperformed CosAE across metrics, as shown in Table 5. Uniform sampling. while initially dense with T 32, leads to sparse frequency coverage as basis maps or resolution increase. This sampling restricts adaptability and imposes strict grid format, limiting modeling accuracy. Thus, learnable frequencies enhance both performance and design flexibility.
B.2 Frequency Regularization During Training?
The flexible output resolution training strategy, as introduced in Sec. 4.1 training Flexible- resolution Super-resolution (FR-SR), yields a potential issue: When adjusting T (0, Tmaa]. fre- quencies of any u and e that are greater than T/2 will result in aliased waveform maps. Will this issue degrade performance? Or in other words, is non-aliased Cosine basis functions necessary for CosAE? The answer is NO, as we detailed the validation experiments in the following.
Training with Non-aliased Basis Functions. investigate whether maintaining non-aliased ba- sis functions the necessity for CosAF, we design and compare with variant, namely -NoAls. Specifically, we introduce channel-dropout approach to adaptively obtain valid frequencies for dynamically varying output resolution. Given T E (m.x.. Hy E {H] when their min (g,, 0x) Z T/2. Consequently, when reconstructing a smaller- sized output, only subset of the basis functions and bottleneck features are utilized. Intuitively, the design also indicates that a smaller image requires fewer bits in the bottleneck (see Figure. 8).
mask out any basis functions
As detailed in Table. 6, -NoAls even outperforms the reported CosAE, which is trained without re- moving the aliased basis functions, terms of FID scores. the other hand, it still underperforms in terms of LPIPS, PSNR and SSIM, indicating less consistent results compared to the ground-truth images. Thus, we conclude that: (a) training CosAE without maintaining a non-aliased of Cosine basis functions achieves satisfactory results and does not degrade the performance,
reported
17

---

in the main paper; (b) the approach -NoAls is also valid alternation that supports even more com- pressive bottleneck representation when smaller output resolution is desired.
B.3 Limitations
Out-of-distributtin Generalization. In contrast implicit image representation methods for continuous super-resolution [10, 32, 33J, where out-of-distriutiio up-sampling can be achieved by interpolating the coordinate input maps, is not advisable for CosAE to follow a similar ap- proach by directly increasing the value of I during inference. This is because frequencies and that are greater than m which is fixed during training, have not been learned and thus crucial high-frreueenee details may be lost.
Alternatively, CosAE can still generalize super-resolution to higher ratios, e.g., super-resolve face images that are equal to or greater than 32 x 32 to 512 x 512, simply by resizing the input image 512 x 512. However, the face model's generalization capability is limited since the model has not been exposed to high-resolution face patches during training. This is in contrast to the natural image domain model, which is trained on random crops diverse natural images, allowing for broader range of generalization. Thus, we suggest two solutions: First, one can utilize face images with a variety of resolutions and crop patches size 256 256. It is important to include higher resolution patches to capture the desired level of detail in the training process. Secondly. if a specific resolution, such as 512 x 512, is desired, one can further fine-tune the FR-SR face model on the specific resolution (all network architectures and training settings typically converges super fast, e.g., within 100 iterations, during this fine-tuning process. We show the qualitative performance of 512 x 512 face model, which is obtained by fine-tuning the FR-SR face model (the second approach), in Figure. 11.
kept the same). The model
T2XT2
TTX
C2
Figure 8. Training with Non-aliased Basis Functions. Left: when T is small, the corresponding high-frequency coefficients stored in the bottleneck are masked, resulting in a smaller output image; Right: when = max all the bottleneck features are adopted to have a full-sized output image.
Table 6. Additional FR-SR evaluation on the val splits of FFHQ+CelebA faces (15k images). We compared CosAE with (a) the variant that initialized with only loweureyy of CosAE-LF (Sec. B) and (b) the variant -NoAls, that maintain non-aliased basis function only during training (Sec. B.2).
FID-15kj
LPIPSL
(db)"
SSIM1
LR-res res
32
48
64
128
32
48
64
128
32
48
64
128
32
48
64
128
CosAE-LF
N/A
0.38
0.31
23.99
25.
26.99
27.80
0.68
0.76
0.78
COSAE
NAA
0.36
0.32
0.28
0.26
24.34
26.18
27.52
28.51
0.70
0.75
0.79
0.81
CosAE-NoAls (G)
10.78
8.12
7.99
7.53
0.25
0.29
0.18
0.17
23.55
24.96
25.88
26.30
0.66
0.70
0.73
0.74
CosAE
12.81
8.67
8.12
7.86
0.24
0.19
0.17
0.14
23,65
25.37
26.66
27.51
0.67
0.72
0.76
0.78
Common Artifacts.
When restoring extremely low-resofution degraded images, CosAE also presents artifacts. See the tree leaves in the second row of Figure. 10 and the fur textures in the last row Figure. 10. Interestingly, the artifacts are similar to 2D sinusoidal waves which is very different from the conventional CNNVtransformer-ba based models that commonly produce over- smooth or aliased regions, see Figure. 10, the first and the second columns.
C Implementation Details
We further introduce the configuration of the CosAE models for both the face and the natural image domains, with particular emphasis the encoder and decoder given sufficient description is pro- vided for the basis construction module in the main paper. We include key training details to ensure code reproduction. Codes will be released upon legal approval.
18

---

(a) LIIF-4X
(b) LIIF-32X
(c) CosAE
Figure 9. Wide narrow bottlenecks for DIV2K. By comparing upsampling with 8x and 2x, (b) LIIF-32x and (c) CosAE that have narrow bottlenecks presents much consistent super-rcsolution results, compared with that (a) LIIF-4x, with wide bottleneck.
Network Architectures. We follow [6] to design the encoder E. and the decoder D module, where the design logic and the detailed descriptions can be found from its appendix. Based on the stride, for the face model, we configure 7 down-sampling blocks resulting in 64x down-sampling in the bottleneck; Likewise for natural images, we configure 32x down-sampling stride. We configure 2 ResNet blocks at each resolution, and 2 vanilla attention blocks at the last two down-sampling blocks, for both models. We adopt the same decoder for face and natural image models, which retains only 1 upsampling layer, and is configured with 7 ResNet blocks in total.
samplin blocks to accommodate its
Specifically in the decoder, in addition to the aforementioned blocks, we also introduce two attention layers besides the first two ResNet blocks. We restrict the attention layers to nonoverlapping local regions- only within each cosine basis function, shown in the right red block of (H) in Figure. the main paper. This design (i) allows inference of arbitrary resolution input images, e.g., super- resolution of the DIV2K validation set, as shown in Figure. 10 and Figure. 5 in the main paper; and (ii) significantly reduces memory consumption, especially for high-resolution images.
Training Details. For all models, we apply the weight À for the discriminative loss of the patch LGAN as 0.8, while fixing the weight for the regression loss free to 1 (see Sec. 3.6 in the main paper). We adopt the same configuration for the Adam optimizer as introduced in the VQVAE training pipeline in [6, 7]. for all the models. We train all the models up to convergence. To reiterate details described in the main paper: (i) for FR-SR task (Sec.4.1) both face and natural image domains, we randomly sample T < Taaz to allow for flexible resolution output. (ii) we fix the T T max for both image restoration tasks (Sec 4.3) and the fixed-ration super-resolution for ImageNet (Sec 4.4). (iii) Except for face image restoration tasks where models are trained with 512 x 512 input images to align with the previous work, all the other models are trained using 256 x 256 images or patches.
D Additional Experimental Results
D.1 Arbitrary-Resollttion Super-resolution
Inference with Varying T. In the main paper, we evaluate flexible-resolutin super-resolution by re-scaling the input images to the desired output resolution and then performing Autoencoder reconstruction. However, in this section, we present an alternative approach for achieving arbitrary- resolution super-resolution. Similar [10, 32, 33] where the output resolution controllable by configuring the size of the coordinate map, in CosAF, the output resolution can be flexibly changed
19

---

(a) LIIF-4X
(b) LIIF-32X
(c) CosAE
Figure 10. More arbitrary-resole ressuuutio super-resolution on DIV2K [24]. We showcase results (the input images are illustrated the left-most column accordingly) as in the Figure. in the main paper. Zoom in to see more details, especially within close-ups.
20

---

Figure II. Super-resolving 64 x 64 face image by to 8 times, via varying the T. We perform arbitrary- resolution super-resolution in different way, by varying T' E [Tminn Tox that is analogous interpolating the coordinate maps [10, 33). The images in red bounding boxes denote the LR inputs. images with the size of (64, 128, 192, 320, 512) are shown.
by varying T < T Tmax- As aforementioned, image by times, where the model fine-tuned on the face FR-SR model on 512 512 training images from the training splits of FFHQ [50] and CelebA-HQ [51] (which is consistent with training the face FR-SR model), to present higher resolution reconstruction results.
show in Figure. 11 that to super-resolve 64 x 64
More Results for Face Image Super-resolution. We show more qualitative evaluations with com- parisons to the CosAE variants introduced at Sec.4.2, i.e., the nocos, incos and FT, in Figure. 12, the case of up-sampling the 32 x LR input images (Figure. 12 (a)) to 256 x 256. We do not include results of the LIIF-64x and the LIIF-4x models here since they significantly worse, as
21

---

Input
CosAE-nocos
CosAE-imcos
CosAE-FT
CosAE
Ground Truth
CosAE-nooos
CosAE-imcos
CosAE-FT
CosAE
Ground Truth
Figure 12. Qualitative evaluation of Face Image Super-resolution. We compare CosAE with its variants, the nocos, imcos and FT, with super-resolution. While the nocos and imcos likely change the identity and style the original subjects, the FT yields artifacts hair and facial regions, shown in close-ups. Zoom-in to see details.
already demostrated in Figure.3 in the paper. We demonstrate that CosAE significantly outperforms the others. Specifically, both nocos and imcos are prone to completely change the facial style (row and 5), expressions (row 4), identities (row 1,3,4), although their image quality visually present. The FT' presents degraded details on the texture regions, such as hair and facial lines. We show zoomed regions of one example in the last row contrast their differences.
More Results for Natural Image FR-SR. We showcase the 8x super-resolution results the validation split of the DIV2k [24] dataset. Please note that all models are designed to upsample low-resolution images by factor ranging from I to 8. detailed in Sec. 4.1 of the main paper.
In Figure. we demonstrate the importance of narrow bottleneck in the natural image domain. Considering that the base and super-resolution models for natural images possess relatively wider bottleneck compared face models specifically with a down-sampling stride of 32x the consis- tency across varying super-resolution ratios is less conspicuous, especially as evidenced in Table 2 of the main paper. Nonetheless, our observations reveal that noticeable artifacts are more pronounced in the results generated by models with wider bottlenecks, for instance, LIIF-x 4x as displayed in Figure.
22

---

LA
LA
LA
AA
LAU
(a) Noisy
(b) GFPGAN
RestorFormer
(d) CodeFormer
(e) CosAE (ours)
Figure 13. Comparisons with the state-of-thc-art face restoration approaches degraded images. We show results on both the synthetic degraded image dataset and the real-world degraded image datasets. That is, row 1-2 for the WebPhoto- Test dataset (median degradation), row 3-4 for the WIDER-Tea (38) dataset (server degradation). row 5 for the LFW-Test dataset (light degradation). and row for the CelebA-Tes: synthetic dataset. Note that we enable Face Alignment and Background Enhancement for CodeFormer to obtain the best performance, and test the latest version v1.4 for GFPGAN [36].
23

---

Table 7. Quantitative evaluation on real-world face degraded images. Among them, LFW-Test,
WebPhoto-Test and WIDER-TeNT include samples exhibiting a range of degradation levels, from light to severe.
2 Model & Metrics
LFW-Test
WebPhoto-
WIDER-
MUSIQ
FID
MUSIQ
MUSIQ
DFDNet
62.57
67.95
100.68
63.81
57.84
59.34
PULSE
64,86
66.98
86.45
66.57
73.59
65.36
GFPGAN
49.96
68.95
87.35
68.04
40.59
68.26
CodeFormer
52.02
71.43
78.87
70.51
39.06
69.31
CosAE
60.78
71.75
76.16
72.10
44.64
72.62
Table 8. Quantitative evaluation with image restoration on COCO-Test with synthetic corruption. The FID scores computed on the COCO validation split, which contains 5000 images.
Model
FID-SkI
LPIPS.
PSNRT
SSIMT
SCUNet
45.47
0.41
22.42
0.62
CosAE
22.34
0.31
22.85
0.63
9 (a), compared to the other two models with a narrow bottleneck. Similar trends can be observed in Figure. 10. Furthermore, even more substantial discrepancies are apparent when comparing its 8x super-resolution results with those of 2x.
It worth noting that in the re-implemented LIIF [10], interpolation (denoted as local ensemble the original paper) on top the bottleneck. This is because the weighted interpolation doesn't 't ensure convergence when coupled with the encoder and decoder networks. As result, the outputted images exhibit grid-like artifacts (see Figure. and 10 (b)), especially those with large super-resolution ratios.
nearest-neighbor instead of weighted
D.2 Face Image Restoration
We provide more quantitative evaluation on the real-world datasets, including LFW-Test, WebPhoto- Test, and WIDER-Test, in Table. 7. important to acknowledge that the FID score may not provide an accurate assessment of realism when dealing with a relatively small test set, such as in the aforementioned datasets. line with the approach [38], we additionally incorporate MUSIQ [63] into our analysis.
Furthermore, show qualitative results in Figure. 13, compared with the SOTA approaches. For both real-world (row 1-5) and synthetic (row 6) degraded images, CosAE tends to reconstruct more textures in the hair and skin regions. For example, the model tends synthesize more wrin- kles, freckles, beards, etc., on the face, than to generate a smoother one, see Figure. 13, row 3-5. We enable Face Alignment and Background Enhancement for CodeFormer to obtain the best perfor- mance, and test the latest version v.44 for GFPGAN [36].
D.3 4x Ruper-Resolution on ImageNet
We follow the setting in [7] to fine-tune 4 super-resolution model on top of the natural image FR-SR models pretrained on ImageNet. That is, image degradation models are fixed to bicubic interpolation, with down-sampling to the 256 x 256 images from the training split. Figure 14, we conducted comparison with the Latent Diffusion Model (LDM) [7]'. Importantly, benchmarked our method against the qualitative evaluations outlined in [7] and [58]. These were trained on the same dataset, detailed in Table 4 of the main paper. It's worth mentioning that the model referenced Table of the main paper may not align with the released model utilized in Figure 14, which pretrained OpenImages [64].
D.4 Natural Image Restoration
We compare our natural image restoration model with SCUNet [65], model that shares similar configurations to ours. Notably, neither approach utilizes ground-truth noisy-clean pairs while train- ing. Similar to the face restoration model, we fine-tune the FR-SR model from the natural image
Sourced from

---

domain with synthetically degraded images. To do so, we follow the design of image corruption operators SCUNet [65] to produce training pairs online on the ImageNet training split. To eval- uate the approach, we further introduce synthetic dataset produced on the COCO validation split [66], namely COCO-Test, using the same corruption operators training. Adopting the metries as Table. 3 in the main paper (except IDD, which is not applicable for images other than faces), the quantitative evaluation on the synthetic dataset shown Table.
We further showcase the image restoration results on both COCO-Test and a real-world noisy image dataset RNI15 [65], in Figure. 20. On broad level, CosAE and SCUNet [65] exhibit different propensities. CosAE typically emphasizes the preservation of texture and intricate structural details, while SCUNet shows tendency maintain the integrity of larger structures.
25

---

Olaf Thorsen
Olaf
Thorgen
Olaf
ThoTBon
Olaf
Thorsen
of Ahuty
D
Input
LDM
CosAE
Ground Truth
Figure 14. Comparisons with LDM 4x super-resolution ImageNet. In comparison, the CosAE produces more fine-grained details, in terms of both the structure and the texture. Zoom in to details.
26

---

STOP
STOP
STO
(2) Noisy
(b) SCUNct
(c) CosAE (ours)
Figure 15. Qualitative evaluation for image restoration. We compared CosAE with SCUNet [65]. We show denoising results on the synthetic noisy datasel created from the validation split of COCO [66]. in the first rows. We also show the denoising results on real-world noisy images from RNI15 [65], in the last 2 rows. Zoom in to see details.
27

---

Figure 16. x8 Super- Resolution for Natural Images.
28

---

Figure 17, x8 Coppe-Resolution for Natural Images.
29

---

Figure 18, Blind face image restoration, with 512 x 512 resolution.
30

---

Figure 19, Blind face image restoration, with 512 x 512 resolution.
31

---

Figure 20, Blind face image restoration, with 512 x 512 resolution.
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We believe so. We summarize our contributions at the end of the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We explicitly list limitations of our work in Section B.3.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Ys]]
33

---

Justification: It is explained in the approach.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yss]
Justification:
We have included implementation details in both main paper and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
34

---

Answer: [No]
Justification: Although we have answered "No" for now, we intend to release the code and models enable the reproducibility of our main experimental results, pending approval from the legal department. This temporary status reflects our commitment to open access once all necessary permissions secured.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines bnttps://nips.. cc/
guuies//CoeeSumiiee
for more details.
While we encourage the release code and data, we understand that this might not be possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//nips wwtpppccc/caacccccccccccccccecco
cuubicc/guides guwdes/ Codeeubmi sisissss ssi ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: (Yss]
Justification:
It is explained in both the experiment section and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details be provided either with the code, in the appendix, supple- mental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We discussed the details in the paper and it statistically meaningful.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
35

---

The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard emror of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer:
[Yes]
Justification:
is explained in the experimental section and Table 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //tp://in.u.../eeeee
Answer: [Yes]
Justification: Yes, we confirm.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
This is discussed appendix Section A.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
36

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer:
[No]
Justification:
We believe there is no such risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We credit the original owners of all assets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the pack- age should be provided. For popular datasets, paperswi thcode com/ dataasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
37

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification: We applying to release the code and part of the models reproduce the main results in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country which research conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
38


Reviews:
[{"review_id": "EHZErhM8K3", "reviewer": "Reviewer_Gqnq", "summary": "The paper introduces a novel autoencoder that represents an input image using a series of paired Fourier coefficients, representing amplitude and phase. Each pair corresponds to a specific frequency, with all frequencies being learnable parameters of the autoencoder, shared across all images. During decoding, the coefficients are used to construct 2D harmonic functions on a predefined grid. These functions are then input into a decoder network, which outputs the reconstructed image.\nThrough experiments on super-resolution and blind image restoration, the authors demonstrate the effectiveness of the proposed cosine autoencoder in significantly compressing input images into compact representations while preserving both low and high-frequency details", "questions": "Please address weaknesses.", "limitations": "I find that the discussion and limitations in the supplementary material adequately address the major limitations of the proposed work.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* The paper is well-written, with a clear and comprehensive presentation of the background and relevant literature.\n* The derivations of the proposed autoencoder are easy to follow and well-justified, resulting in a simple yet elegant solution.\n* The proposed encoder, grounded in well-established Fourier theory, is likely to have a significant impact on the community, providing a strong foundation for further research.\n* The ability to construct harmonic functions from an image representation allows for visualization of the learned representation, aiding in analysis and interpretation.\n* The authors provide sufficient experiments to demonstrate the effectiveness of their method compared to state-of-the-art approaches.", "weaknesses": "* While most of the work is well-justified and grounded in well-established concepts, this does not extend to the decoding part, specifically the decoding network. The encoding part can be viewed (in a simplified manner) as a non-uniform Fourier transform, so it would be expected that the decoding part would mimic its inverse or at least be more structured than a standard network accepting harmonic images and outputting the recovered image. Although the decoding part is discussed in the paper, I would like to see both the discussion and the ablation study on this point expanded, providing more explanation as to why simple summation does not work in the authors' opinion.\n\n* The authors state that the learned frequencies effectively capture both low and high frequencies without significant deviation from their initial uniform values. While I agree that the learned frequencies do capture both low and high frequencies, Figure 7 suggests they do deviate in practice from their initial values, as there are clear regions of high and low density of learned frequencies. Furthermore, if the frequencies do not deviate significantly from their initial uniform values, is it necessary to learn them? Does fixing them to a uniform grid lead to significantly degraded performance? Finally, does fixing them facilitate the decoding part, leading to a more structured decoding network?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate Reviewer `Gqnq`’s positive assessment of our work regarding the presentation, the technique contribution, and the soundness of our experiments. We address the questions and concerns in the following.\n\n**Q1: More justification for the decoding part.**\n\nThe Fourier inverse transform is explicitly mimic by CosAE through two modules: (i) the HCM module, which composes the learned amplitude, phase, and cosine functions exactly as in Eq. (2); and (ii) the Decoder, which maps the harmonics directly to the RGB feature.  \n\nIf we exactly follow the classical Fourier inverse transform as shown in Eq. (1), we need to perform (i) summation of the harmonics, and (ii) mapping the latent space to the RGB space via a Decoder – this is exactly what CosAE-FT does in the paper, as introduced in line 273-276, and evaluated in Table 1and Figure 12. The results show that our CosAE, which does not sum the harmonics, performs better. We also consistently observed that removing the summation yields much better results since the beginning of the exploration CosAE. \n\nTo explain intuitively, the bottleneck space is a latent space very different from the RGB space. Simple summation in this space does not equate to summation in RGB and can cause high-frequency information loss. Instead, the Decoder aligns the summation operator with a learnable network, resulting in better performance. \n\n\n**Q2: Do the learned frequencies deviate from the initial values?**\n\nBy saying \"the frequencies are not deviating significantly\", we mean the learned $(u,v)$ are still widely distributed across both low and high frequencies. This is advantageous because low frequencies occupy a larger area than other bandwidths in natural images. Previous models, like LTE [32], tend to converge to predominantly low frequencies. In contrast, our method maintains a more balanced frequency distribution, which is what we meant by 'not significantly deviate'. \n\nWe acknowledge that our presentation was not entirely accurate. The learned frequencies differ from the initialized ones, with mid-frequencies being less prominent compared to low and high frequencies. We will analyze this further and revise this part to provide a more accurate depiction. \n\n**Q3: Does fixing them to a uniform grid lead to significantly degraded performance? Should we learn the frequencies?**\n\nWithin the settings of our paper, fixing $(u,v)$ to a uniform grid results in mildly degraded performance. However, learning $(u,v)$ **remains necessary** to achieve a more generalizable network design. We discuss them in the following.\n\nFirst, we retrained the model with $(u,v)$ fixed to a uniform grid and reported the results in Table 1 and Figure 1 of the rebuttal PDF. As shown, CosAE-uniform underperforms CosAE on all the metrics. Figure 1 illustrates that images recovered by CosAE-uniform generally exhibit less high-frequency details on the skin, hair, and teeth regions, compared to those restored by CosAE with learnable $(u,v)$. \n\nTo explain the phnominal, in our paper, CosAE regularizes $(u, v) < T/2$. Since we set T=32, initializing $(u, v)\\in\\[0, 15\\]$ resulted in $16^2=256$ $(u, v)$ pairs, which correspond to 256 basis maps (channels). This uniform sampling works reasonably well in our setting because the frequencies $(u,v)$ are quite dense. \n\nHowever, if one increases the $T$ (e.g., for higher resolution training), or reduces the number of channels (e.g., for better model efficiency), uniform sampling will result in sparsely sampled frequencies. For example, with $T=64$ and the number of basis maps set to $64$, both $u$ and $v$ are sparse, sampled as $\\[0, 4, 8, ..., 32\\]$. Since frequencies in natural images are not uniformly distributed, not allowing them to be adjusted during training prevents effective modeling of these frequencies. Additionally, uniform sampling requires the number of basis maps to always be the square of an integer, which is overly restrictive. Thus, considering the superior performance and the generalization of network design, we suggest making it learnable. We will include the ablation in the revised paper.\n\n**Q4: Does fixing them facilitate the decoding part, leading to a more structured decoding network?**\n\nNo. As explained in Q1, we do not directly sum over the harmonics simply because it yields worse performance. It is irrelevant whether the frequencies are uniform or not.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cbgBk8bzYI", "reviewer": "Reviewer_Yyg3", "summary": "This paper proposed CosAE, a novel autoencoder architecture integrated with the Fourier series for image restoration tasks. Unlike traditional autoencoders that use spatially compressed latent spaces, CosAE encodes images using frequency coefficients, which allows for significant spatial compression while preserving fine details. CosAE outperforms in continuous super-resolution and blind image restoration, with its ability to generalize across various types of image degradations.", "questions": "- In line 232, GAN loss is adopted for LIIF for a fair comparison. Is the LPIPS loss function also adopted, considering that the original LIIF does not included LPIPS loss?\n\n- In lines 238 and 239, doesn't CosAE also require the parameter $T$, which corresponds to the upsampling ratio? What is the fundamental difference between CosAE and other methods for blind super-resolution, regarding the required hyperparameters?\n\n- In line 283, while the authors described that CosAE does not support a wider bottleneck, isn't increasing the number of channels $c$ playing a similar role (That increases the dimension of intermediate features)? \n\n- Does citation [38] refer to the blind face image restoration? This paper is cited several times, but it does not contain any content of face image restoration (referred to in Section 4.3) or dictionary learning (referred to in Line 89).. Please check if this citation is correct.\n\n- In Line 211, is the ratio $r\\in (1, 8]$ an integer or rational number?\n\n.\n\nMinor comments and typos\n- Caption at Figure 5 STOA --> SOTA\n\n.\n\nReference\n[38] Guangming Liu, Xin Zhou, Jianmin Pang, Feng Yue, Wenfu Liu, and Junchao Wang. Code441 former: A gnn-nested transformer model for binary code similarity detection. Electronics, 442 12(7):1722, 2023.", "limitations": "The authors adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Simple idea, but powerful and intuitive framework\n- Paper is overall well-written", "weaknesses": "- No major concern exists, please check Questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's acknowledgment that our approach is simple but insightful. We also thank the reviewer for the valuable feedback. In regards to the questions, please see our responses below:\n\n**Q1: Is the LPIPS loss function also adopted, considering that the original LIIF does not include LPIPS loss?**\n\nYes, we employ the same loss modules for LIIF, including LPIPS and GAN loss, to ensure complete alignment with CosAE.\n\n\n**Q2: Doesn't CosAE also require the parameter, which corresponds to the upsampling ratio? What is the fundamental difference between CosAE and other methods for blind super-resolution, regarding the required hyperparameters?**\n\nThank you for the good question! Most previous methods, such as LIIF, LTE, and ITNSR, require explicitly providing a \"cell\" map to the decoder, which corresponds to the upsampling ratio. In other words, their networks need this ratio as input guidance.\n\nThis doesn't impact super-resolving a low-resolution (LR) image where the ratio, or cell map, can be obtained by dividing the desired output size by the LR image size. However, for random LR images from the internet that may have been zoomed with an unknown ratio, determining the cell map needs to know the actual LR image size, which is difficult. In contrast, CosAE can still handle this by rescaling the image to the desired HR size as the network input. It does not need such upsampling ratio as guidance.\n\nWe will further clarify it in the paper. \n\n**Q3: While CosAE does not support a wider bottleneck, isn't increasing the number of channels c playing a similar role?**\n\nIncreasing the number of channels is not equivalent to increasing the size of the bottleneck. Typically, a larger bottleneck results from an encoder with fewer pooling operations, which better preserves spatial information. While increasing the number of channels in a narrow bottleneck expands the latent space's capacity, it does not compensate for the loss of spatial information due to pooling.  In CosAE, increasing the number of channels means using more cosine basis functions, which is fundamentally different from preserving larger spatial resolutions. \n\nIdeally, we would compare a wider version of CosAE with the narrower one we proposed. However, since CosAE does not support a wider bottleneck design, we instead compare LIIF-4x with LIIF-64x to demonstrate the impact of bottleneck size. The comparison shows that wider bottlenecks maintain more consistent performance across different upsampling ratios. Although our comparison isn't direct, it provides valuable insights: networks with narrower bottlenecks tend to perform more consistently regardless of the input resolution. We will further clarify it in the revised paper. \n\n**Q4: Does citation [38] refer to the blind face image restoration?**\n\nThank you for pointing it out! It is a typo, the correct one to cite is the following. We will fix it in the revised paper. \n\n*Shangchen Zhou, Kelvin C.K. Chan, Chongyi Li, Chen Change Loy. \"Towards robust blind face restoration with codebook lookup transformer.\" NeurIPS 2022.*\n\n**Q5: In Line 211, is the ratio an integer or rational number?**\n\nThe ratio $r$ can be a rational number. For example, when upsampling a $48\\times 48$ input to $256\\times 256$, $r=5.33$ (See Table 1 and 2).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DDD71zZiHJ", "reviewer": "Reviewer_gTQ7", "summary": "The paper introduces the cosine auto encoder method for image restoration. CosAE encodes frequency coefficients  to enable high spatial compression. Experiments on flexible resolution super-solution and blind image restoration demonstrate its effectiveness and generalization.", "questions": "1. Does the method have the potential to construct a very low dimensional latent space to facilitate image generation?", "limitations": "The authors have discuss the limitations carefully.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Nicely presented paper. The paper is well-written. The figures and tables can convey their information clearly.\n2. The idea is interesting. The novel observation is that Fourier space can enable extreme compression ratio. \n3. Rich Experiments across face image restoration and natural image restoration. The proposed method can achieve well performance across multiple image restoration benchmarks.", "weaknesses": "1. The paper lacks theoretical interpretations about the superiority of Fourier space over previous methods involving latent space.\n2. Can the method be applied to other image restoration tasks? such as image deblurring.\n3. The authors should add more visual comparisons with previous methods (not just LIIF).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's recognition of our idea, paper presentation, and solid experiments, as well as their valuable feedback. In regards to the weaknesses and the questions, see our responses below:\n\n**Q1: The paper lacks theoretical interpretations about the superiority of Fourier space over previous methods involving latent space.**\n\nWhile we have very detailed theoretical deductions from Sec. 3.1 to 3.5, as well as B.1 and B.2 in the supplementary. Also, as acknowledged by Reviewer `Gqnq`, these derivations, grounded in well-established Fourier theory, are easy to follow and well-justified. To summarize from the theatrical part of the paper, we have the following advantages: \n\n* **Compact Representation (line 128)**: Unlike most existing architectures that preserve details by maintaining a wider bottleneck or using skip links, our narrow bottleneck representation is highly compact due to the inherently compressive nature of Fourier space, yet it still models both low and high-frequency details.\n\n* **Learnable Fourier Coefficients (Sec. 3.2)**:  CosAE designs amplitude and phase to be learnable, allowing flexible and adaptive encoding of spatial information, e.g., via HCM, compared to fixed transformations in the latent spaces of conventional networks. \n\n* **Consistency and Robustness**: Fourier-based representations are intrinsically less sensitive to variations in image resolution and degradation types. The harmonic functions used in CosAE ensure consistent performance across different image resolutions and degradation scenarios. \n\nWe will further discuss this to strengthen the paper.\n\n**Q2: Can the method be applied to other image restoration tasks? such as image deblurring?**\n\nYes. Our model works favorably on common types of blur images, including Gaussian and Poisson noise, generalized Gaussian blurring, and JPEG artifacts. This is because we explicitly synthesize these degradation operators to generate the training data (line 298). Since these operators mimic the most common degradations caused by camera sensors and the image compression process, our model works well on most real-world blur images, even if the degradation is severe, as shown in Figure 2 (a) in the rebutal PDF.\n\nOn the other hand, we didn't include any motion blur kernel in the data synthesize pipeline, nor pairs of training data. However, we found that CosAE still can generalize well to mild motion blur, as shown in Figure 2 (b). The model performs less effectively on severe motion blur images, as shown in Figure 2 (c). We anticipate that this can be resolved by further incorporating synthetic training images augmented with diverse blur kernels.\n\n**Q3: The authors should add more visual comparisons with previous methods (not just LIIF)**\n\nThanks for the suggestion! We performed visual comparisons with LIIF for the FR-SR task because other approaches, such as LTE and ITNSR, do not perform well when the same combination of objectives (i.e., LPIPS and GAN losses) is added to their original models. For fair comparisons, we used models with only the MSE loss, as shown in the upper part of Table 1. However, the upsampled images predicted by these models lack details with the single MSE loss, making visual validation of detail preservation capability difficult.  \n\nHowever, we note that for other tasks such as blind restoration, we include the latest, and so far the best methods for visual comparison, such as GFPGAN, RestorFormer, CodeFormer, as well as SCUNet. These provide comprehensive qualitative comparisons to highlight the strengths of our method relative to others. \n\n**Q4: Does the method have the potential to construct a very low dimensional latent space to facilitate image generation?**\n\nYes. Although this is beyond the scope of this paper as a plain auto-encoder already performed favorably for blind image restoration, CosAE is suitable for facilitating image generation. A direct way is to equip a KL, or a VQ sampling block in the bottleneck. The advantage is obvious: compact and low-dimensional latent space could potentially benefit high-resolution image generation. It could also benefit the latent diffusion model for high-resolution image generation, or VLM for compressive tokenization for high-resolution images.  \n\nHowever, we note that this exploration may be non-trivial. For instance, it raises questions such as: (a) whether the basis functions need to be conditioned on the sampling block, and (b) the best way to define the dictionaries for Amplitude and Phase across different channels (basis functions), etc. Therefore, we consider this a new topic for future work that is beyond the scope of this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2hXeK8JlS0", "reviewer": "Reviewer_Fx9w", "summary": "This paper formulates the latent space of the autoencoder as a set of Fourier series space, and the encoded images are represented as corresponding amplitude and phase coefficients, which formulates a highly compressed latent space with faithful reconstruction ability. Extensive experiments  on natural images and face images and conducted.", "questions": "Please see the paper weakness.", "limitations": "The authors have adequately addressed the limitations and broader impact of their work.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The reformulated fourier latent space seems work well with reconstruction ability through encoded coefficients.\n2. Experiments on natural images and face images are conducted.", "weaknesses": "1. It is not clear what is the necessity to formulate a highly compressed latent space (or so called information bottleneck) with detailed reconstruction ability for image restoration, as the skip connection will compensate the downsampling lost, which is also the common practices in current restoration network. On the other hand, the compressed latent space may be useful for latent generation, e.g., ldm, however, the corresponding experiments are lacked.\n2. In order to establish a highly compressed latent space for autoencoder, the comparisons with VAE or VQ-VAE is lacked, and it is hard to analysis if the progress is made. \n3. As both experiments are conducted on image restortation, and it is hard to connect the significance of compressed latent space with image restoration tasks, the effectiveness of the proposed method is unable to evaluate with current experimental results.\n4. Is there any ablations that the basis of the latent space are formulated with original fourier basis instead of learnable.\n5. Is the encoder capable of encoding higher resolution images, such as 512, 1024, 2048, etc., will the reconstruction performance be declined, and the compression ratio should be ablated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback. In regards to your questions, see our responses below:\n\n**Q1: It is not clear what is the necessity to formulate a highly compressed latent space (or so called information bottleneck) with detailed reconstruction ability for image restoration, as the skip connection will compensate the downsampling lost.**\n\nThank you for your insightful question. We emphasize exploring information bottleneck is definitely valuable for image restoration, from the following points: \n\n* First, the key to image restoration is learning the intrinsic structure from noisy data, not merely preserving details. Creating an information bottleneck that captures the main structure while removing noise has been studied extensively, such as all the previous work on Denoising Autoencoders (DAE) (see Sec.2). Those methods are less popular for image restoration due to their long-standing limitations in detail preservation with a narrow bottleneck – and we are here to address it. Our CosAE does explore an effective way of making use of the information bottleneck, without loss of details. \n\n* Second, our CosAE architecture has demonstrated STOA performance on numerous image restoration tasks, which strongly supports the value of our approach. This demonstrates that CosAE is valuable not only for research exploration on information bottlenecks, but also as a practical application.\n\n* Additionally, we want to point out that networks with skipped links (e.g., RestoreFormer, LTE, ITNSR, etc.) or wider bottlenecks (LIIF-4x) can also retain noise and degradation signals. For example, our LIIF-4x performs poorly under larger degradations, demonstrating inconsistency and less robustness compared to narrower networks like LIIF-64x and CosAE.  On the other hand, many recent works also explore bottleneck architectures to balance detail preservation and noise reduction, such as CodeFormer. Our work aligns with this trend but introduces a novel Fourier-based approach for a more compact and effective representation. \n\n**Q2: The compressed latent space may be useful for latent generation, e.g., ldm, however, the corresponding experiments are lacked.**\n\nWhile latent diffusion models (LDM) with KL or VQ regularization support generative tasks, the scope of CosAE in this paper focuses on image restoration. These two goals and applications are all very different, making direct comparisons impractical. \n\nIn addition, please refer to the answer to Q4 from Reviewer `gTQ7`, for the discussion of how to further develop CosAE to have image generation capability. Again, we regard it as a different, future direction.\n\n**Q3: The comparisons with VAE or VQ-VAE is lacked.**\n\nFirst, VAEs and VQ-VAEs differ in that they are primarily proposed to enable image generation, by including a sampling module in the bottleneck. In contrast, CosAE does not have it and is not designed for image generation. Also, VAEs and VQ-VAEs do not directly work for blind image restoration or super-resolution tasks. Consequently, a direct comparison is not applicable. \n\nHowever, it is important to note that while VQ-VAE is not directly applicable, CodeFormer, built on top of it, facilitates blind face restoration. We compare CosAE with CodeFormer in Figure 5, 13, Table 3, and 6 across multiple datasets. Since both models utilize similar encoder and decoder architectures from VQ-VAE [6], this allows for an indirect comparison of our method to the VQ-based image restoration approach.\n\n**Q4: It is hard to connect the significance of compressed latent space with image restoration tasks.**\n\nPlease refer to Q1 for why the proposed method is effective. \n\nNote that to validate \"the significance of compressed latent space\", we conducted ablation studies comparing wider and narrower bottlenecks, such as LIIF-4X, LIIF-64x, and CosAE under the same settings (see Figures 3, 4, 9, 10, 12, and Tables 1 and 2). The results consistently show that a narrow bottleneck network performs favorably. \n\n**Q5: Is there any ablations that the basis of the latent space are formulated with original fourier basis instead of learnable?**\n\nYes, we did include that. Since the “original Fourier basis” is ambiguous, we discuss it with the following possibility: \n\n* If \"Original Fourier basis\" means conducting a Fourier transformation on the RGB space, it's important to note that without any network learning capabilities, one can only perform basic Fourier transforms or inverse transforms. This allows for fundamental image processing techniques such as low-pass or high-pass filtering. However, it does not enable advanced tasks like image restoration. \n\n* We have an ablation model named **CosAE-imcos**, introduced in lines 269-272, which encodes the RGB using the original Fourier basis. To facilitate image restoration, we utilize the same auto-encoder, but without the learnable Fourier module in the bottleneck, to process the encoded input signals. Both quantitative and qualitative results are reported in Figure. 3, Table 1, and Figure. 12. \n\n* Additionally, we experimented with a uniform Fourier basis on the latent space. Although it remains in the latent space, it mimics the original Fourier basis instead of being learnable. For quantitative and qualitative results, please refer to Q3 of Reviewer Gqnq, Table 1, and Figure 1.\n\n**Q6: Is the encoder capable of encoding higher resolution images, such as 512, 1024, 2048, etc.?**\n\nYes, all experiments, except for FR-SR on face images and 4x SR on ImageNet, involve restoring images with resolutions of 512x512 or higher. For instance, we perform blind face restoration on 512x512 images and SR on DIV2K with a maximum resolution over 2K (see Figures 4, 9, 10, 16, and 17). We also show SR results for face images at various resolutions, from 64x64 to 512x512, in Figure 11. CosAE can accept any resolution as input, and all these experiments validate its effectiveness on high-resolution images.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for recognizing the technique contribution of our work (Reviewer `Gqnq`), the good-quality presentation (Reviewer `Gqnq`, `Yyg3`, `gTQ7`), simple and intuitive idea (Reviewer `Yyg3`, `gTQ7`), and acknowledging its potential impact (Reviewer `Gqnq`).

While we address the individual questions and concerns in detail below, we have included the following experiments and comparisons in the PDF:

**(A) Additional ablation studies.**

We include the **CosAE-uniform** model with $(u,v)$ uniformly sampled on grids, and un-learned during training. Both qualitative and quantitative results are shown in Figure. 1 and Table. 1. 

Please refer to the answers for Q3 (Reviewer `Gqnq`) and Q5 (Reviewer `Fx9w`), for more discussion.

**(B) Qualitative evaluation on image deblurring.**

In Figure 2, we show how our method performed on blurry images, including (i) real, severely degraded face images from WebPhoto-test [38], (ii) motion blur sample synthesized from CelebA and TextOCR, as publically available from kaggle.

Please refer to the answers for Q2 (Reviewer `gTQ7`) for more discussion.

Meta-review: The paper proposed an autoencoder with Fourier modulation on feature space. Most reviewers appreciated the technical advancements and its contribution leaning toward acceptance although one of them tends to reject it.  The raised concerns are the clarity of writing (principle of algorithm, details of decoder), lack of baselines (i.e. VAE or VQ-VAE) and need for demonstration about scalability. The authors resolved most of raised concerns and got the raised score from one reviewer making consensus to accept in rebuttal phase. One reviewer is still unsatisfied with lack of generative experiments, but this AC believes the requested generative experiments are less relevant with the original content of the paper because the paper is designed for image restoration already mentioned in the title "image restoration". 
With most of consensus toward acceptance, I recommend acceptance of this paper.
Decision: Accept (poster)