Efficient Face Super-Resolution via Wavelet-based Feature Enhancement Network

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-06 06:33:53

Abstract: Face super-resolution aims to reconstruct a high-resolution face image from a low-resolution face image. Previous methods typically employ an encoder-decoder structure to extract facial structural features, where the direct downsampling inevitably introduces distortions, especially to high-frequency features such as edges. To address this issue, we propose a wavelet-based feature enhancement network, which mitigates feature distortion by losslessly decomposing the input facial feature into high-frequency and low-frequency components using the wavelet transform and processing them separately. To improve the efficiency of facial feature extraction, a full domain Transformer is further proposed to enhance local, regional, and global low-frequency facial features. Such designs allow our method to perform better without stacking many network modules as previous methods did. Extensive experiments demonstrate that our method effectively balances performance, model size, and inference speed. All code and data will be released upon acceptance.

Main paper: 
 introduction: 
 Face super-resolution (FSR), also known as face hallucination, aims to convert a low-resolution (LR) face image into a high-resolution (HR) face image. Different from image super-resolution, FSR focuses on reconstructing essential structural information about the face, including facial contours and the shape of facial components. This paper aims to propose a high-fidelity FSR method while maintaining efficiency in model size and inference speed, as depicted in Fig. 1 .Existing FSR methods [2, 4, 15] typically apply an encoderdecoder structure, which is due to this structure facilitates the model to grasp the overall facial structure during the encoder stage with a larger receptive field and enhances facial details during the decoder stage. Specifically, the encoder initially downsamples the LR input, extracting facial features at various scales. Subsequently, the decoder progressively upsamples the features from the encoder FLOPs [28] [32][4][2][33][23][15]26.9 Ours [28] [32][4] [33] [15] [22] [2] [5] (a) PSNR, FLOPs and Params Tradeoffs. (b) PSNR, FLOPs and Speed Tradeoffs.Figure 1 : Efficiency trade-offs between ours and state-of-theart methods on the CelebA [21] test set. Our method achieves a better balance in terms of performance (PSNR), model size (number of Params and FLOPs), and inference speed. output, refines details, and ultimately generates the HR output. However, previous methods neglect the impact of the chosen downsampling technique on the restoration outcomes. For example, some methods [4, 7] employ bicubic interpolation or strided convolution for downsampling. These operations reduce the number of image pixels, potentially leading to the loss of critical facial details essential for reconstruction. As depicted in Fig. 2 (b ) and (c), bicubic interpolation and strided convolution lead to a significant loss of texture in the overall facial structure, resulting in distortions in the subsequent reconstruction of the face profile. Another example from [12] utilizes downsampling through avgpooling operations. As illustrated in Fig. 2 (d) , this results in the aliasing artifact of high and low-frequency facial features. This phenomenon is particularly evident in the eye features and significantly hampers the accurate representation of facial components.To address the above problem, we propose to utilize discrete wavelet transform to enhance the features. Specifically, following the Nyquist sampling theorem, the standard downsampling process involves a low-pass filter followed by downsampling to prevent frequency domain aliasing. Discrete wavelet transform [24] can simulate standard downsampling by decomposing the input image 𝑰 ∈ R 𝐻 ×𝑊 into four components at different frequencies, which consist of one low-frequency component 𝑰 𝐿𝐿 ∈ R 𝐻 2 × 𝑊 2 and three high-frequency components {𝑰 𝐿𝐻 , 𝑰 𝐻 𝐿 , 𝑰 𝐻𝐻 } ∈ R 𝐻 2 × 𝑊 2 . The low-frequency component 𝑰 𝐿𝐿 can be approximated as the result obtained after low-pass filtering followed by downsampling. Simultaneously, the acquired high-frequency components can still be fused with low-frequency features to enhance information, such as face edges, to alleviate the problem of facial structure loss caused by downsampling. As shown in Fig. 2 (e), employing wavelet feature decomposition and fusion results in significantly clearer facial contours, and there is no occurrence of frequency domain aliasing. This result demonstrates the strategy's effectiveness involving using wavelet transform to decompose features for downsampling and further utilizing the decomposed high-frequency features to enhance the face profile. Inspired by this observation, we introduce wavelet feature downsample (WFD) and wavelet feature upgrade (WFU). WFD aims to minimize distortion of crucial facial structures during downsampling in the encoder phase. WFU aims to enhance facial contour by utilizing extra features obtained through wavelet decomposition in the decoder phase.To better enhance the low-frequency facial information obtained after wavelet transform decomposition, we introduce a full-domain Transformer (FDT). Specifically, as the low-frequency information encapsulates the main features of an image, extracting comprehensive low-frequency information is crucial. Despite the Transformer demonstrating efficacy in handling low-frequency information, the Transformer utilized in existing FSR methods struggles to effectively concentrate on local features (e.g., skin details), regional features (e.g., components like eyes, noses), and global features (e.g., the overall face profile of the face). To address this problem, FDT is proposed to explore diverse receptive fields and uncover deeper correlations within facial features to extract more comprehensive information to enhance facial details.By utilizing WFD and WFU to alleviate facial feature distortion and employing FDT for comprehensive extraction of facial features, our wavelet-based feature enhancement network (WFEN) achieves robust performance without the need for excessive network module stacking like previous methods. Results as Fig. 1 , our WFEN demonstrates outstanding efficiency compared to state-of-the-art methods. In summary, the contributions of this paper are as follows:• We propose WFD and WFU modules utilizing wavelet transform to minimize the distortion of facial features and enhance face contour in the encoder-decoder structure. • We propose an FDT module that extends interactions to the local, regional, and global levels, providing our model with richer facial receptive field information. • We propose a WFEN, which is more efficient than stateof-the-art methods and achieves a better balance in performance, model size, and inference speed. SPARNet [4] No Spatial attention. Restormer [33] No Channel-based self-attention. LAAT [15] Yes Feature fusion from coarse to fine. SFMNet [28] No Utilizing Fourier domain feature.Yes Mitigating feature corruption in downsample. 
related work: 
 Since our method enhances FSR performance through the application of wavelet theory, we present recent advances in FSR and discuss the utilization of wavelet transform in super-resolution. The difference with the main related methods can be seen in Table 1 .With the advancements in deep learning, numerous neural networks [18] for FSR have been proposed to enhance performance. Due to the highly structured nature of the human face, one category of methods aims to leverage facial priors, such as facial landmarks [23] , facial parsing maps [5] , facial attribute [32] , 3D facial shapes [10] , etc., to assist in the restoration process. However, incorporating the face prior estimation module into the network will unavoidably introduce an additional computational burden. Moreover, accurately estimating facial geometric priors becomes highly challenging when dealing with very low-resolution face images. Inaccurate estimation of face priors frequently results in a substantial distortion of the restoration outcomes, which limits the development of prior-based methods in the field of FSR. Consequently, attention-based FSR methods have gained prominence. RAAN [31] utilizes channel attention to extract face shape features, significantly enhancing the model's expressive power. SPARNet [4] introduces spatial attention, allowing it to capture facial structural features efficiently. SISN [22] separately explores facial structural information and facial texture details through the external-internal separation of attention. AD-GNN [2] leverages a series of spatial attention to explore feature relationship modeling and complement facial details. To simulate long-distance modeling, FaceFormer [29] leverages Transformer's capabilities for longdistance modeling to capture global facial information. LAAT [15] further enhances fine-grained regions of features by introducing a self-refinement mechanism into the Transformer. CTCNet [7] and SCTANet [3] integrate spatial attention and self-attention within the model to effectively leverage local and global information. SFM-Net [28] employs frequency domain branching and spatial branching to extract global and local correlations, respectively. However, none of them consider the detrimental effect of downsampling in the encoder-decoder on reconstruction. Moreover, unlike these methods that focus only on local or global facial features, our method simultaneously focuses on local, regional, and global facial features.Recently, wavelet theory has gained prominence in super-resolution. DWSR [9] employs CNN representations on low-resolution wavelet subbands to recover missing details. Wavelet-SRNet [11] reconstructs a face image from a sequence of wavelet coefficients of the HR corresponding to the LR learned by the network. SRCliqueNet [36] explores relationships between wavelet transform subbands to aid the reconstruction process. JWSGN [37] employs wavelet transform to reconstruct the frequency domain details of images. WTRN [20] reconstructs the texture by computing the correlation of the wavelettransformed subbands with the reference image. FOF [16] considers data characteristics in the frequency domain through wavelet transforms, thereby enhancing the efficiency of the network. LAAT [15] employs a wavelet fusion module to combine shallow structures and deep details to recover realistic images in the frequency domain. Unlike these methods, we focus on utilizing wavelet transform to decompose the high and low-frequency components for lossless downsampling, which in turn reduces the feature corruption of downsampling in encoder-decoder structure. 
proposed method 3.1 overview: 
 As shown in Fig. 3 , from a given degraded face image 𝑰 𝐿𝑅 ∈ R 𝐻 ×𝑊 ×3 , we aim to reconstruct a clean face 𝑰 𝑆𝑅 ∈ R 𝐻 ×𝑊 ×3 by employing a wavelet-based encoder-decoder structure integrated with residual block and our full-domain transformer. The wavelet-based encoderdecoder structure encompasses our wavelet feature downsample in the encoder stage and wavelet feature upgrade in the decoder stage for downsampling and upsampling.Specifically, our method initially extracts shallow facial features 𝑭 0 ∈ R 𝐻 ×𝑊 ×𝐶 from 𝑰 𝐿𝑅 , where 𝐻 × 𝑊 denotes the spatial resolution, and 𝐶 denotes the number of channels. Subsequently, 𝑭 0 undergoes hierarchical level-by-level processing through wavelet feature downsample, gradually transforming the 𝑭 0 into a lowresolution latent representation 𝑭 l ∈ R . Then, we incorporate wavelet feature upgrade before each decoding level, which effectively performs cross-scale feature fusion to obtain accurate depth features 𝑭 d ∈ R 𝐻 ×𝑊 ×𝐶 . Finally, the output 𝑭 d from the decoder stage recovers a clean face image 𝑰 𝑆𝑅 after residual concatenation and dimensionality reduction. In the following subsections, we provide a detailed description of the core modules we have constructed.As depicted in Fig. 3 , the central components of the wavelet-based encoder-decoder structure consist of a series of wavelet feature downsamples in the encoder and a series of wavelet feature upgrades in the decoder. They are tasked with progressively downsampling and upsampling, forming the main structure of our network.Wavelet Feature Downsample (WFD). During the encoder process, downsampling is typically employed to decrease the size of the feature map. However, as mentioned above, existing methods overlook the irreversible distortion caused by downsampling, resulting in unclear edges in the FSR results. It occurs because traditional downsampling operations, which decrease resolution by merging neighboring pixels, can result in feature distortion, particularly in regions with significant gradient changes, due to the reduction in sampling points. In this context, as shown in Fig. 3 , we introduce a Anon. WFD using wavelet transform to alleviate this phenomenon. Additional information about wavelet transform can be found in our supplementary materials.𝐻 × 𝑊 × 𝐶 𝐿 × 𝐿 × 𝐶 𝐿 × 𝐿 × 𝐶 𝐿 × 𝐿 × 𝐶 𝐿 × 𝐿 × 𝐶 … K V Q x R R R x 𝐻𝑊 𝑁 2 × 𝑁 2 × 𝐶 𝐶 × 𝐶 𝐻𝑊 𝑁 2 𝐻𝑊 𝑁 2 × 𝑁 2 × 𝐶 𝐻𝑊 𝑁 2 × 𝑁 2 × 𝐶 + 𝐻 × 𝑊 × 𝐶 Regional Self-Attention (RSA) R R x x R Global Self-Attention (GSA) Q K V R ℎ × መ 𝐶 × 𝐻𝑊 ℎ × መ 𝐶 × 𝐻𝑊 ℎ × መ 𝐶 × 𝐻𝑊 መ 𝐶 × መ 𝐶 ℎ + … Shift Windows (SW)For input facial feature 𝑭 1 ∈ R 𝐻 ×𝑊 ×𝐶 , we initially apply a wavelet transform WT , allowing us to decompose 𝑭 1 into four sub-wavelet bands: low-pass feature 𝑨 1 𝐿𝐿 , and high-frequency facial in horizontal, vertical, and diagonal directions 𝑯 1 𝐿𝑅 , 𝑽 1 𝑅𝐿 , andEQUATIONwhere{𝑨 1 𝐿𝑅 , 𝑯 1 𝐿𝑅 , 𝑽 1 𝑅𝐿 , 𝑫 1 𝑅𝑅 } ∈ R 𝐻 2 × 𝐻 2 ×𝐶 .As the low-frequency part predominantly carries essential information in the image, we focus on low-frequency face details on the main path while paying attention to high-frequency face textures on the residual path:EQUATIONwhere 𝑭 low denotes the enhanced low-frequency features, 𝑭 high denotes the enhanced high-frequency features, T denotes our fulldomain Transformer, and R denotes residual block. We opt for different structures to extract high and low-frequency features because prior researches [14, 26] indicate that Transformer is more sensitive to low-frequency features, while CNN is more sensitive to high-frequency features. Next, the full downsampled enhanced feature𝑭 2 ∈ R 𝐻 2 × 𝐻 2 ×2𝐶is obtained by fusing 𝑭 low and 𝑭 high . With this thoughtful design, our model can enhance efficiency in handling both high and low-frequency facial features.Wavelet Feature Upgrade (WFU). To obtain more details, several methods [2, 7, 15] propose using residual concatenation to enable the decoder to leverage information from the encoder. As the resolution of features at different scales differs, upsampling is employed to align them to the same resolution before feature fusion. Nevertheless, direct fusion operation is not optimal as it may introduce some degree of high and low-frequency aliasing. To better fuse features from the encoder, we leverage the wavelet transform for image scale transformations, developing a WFU to effectively utilize features from different scales in the decoder to enhance facial details by fusing high and low-frequency features separately. Specifically, as shown in Fig. 3 , for larger scale feature𝑭 𝑠 ∈ R 𝐻 4 × 𝑊 4 ×4𝐶from the encoder and smaller scale feature𝑭 𝑠+1 ∈ R 𝐻 8 × 𝑊 8 ×4𝐶from the decoder, we initially apply the wavelet transform to the larger scale feature 𝑭 𝑠 , resulting in four wavelet subbands of same scale asEQUATIONwhere 𝑨 𝑠 𝐿𝐿 represents the low-frequency part of 𝑭 𝑠 and 𝑯 𝑠 𝐿𝑅 , 𝑽 𝑠 𝑅𝐿 , 𝑫 𝑠 𝑅𝑅 represent the three high-frequency parts of 𝑭 𝑠 . Considering that small-scale feature 𝑭 𝑠+1 is presumed to contain low-frequency information predominantly, we combine 𝑨 𝑠 𝐿𝐿 with it as the enhanced low-frequency subband. Simultaneously, we employ a residual block to strengthen the high-frequency components of the image, and ultimately, outputEQUATION×4𝐶 can be obtained through the inverse wavelet transform:EQUATIONwhere IWT denotes the inverse wavelet transform, C denotes the concatenation, and R denotes the standard residual block.As analyzed above, the main path of the framework consists mainly of low-frequency information. Therefore, utilizing the Transformer structure, which exhibits greater sensitivity to low-frequency information, is more advantageous for facial feature extraction. To enhance the restoration of facial images, it is crucial to effectively utilize face features at local, regional, and global levels. Specifically, local regions encompass multiple pixels and are most effectively modeled using small 1 × 1 or 3 × 3 kernels, capturing typical features such as local facial details. Regional features encompass dozens of pixel points, such as eyes, nose, and other facial components. Due to their larger spatial extent, they are better modeled using convolution with large kernels [6, 27] or window-based Transformers [19] . Global features involve the structural correlation of the entire face, such as the overall facial contour, and are Table 2 : Quantatitive evaluation for ×8 FSR on CelebA [21] and Helen [13] test sets. The best and second-best results are emphasized in bold and underlined. Our method achieves the best results with the second least computational load and speed.CelebA [21] Helen [ best modeled using the global Transformer. However, many methods [7, 17, 29, 33] only focus on leveraging local and global features or local and regional features. Thus, as shown in Fig. 4 , we propose a full-domain Transformer as our primary module for feature extraction, which consists of two main parts: regional self-attention (RSA) focuses on extracting local and regional facial features, while global self-attention (GSA) is responsible for extracting local and global facial features. Subsequently, we will elaborate on how FDT effectively captures local, regional, and global facial features.Regional Self-Attention (RSA). For an input layer normalized facial feature 𝑿 ∈ R 𝐻 ×𝑊 ×𝐶 , we first extract a set of window features from input 𝑿 :EQUATIONwhere {𝑿 1 , 𝑿 2 , ..., 𝑿 𝑛 } ∈ R 𝐻𝑊 𝑁 2 ×𝑁 ×𝑁 ×𝐶 denotes a set of window feature patches, 𝑁 denotes the size of the window, and 𝑛 = 𝐻 𝑁 = 𝑊 𝑁 . Subsequently, the model initially captures the local facial details to enhance the network's contextual information. Local details are captured by combining a 1 × 1 point-wise convolution and a 3 × 3 depth-wise convolution. Then for each window feature patch 𝑿 𝑖 that enhances the local context, we project it into query𝑸 𝑖 ∈ R 𝐻𝑊 𝑁 2 ×𝑁 2 ×𝐶 , key 𝑲 𝑖 ∈ R 𝐻𝑊 𝑁 2 ×𝑁 2 ×𝐶 , and value 𝑽 𝑖 ∈ R 𝐻𝑊 𝑁 2 ×𝑁 2 ×𝐶 .This process can be described as:EQUATIONwhere RS denotes a reshape operator, D denotes a depth-wise convolution layer, and P denotes a point-wise convolution layer. On this basis, for each window feature patch, regional self-attention can be formulated as:EQUATIONHere, 𝛼 denotes a learnable parameter. To avoid the computational complexity of O (𝐻 2 𝑊 2 ), we choose to implicitly encode global features across channels when computing the feature covariance 𝑨. Specifically, we replace the attention map of size 𝑨 ∈ R 𝐻𝑊 ×𝐻𝑊 in the traditional sense with a regional attention map of size 𝑨 ∈ R 𝐶 ×𝐶 . Furthermore, to address the absence of connectivity among different windows, as shown in Fig. 4 , we introduce a straightforward yet effective information exchange mechanism for RSA to facilitate communication between adjacent windows by shifting windows. Hence, our meticulous design allows RSA to enhance regional and local facial features effectively.Global Self-Attention (GSA). For an input layer normalized facial feature 𝑿 ′ ∈ R 𝐻 ×𝑊 ×𝐶 , similarly, we initially employ 1 × 1 pointwise convolution and 3 × 3 depth-wise convolution to extract local information from 𝑿 ′ , ensuring the accurate recovery of facial details. Subsequently, we adhere to prior methods [33] by subdividing the channel into multi-heads ℎ and concurrently learning distinct self-attention maps. Specifically, we generate query 𝑸 ∈ R ℎ× Ĉ ×𝐻𝑊 , key 𝑲 ∈ R ℎ× Ĉ ×𝐻𝑊 , value 𝑽 ∈ R ℎ× Ĉ ×𝐻𝑊 projections based on the overall face feature after enhanced local detail, where Ĉ is the number of channels in each head. Next, we create a global attention map of size 𝑨 ′ ∈ R Ĉ × Ĉ by computing the dot product of vectors 𝑸 and 𝑲 . This process emphasizes the relationships between channels while implicitly encoding global facial features. In summary, the full process of global self-attention can be formulated as follows:EQUATIONwhere 𝛽 denotes a learnable parameter. To augment information exchange between the multi-heads, as illustrated in Fig. 4 , we achieve this by blending multi-head feature mechanisms. Through meticulous design, our GSA effectively enhances the local and global features of the input face images. 
experiments 4.1 datasets and evaluation metrics: 
 In this paper, We employ the CelebA [21] dataset for training and evaluate the models on the Helen [13] and SCface [8] datasets. Due to variations in the length and width of the original face image, we pre-detect the 68 landmarks of the face using OpenFace [1] . The face images are then cropped based on these landmarks and resized to 128×128 pixels to serve as the ground truth. The ground truth images are further downsampled to 16×16 to generate LR images using bicubic interpolation. Based on this foundation, we utilize We add them to methods SPARNet [4] , LAAT [15] , and SFM-Net [28] and observe PSNR enhancement.18,000 samples from CelebA for training. For testing purposes, we selected 1,000 samples from CelebA and 50 samples from Helen.As for the quality assessment metrics, we used PSNR, SSIM [30] , LPIPS [34] and VIF [25] . Recognizing the significance of identity consistency, we introduced the identity comparison accuracy, denoted as ID. This metric uses SFace [35] as a recognition model, determining whether the restored and original faces belong to the same identity.We implement all experiments using the PyTorch framework with a single NVIDIA GeForce RTX 4090. In the network, we first extend the number of channels 𝐶 to 40. And in R 𝐻 ×𝑊 ×𝐶 stage, the number of full-domain Transformer is set to 2, in R 𝐻 8 × 𝑊 8 ×4𝐶 stage the number is set to 6, and in all the remaining stages the number is set to 1. During the training stage, our model is optimized with an L1 loss with a coefficient of 1, and we use the Adam optimizer with 𝛽 1 =0.9, 𝛽 2 =0.99. We set the initial learning rate to 2 × 𝑒 -4 and the batchsize to 12. In addition, the ID metric's cosine threshold for identity matching is set to 0.5 in the experiment.We benchmark our method against several state-of-the-art FSR methods using a unified dataset. The compared methods include prior-based approaches like FSRNet [5] , FACN [32] , and DIC [23] , attention-based CNN methods such as SPARNet [4] and AD-GNN [2] , and Transformer-based methods like Restormer-M [33] , LAAT [15] , LR FSRNet [5] FACN [32] SPARNet [4] AD-GNN [2] Restormer-M [33] LAAT [15] SFMNet [28] Ours Figure 8 : Qualitative comparison of state-of-the-art methods on the SCface [8] test set. Our method can restore the clearer face components, especially the eye region, which is critical for downstream face recognition tasks.Table 5 : Ablation studies about the efficiency of our fulldomain Transformer (FDT). We use main modules in SPAR-Net [4] , Restormer [33] , and LAAT [15] for feature extraction instead of FDT, respectively. Our FDT can achieve gained performance with reduced computational costs.Helen and SFMNet [28] , where Restormer-M is a generalized image restoration method fine-tuned on face training sets. We present the quantitative results for the CelebA and Helen test datasets in TABLE 2 .The best and the second-best results are emphasized in bold and underlined in this paper. Our method excels in various metrics, including image structure similarity (PSNR and SSIM), visual quality (LPIPS), fidelity (VIF), and face identity consistency (ID), achieving the best performance. Furthermore, we provide quantitative data about models, including the number of model parameters, FLOPs, and inference speed, in TABLE 2 to assess the model's efficiency.Compared with the methods above, our method is less parametric and computationally intensive, and faster in inference, exhibiting excellent efficiency. Next, we visually compare the restoration results of various methods. As shown in Fig. 5 , the high-frequency face profile achieved by our method is significantly sharper and more closely resembles the ground truth, such as key facial components such as the eyes. More qualitative comparisons can also be found in supplementary materials.Additionally, we validate the efficacy of our method in a practical surveillance scenario. For this purpose, we chose HR face images of test subjects from the SCface dataset as the source samples. The corresponding LR face images captured by surveillance cameras are regarded as the target samples. We created five case groups, each consisting of 5 pairs of randomly selected face samples. The evaluation metric is the average similarity between the restored and HR faces. As shown in TABLE 3 , our method consistently reconstructs faces with higher similarity in all cases, which indicates that our method can be better applied to a practical scenario. In addition, visual comparisons on the SCface test set of various methods can be found in Fig. 8 , where prior-based FSR methods exhibit varying degrees of distortion in key facial components. This distortion could be attributed to inaccurate prior estimation, particularly at the current very low resolutions. Attention-based and Transformer-based methods improved the clarity of the restored face to some extent, but the face contours and edges were still not clear. In contrast, our method excels at restoring the contours of the face and facial components with superior clarity, a crucial aspect for downstream tasks like face matching. In summary, the comprehensive results, both quantitative and qualitative, illustrate the efficiency of our model's performance as well as its applicability across various scenarios.This subsection presents an experimental ablation analysis of the causes of our method's effectiveness, including two reasons: the proposed wavelet-based downsample and upgrade modules and the proposed full-domain Transformer.Wavelet Feature Downsample and Upgrade. WFD and WFU are important components in our wavelet-based encoder-decoder structure. To assess the efficacy of our proposed WFD and WFU, we conducted experiments by substituting WFD with stride convolution for downsampling and WFU with interpolation for upsampling. As indicated in TABLE 4 , the computational burden imposed by the WFD module for downsampling is nearly negligible. However, leveraging the WFD for downsampling significantly enhances the model's performance, resulting in a noteworthy PSNR gain of 0.14 dB. Subsequently, we observe that employing WFU to enhance facial details in the decoder stage yields a modest performance improvement compared to the conventional interpolation method of upsampling. This enhancement leads to a PSNR gain of 0.09 dB while maintaining a relatively modest computational load. Meanwhile, corresponding visual comparisons are presented in Fig. 6 . In Fig. 6 (b), without using WFD, the contours around the eyes, mouth, and corners of the face appear somewhat blurred. Similarly, in Fig. 6 (c), without using WFU, the contours around the right eye corner and mouth appear blurred. In contrast, Fig. 6 (d ) with the complete WFD and WFU, reconstructed face component contours are sharper and closer to ground truth. This portion of the experiment strongly demonstrates the effectiveness of our WFD and WFU. Subsequently, to assess the generalization of WFD plus WFU, we incorporate both into several existing methods, replacing their native downsampling and upsampling while preserving their proposed feature extraction modules. These methods include SPAR-Net [4] , LAAT [15] and SFMNet [28] . As depicted in Fig. 7 , all these methods exhibit significant performance enhancements when integrated with WFD plus WFU, with PSNR gains greater than 0.1dB. Notably, our proposed WFD and WFU are remarkably lightweight, imposing an almost negligible additional computational load. In summary of the two-part ablation studies presented above, WFD and WFU are efficient downsampling and upsampling approaches that can be seamlessly integrated into existing methods.Full-domain Transformer. To assess the impact of extracting local, regional, and global facial features on facial reconstruction, we replace the combinations of RSA and GSA in FDT with all-RSA or all-GSA, respectively. This simulates existing FSR methods that exclusively focus on global and local or only regional and local facial features. Meanwhile, it can ensure that the calculated loads of the three ablation methods are comparable for a fair comparison. As depicted in Fig. 9 (a) and (b), our full-domain Transformer exhibits faster training convergence and superior performance on both test sets. Additionally, as illustrated in Fig. 9 (c), our proposed full-domain Transformer exhibits a balanced computational load, including inference speed and training memory. To further showcase the effectiveness of simultaneously capturing local, regional, and global features of a face image, we illustrate the separated case, mixed case, and our full-domain case in Fig. 9 (d) . The separated case refers to situations where only one of the local, regional, or global features of the face image is focused on. Mixed case refers to situations where two of the three facial features-local, regional, and global-are attended to. In contrast to the above two cases, our method achieves a notable performance enhancement by incorporating complementary features across various scales, making it the optimal solution. All experimental results demonstrate that simultaneously focusing on local, regional, and global features of the face image can effectively enhance performance without significantly increasing computational load.Next, TABLE 4 substantiates the significance of information exchange mechanisms, including exchanging facial information across distinct regions via shifting windows in RSA and exchanging different multi-head information via shuffling heads in GSA. As indicated in TABLE 4 , these information exchange mechanisms incur almost no computational cost, yet incorporating both mechanisms separately results in a PSNR gain of 0.05 dB in the model's performance. Moreover, we determine the efficiency of FDT by using the basic feature extraction modules in the three methods, SPARNet [4] , Restormer [33] , and LAAT [15] , instead of FDT. As can be seen from TABLE 5 , FDT achieves the best performance in several metrics with fewer numbers of parameters and FLOPs compared to these modules. This result fully demonstrates that the proposed FDT is a more efficient module to deal with FSR. Therefore, with our proposed FDT as the main feature extraction module, our method has a more powerful feature extraction capability than existing methods. More ablation studies can also be found in supplementary materials. Discussion. Experimental results show that the efficiency of our method is contributed by two parts: wavelet-based coder-decoder structure and full-domain Transformer. Both parts can be integrated into existing encoder-decoder-based methods [4, 15, 28] to further enhance their performance. Therefore, our method is not only efficient in performance but also generalizable. 
conclusion: 
 This paper presents a wavelet-based feature enhancement network for efficient FSR. To address the feature distortion caused by direct downsampling in the encoder-decoder structure, we integrate WFD and WFU into the encoder-decoder structure. Additionally, by further employing our FDT to extract low-frequency facial features comprehensively, our method can achieve a more accurate reconstruction of facial structures. We verify the effectiveness of WFD and WFU in minimizing facial structure distortion during reconstruction and the comprehensive facial feature perception capability provided by FDT. Extensive experiments, including face matching in surveillance scenarios, demonstrate that our method effectively achieves FSR with higher fidelity, achieving an excellent balance between performance, model size, and inference speed.  
 
 Appendix: The clearer input for the first line is to make the feature map easily observable and the contrast pronounced. 


Decision: All 3 reviewers suggested acceptance for this paper. The AC concurs that this is a quality paper with an interesting idea of processing the high-frequency and low-frequency components from wavelet transform and with good performance demonstrated. Congratulations and pls revise the paper per the reviewer's comments.

Reviews: review: paper_summary: paper_summary: This paper describes a wavelet-based feature augmentation network for efficient FSR. To reduce feature distortion due to direct downsampling in the encoder-decoder structure, they incorporate WFD and WFU. Using our FDT to extract low-frequency face information completely leads to a more accurate reconstruction of facial structures. Extensive experiments show that their technique accomplishes FSR with high fidelity while maintaining an outstanding combination of performance, model size, and inference time.
strength_weakness: strength_weakness: By employing the wavelet transform to decompose the input facial features into high-frequency and low-frequency components, the method effectively addresses the common issue of feature distortion introduced by direct downsampling. This approach ensures that high-frequency features, such as edges, are preserved and processed with greater accuracy. Additionally, the integration of a full domain transformer enhances the extraction of local, regional, and global low-frequency facial features, contributing to a more comprehensive and efficient feature representation.
limitations: limitations: The method's performance has been validated through extensive experiments, but its generalizability to a wide range of face super-resolution scenarios, including diverse lighting conditions, facial expressions, and occlusions, remains to be thoroughly tested. 

While quantitative evaluation shows some improvement, this marginal enhancement does not conclusively demonstrate that the results are attributable to the design module rather than the network model itself.
paper_summary: paper_summary: This paper proposes a novel lightweight face super-resolution network. The main contributions include introducing a feature enhancement network based on wavelet transform, which mitigates feature distortion. Additionally, a global transformation method is proposed to enhance local, regional, and global low-frequency facial features, improving the efficiency of facial feature extraction. The proposed method achieves good evaluation metrics on widely used CelebA and Helen datasets.
strength_weakness: strength_weakness: The method proposed in this paper not only improves the effectiveness of face super-resolution but also balances model parameter size and runtime speed, achieving a new lightweight network for face super-resolution.
limitations: limitations: (1) The authors claim that the proposed method in this paper performs well in restoring facial contours and facial components, which are crucial for downstream tasks in facial analysis. However, the paper lacks experiments on downstream tasks. Could downstream tasks such as face recognition and face alignment be added to highlight the effectiveness of the proposed method?
(2)Some of the ablation experiments lack persuasiveness. While the authors presented visualizations of the ablation experiments for WFD and WFU in the paper, visualizations for other ablation experiments are lacking. For example, for the full-domain case proposed in this paper, specific PSNR values are shown in Figure 9(d) of the main text. Could intuitive visualizations be added to supplement the explanation?
paper_summary: paper_summary: To alleviate the image aliasing caused by naive downsampling operations in face super-resolution (FSR) models, this paper presents an efficient FSR method with novel wavelet-based upscaling and downscaling modules for enhanced facial details. Additionally, a domain Transformer is devised to explore and utilize latent relationships among facial features. Extensive experiments demonstrate the effectiveness and efficiency of the proposed method compared with several state-of-the-art methods.
strength_weakness: strength_weakness: 1. Using wavelet transformations to enhance the downsampling and upsampling operations in FSR models seems interesting and effective.

2. This paper is overall well-organized and easy to follow, with clear figures to facilitate comprehension.

3. Both quantitative and qualitative results show the promising FSR performance of the proposed method.
limitations: limitations: 1. Wavelet-based transformation has been widely used in super-resolution. Some classic wavelet-based works should be included for discussion and comparison, such as "Wavelet Domain Generative Adversarial Network for Multi-scale Face Hallucination" (IJCV 2019) and "Wavelet-based dual recursive network for image super-resolution" (TNNLS 2020).

2. Please revise and reorganize some sentences for grammatical correctness and logical smoothness, e.g., Lines 42-46, Lines 114-136 and Lines 144-146.

3. The underline highlighting in Table 2, columns 2-4 is not clear and needs improvement.