Towards Robust Blind Face Restoration with Codebook Lookup Transformer
Authors: Shangchen Zhou
Kelvin C.K. Chan
Chongyi Li
Chen Change Loy
Venue: NeurIPS 2022 Accept (2022)
Keywords: Blind Face Restoration
Image Restoration
Codebook Learning
VQGAN
Discrete Prior

Abstract: Blind face restoration is a highly ill-posed problem that often requires auxiliary guidance to 1) improve the mapping from degraded inputs to desired outputs, or 2) complement high-quality details lost in the inputs. In this paper, we demonstrate that a learned discrete codebook prior in a small proxy space largely reduces the uncertainty and ambiguity of restoration mapping by casting \textit{blind face restoration} as a \textit{code prediction} task, while providing rich visual atoms for generating high-quality faces. Under this paradigm, we propose a Transformer-based prediction network, named \textit{CodeFormer}, to model the global composition and context of the low-quality faces for code prediction, enabling the discovery of natural faces that closely approximate the target faces even when the inputs are severely degraded. To enhance the adaptiveness for different degradation, we also propose a controllable feature transformation module that allows a flexible trade-off between fidelity and quality. Thanks to the expressive codebook prior and global modeling, \textit{CodeFormer} outperforms the state of the arts in both quality and fidelity, showing superior robustness to degradation. Extensive experimental results on synthetic and real-world datasets verify the effectiveness of our method.

Towards Robust Blind Face Restoration with Codebook Lookup Transformer
Shangehen Zhou
Kelvin C.K. Chan Chongyi Li Chen Change Loy
S-Lab, Nanyang Technological University {s200094, chan0899, chongyi 11, ccloy) eew.euuuuuuu..... edu.sg sg
httpe://shangchenzzorr com/pprojets/CooeForre
Abstract
Blind face restoration a highly ill-posed problem that often requires auxiliary guidance 1) improve the mapping from degraded inputs to desired outputs, or 2) complement high-quality details lost in the inputs. In this paper, we demonstrate that learned discrete codebook prior in small proxy space largely reduces the uncertainty and ambiguity restoration mapping by casting blind face restoration code prediction task, while providing rich visual atoms for generating high- quality faces. Under this paradigm, we propose a Transformer-basse network, named CodeFormer, low-quality faces for code prediction, enabling the discovery of natural faces that closely approximate the target faces even when the inputs are severely degraded. To enhance the adaptiveness for different degradation, we also propose a controllable feature transformation module that allows a flexible trade-off between fidelity and quality. Thanks to the expressive codebook prior and global modeling, CodeFormer outperforms the state of the arts both quality and fidelity, showing superior robustness to degradation. Extensive experimental results on synthetic and real- world datasets verify the effectiveness of method.
prediction model the global composition and context of the
1 Introduction
Face images captured the wild often suffer from various degradation, such as compression, blur, and noise. Restoring such images is highly ill-posed as the information loss induced by the degradation leads to infinite plausible high-quality (HQ) outputs given low-quality (LQQ) input. The ill ill-posedness is further elevated in blind restoration, where the specific degradation is unknown. Despite the progress made with the emergence of deep learning, learning LQ-HQ mapping without additional guidance in the huge image space is still intractable, leading to the suboptimal restoration quality of earlier approaches. To improve the output quality, auxiliary information that 1) reduces the uncertainty of LQ-HQ mapping and 2) complements high-quality details is indispensable.
Various priors have been used mitigate the ill ill-posedness of this problem, including geometric priors [5, 6, 30, 44], reference priors [24-26], and generative priors [2, 37, 43]. Although improved textures and details are observed, these approaches often suffer from high sensitivity to degradation or limited prior expressiveness. These priors provide insufficient guidance for face restoration, thus their networks essentially resort to the information of I.Q input images that are usually highly corrupted. As result, the LQ-HQ mapping uncertainty still exists, and the output quality is deteriorated by the degradation of the input images. Most recently, based on generative prior, some methods project the degraded faces into continuous infinite space via iterative latent optimization [27] or direct latent encoding [29]. Despite great realness outputs, it is difficult to find the accurate latent vectors in case of severe degradation, resulting in low-fide cow-fidelity results (Fig. (dd). To enhance the fidelity, skip connections between encoder and decoder are usually required in this kind of methods [37, 43, 2]. as
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

2
y
LQ
Comtnuxes prirr
(infinite space)
(finile space)
HU features
LXp features
(a) Frameworks of comtinuous prior (tap) and our discrete prior (botiom)
(b) Distributions HQ (left) / /LQ (right) features and the codebook items
(c) Input
(d) PULSE [25] (contimoos)
(e GAT-GAN [36]
(1 Neareal Neighbor
(g) CodeFormec (disercte)
(h) Code GT (diserete)
@ Ground Truth:
(contimion))
(discrete)
Figure 1: An illustration of motivation. (a) Restoration frameworks of continuous generative prior (top) and our discrete codebook prior (bottom). (b) t-SNE [35] visualization for HQ/LQ face features and codebook items. (c) LQ input. (d-e) Results of existing methods with continuous prior (PULSE [27] and GFP-GAN [37). (f-g) Results of discrete prior (Nearest Neighbor [11, 34] CodeFormer). (h) Reconstruction results from code sequence ground truth. (i) HQ ground truth. As shown, (d) PULSE without skip connection shows the low fidelity. (e) GFP-GAN with skip connection alleviates identity issues but introduces notable artifacts. (f) Utilizing nearest neighbor matching for code lookup recovers more accurate facial structure compared with (d-e), but some details such as glasses cannot be restored and some artifacts could be introduced. (g) Employing Transformer for code prediction, our CodeFormer generates best results with both high quality and fidelity.
illustrated in Fig. (a) (top), however, such designs meanwhile introduce artifacts in the results when inputs are severely degraded, as shown in Fig. ((e).
Different from the aforementioned approaches, this work casts blind face restoration as code prediction task in a small finite proxy space the learned discrete codebook prior, which shows superior robustness to degradation reconstruction of HQ faces using a vector-quantized autoencoder, which along with decoder stores the rich HQ details for face restoration. In contrast to continuous generative priors [11, 37, 43], the combinations codebook items form a discrete prior space with only finite cardinality. Through much smaller proxy space (e.g., 1024 codes), the uncertainty of the
well rich expressiveness. The codebook is learned by self-
mapping the LQ images LQ-HQ mapping is significantly attenuated, promoting robustness against the diverse degradation, compared in Figs. (dd-g). Besides, the codebook space possess greater expressiveness, which perceptually approximates the image space, as shown in Fig. (hh). This nature allows the network to reduce the reliance inputs and even be free of skip connections.
Though the discrete representation based on codebook has been deployed for image generation [4, 11, 34], the accurate code composition for image restoration remains non-trivial challenge. The existing works look up codebook via nearest-neighhor (NN) feature matching, which is less feasible for image restoration since the intrinsic textures of LQ inputs are usually corrupted. The information loss and diverse degradation in LQ images inevitably distort the feature distribution, prohibiting after fine-tuning the encoder LQ images, the LQ features cannot cluster well to the exact code but spread into other nearby code clusters, thus the nearest-neighhor matching is unreliable in such cases.
accurate feature matching. As depicted in Fig. 1(b) (right),
Tailored for restoration, we propose a Transformer- based code prediction network, named Code- Former, to exploit global compositions and long-range dependencies of LQ faces for better code prediction. Specifically, taking the LQ features as input, the Transformer module predicts the code to- ken sequence which is treated the discrete representation of the face images in the codebook space, Thanks to the global modeling for remedying the local information loss in LQ images, the proposed CodeFormer shows robustness to heavy degradation and keeps overall coherence. Comparing the results presented in Figs. (f--g), the proposed CodeFormer is able to recover more details than the nearest-neighhor matching, such the glasses, improving both quality and fidelity of restoration.
Moreover, we propose controllable feature transformation module with an adjustable coefficient to control the information flow from the LQ encoder to decoder. Such design allows flexible trade-off

---

between restoration quality and fidelity so that the continuous image transitions between them can be achieved. This module enhances the adaptiveness CodeFormer under different degradations. e.g., in case of heavy degradation, one could manually reduce the information flow of I.Q features carrying degradation to produce high-quality results.
Equipped with the above components, the proposed CodeFormer demonstrates superior performance in existing datasets and also our newly introduced WIDER-Test dataset that consists of 970 severely degraded faces collected from the WIDER-Face dataset [42]. In addition to face restoration, our method also demonstrates its effectiveness on other challenging tasks such as face inpainting, where long-range clues from other regions are required. Systematic studies and experiments are conducted to demonstrate the merits of our method over previous works.
2 Related Work
Blind Face Restoration. Since face is highly structured, geometric priors of faces are exploited for blind face restoration. Some methods introduce facial landmarks [6], face parsing maps [5, 30, 41], facial component heatmaps [44], or 3D shapes [16, 28, 48] in their designs. However, such prior information cannot be accurately acquired from degraded faces. Moreover, geometric priors are unable to provide rich details for high-quality face restoration.
Reference- approaches [9, 24-26] have been proposed to circumvent the above limitations. These methods generally require the references to possess same identity as the input degraded face. For example, Li et al. [26] propose guided face restoration network that consists of a warping subnetwork and a reconstruction subnetwork, and a high-quality guided image of the same identity as input is used for better restoring the facial details. However, such references are not always available. DFDNet [24] pre-constructs dictionaries composed of high-quality facial component features. However, the component-speciff dictionary features are still insufficient for high-quality face restoration, especially for the regions out of the dictionary scope (e.g., skin, hair). To alleviate this issue, recent VQGAN-based methods [39, 46] explores learned HQ dictionary, which contains more generic and rich details face restoration.
Recently, the generative facial priors from pre- pre-trained generators, c.g., StyleGAN2 [21], have been widely explored for blind face restoration. It is utilized via different strategies of iterative latent optimization for effective GAN inversion [12, 27] direct latent encoding of degraded faces [29]. However, preserving high fidelity of the restored faces is challenging when one projects the degraded faces into the continuous infinite latent space. To relieve this issue, GLEAN [2, 3]. GPEN [43], and GFPGAN [37] embed the generative prior into encoder-decoder network structures, with additional structural information from input images as guidance. Despite the improvement of fidelity, these methods highly rely on inputs through the skip connections, which could introduce artifacts to results when inputs are severely corrupted.
Dictionary Learning. Sparse representation with learned dictionaries has demonstrated its su- periority in image restoration tasks, such as super-resolution [13, 32, 33, 40] and denoising [10]. However, these methods usually require an iterative optimization to learn the dictionaries and sparse coding, suffering from high computational cost. Despite the inefficiency, their high-level insight into exploring a HQ dictionary has inspired reference- based restoration networks, e.g., LUT [18] and self- reference [47], as well synthesis methods [4]. Jo and Kim [18] construct look-up table (LUT) by transferring the network output values to LUT, so that only simple value retrieval is needed during inference. However, storing HQ textures in the image domain usually requires a huge LUT, limiting its practicality. VQVAE [34] is first to introduce a highly compressed codebook learned by a vector-quantized autoencoder model. VQGAN [11] further adopts the adversarial loss and perceptual loss to enhance perceptual quality at high compression rate, significantly reducing the codebook size without sacrificing its expressiveness. Unlike the large hand-crafted dictionary [18, 24], the learnable codebook automatically learns optimal elements for HQ image reconstruction, providing superior efficiency and expressiveness as well as circumventing the laborious dictionary design. Inspired by the codebook learning, this paper investigates a discrete proxy space for blind face restoration. Different from recent VQGAN-based approaches [39, 46], we exploit the discrete codebook prior by predicting code sequences via global modeling, and we secure prior effectiveness by fixing the encoder. Such designs allow our method to take full advantage of the codebook so that it does not depend on the feature fusion with LQ cues, significantly enhancing the robustness of face restoration.
3

---

Foud medills
(
Encader
En
Da Desoder
IIQ
Miltiplionaa
Colebok C
2n
la
mani A
lvve
Nearesh New ghtee Meaching
Encoder Fg
DH Desoder
HQ
Cabbbok
Commollbble Foature Trandomation (CPT)
Code Prediction
Figure Framework of CodeFormer. (a) We first learn a discrete codebook and decoder store high-quality visual parts of face images via seff-roconstruction learning. (b) With fixed codebook and decoder, we then introduce a Transformer module for code sequence prediction, modeling the global face composition of low- quality inputs. Besides, controllable feature transformation module is used to control the information flow from LQ encoder decoder. Note that this connection optional, which be disabled avoid adverse effects when inputs are severely degraded, and one can adjust scalar weight w to trade between quality and fidelity.
3 Methodology
The main focus of this work exploit a discrete representation space that reduces the uncertainty of restoration mapping and complements high-quality details for the degraded inputs. Since local textures and details are lost and corrupted in low-quality inputs, we employ a Transformer module to model the global composition of natural faces, which remedies the local information loss, enabling high-quality restoration. The overall framework is illustrated in Fig.
We first incorporate the idea of vector quantization [11, 34] and pre- pre-train& quantized autoencoder through self- reconstrintttii obtain discrete codebook and the corresponding decoder (Sec. 3.1). The prior from the codebook combination and decoder is then used for face restoration. Based this codebook prior, we then employ a Transformer for accurate prediction of code combination from the low-quality inputs (Sec. 3.2). In addition, a controllable feature transformation module is introduced to exploit a flexible trade-off between restoration quality and fidelity (Sec. 3.3). The training of our method is divided into three stages accordingly.
3.1 Codebook Learning (Stage I)
To reduce uncertainty of the LQ-HQ mapping and complement high-quality details for restoration, first pre-train the quantized autoencoder to learn a context-rich codebook, which improves network expressiveness well robustness against degradation.
As shown in Fig. 2(a), the HQ face image In e " is first embeded as a compressed feature Zn E x by encoder En. Following VQVAE [34] and VQGAN [11], replace each "pixel" in ZA with the nearest item in the learnable codebook C = {or E 1 N 0k=0 to obtain the quantized
feature Ze E mmxxxd and the corresponding code token sequence s E {0, ... .....NN- 111
((1.)) arg min (1/( Ck lll
((t.)) arg min |Z 0x112-
oREC
The decoder DH then reconstruets the high-quality face image Tree given Ze. The m code token sequence s forms new latent discrete representation that specifies the respective code index of the learned codebook, i.c., (((3)) c when (f-)) k.
Training Objectives. To train the quantized autoencoder with codebook, we adopt three image-level reconstruction losses: L1 loss L1, perceptual loss [19, 45] fper. and adversarial loss [11] Cadv:
C = |14 freeleelll Sper (1,) (Irec) 11:: Cade [log D(I,) +lg(())((el))ee))))
(2)
where & denotes the feature extractor of VGG19 [31]. Since, image-level losses are underconstrained
when updating the codebook items, also adopt the intermediate code-level loss [11, 34] L f to reduce the distance between codebook C and input feature embeddings ZA
C font
||s[(Zn Zell + B sg( zg(Z))
(3)

---

where sg()) stands for the stop-gr coop-gradeet operator and | = 0.25 is weight trade-off for the update rates of the encoder and codebook. Since the quantization operation in Eq. (1) is non-differeniabbe. we adopt straight-through gradient estimator [11, 1, 34] to copy the gradients from the decoder to the
encoder. The complete objective of codebook prior learning Londebook is:
coodebook = L1 + Per + Lleat Aadv
Ladv:
(4)
where Aadv is set to 0.8 in our experiments.
Codebook Settings. Our encoder Eµ and decoder DH consist of 12 residual blocks and 5 resize layers for downsampling and upsampling. respectively. Hence we obtain large compression ratio of r = H/n W/m 32, which leads to a great robustness against degradation and tractable computational cost for our global modeling in Stage II. Although more codebook items could ease reconstruction, the redundant elements could cause ambiguity in subsequent code predictions. Hence, we set the item number N of codebook to 1024, which is sufficient for accurate face reconstruction. Besides, the code dimension d is set to 256.
3.2 Codebook Lookup Transformer Learning (Stage II)
Due to corruptions of textures in LQ faces, the nearest-neighhout (NN) matching in Eq. (1) usually fails in finding accurate codes for face restoration. As depicted in Fig. 1(b), LQ features with diverse degradation could deviate from the correct code and be grouped into nearby clusters, resulting in undesirable restoration results, as shown in Fig. 1(f). To mitigate the problem, we employ Transformer to model global interrelations for better code prediction.
Built upon the learned autoencoder presented in Sec. 3.1, as shown in Fig. 2(b). we insert a Trans- former [36] module containing nine self-attention blocks following the encoder. We fix the codebook C. and decoder DH and finetune the encoder Ey for restoration. The finetuned encoder is denoted as EL. To obtain the LQ features Zi E pmxnxd through EL, we first unfold the features into m n vectors Z R (m-m)) xd and then feed them to the Transformer module. The s-th self-attention block of Transformer computes as the following:
Xs+1 Softmax (Q.K)), + X.,
where X1, = ZJ. The query Q, key K. and value V are obtained from X, through linear layers. We add a sinusoidal positional embedding P (m--)xxd [1, 7] on the queries Q and the keys K to increase the expressiveness of modeling sequential representation. Following the self-attention blocks, Linear layer is adopted to project features to the dimension of (m n) N. Overall, taking the encoding feature Z as an input, the Transformer predicts the
code sequence E (O...., , |N| - 1 in form of the probability of the N code items. The predicted code sequence then retrieves the m respective code items from the learned codebook, forming the quantized feature 2. E mxxx that produces high-quality face image through the fixed decoder Dy- Thanks to our large compression ratio (i.e., 32), our Transformer is effective and efficient in modeling global correlations of LQ face images.
Training Objectives. We train Transformer module T as well as finetune the encoder E,, for restoration, while the codebook C and decoder DHr are kept fixed. Instead of employing reconstruction loss and adversarial loss in the image-level, only code-level losses are required in this stage: 1) cross- entropy loss Ltoken for code token prediction supervision, and 2) L2 loss L to force the LQ feature Z, to approach the quantized feature Z, from codebook, which eases the difficulty of token prediction learning:
7797 -1
Coode
8y log(
Cfeat' code = - Z3
(6)
t=
where the ground truth of latent code is obtained from the pre-trained autoencoder in Stage I (Sec. 3. thus the quantized feature Z is then retrieved from codebook according to the 8. The final objective of Transformer learning is:
Las Atoken -
where Atoken is set to 0.5 in our method. Note that our network after this stage has already equipped with superior robustness and effectiveness in face restoration, especially for severely degraded faces.
5

---

3.3 Controllable Feature Transformation (Stage III)
Despite our Stage II has obtained a great face restoration model, we also investigate a flexible tradeoff between quality and fidelity of face restoration. Thus, we propose the controllable feature transformation (CFT) module to control information flow from LQ encoder EL to decoder H- Specifically, as shown in Fig. 2, the IQQ features F. are used to slightly tune the decoder features Fa through spatial feature transformation [38] with the affine parameters of and B3. An adjustable coefficient w E (0, 1] is then used to control the relative importance of the inputs:
Pa P Fa + (a Fa + B) x w;; a, -Pa(c(Fa F)))
where Pe denotes a stack of convolutions that predicts a and [3 from the concatenated features of c(F,, Fa). We adopt the CFT modules at multiple scales s E {32, 64, 128, 256) between encoder and decoder. Such design allows our network to remain high fidelity for mild degradation and high quality for heavy degradation. Specifically, one could use small w to reduce the reliance on input LQ images with heavy degradation, thus producing high-quality outputs. The larger w will introduce more information from LQ images to enhance the fidelity case of mild degradation.
Training Objectives. To train the controllable modules and finetune the encoder Fiz, in this stage, we keep the code-level losses of Cef Stage II, and also add image-level losses of Li, Eper. and Lodv. which are the same as that in Stage I except that Lee is replaced by restoration output Lyes- The complete loss for this stage is the sum of above losses weighted with their original weight factors. We set the w to I during training of this stage, which then allows network to achieve continuous transitions of results by adjusting w in [0, 1] during inference. For inference, unless otherwise stated, we set the w = 0.5 by default to make a good balance between the quality and fidelity of outputs.
4 Experiments
4.1 Datasets
Training Dataset. We train our models on the FFHQ dataset [21], which contains 70,000 high-quality (HQ) images, and all images are resized to 512x for training. To form training pairs, we synthesize LQ images I from the HQ counterparts Ia with the following degradation model [24, 37, 43]:
4 = (((A @ ko) + n PoJ
where the HQ image J, is first convolved with Gaussian kernel k., followed by a downsampling of scale r. After that, additive Gaussian noise ny is added to the images, and then JPEG compression with quality factor q is applied. Finally, the LQ image is resized back 512x512. We randomly sample o, r, 8, and from [1, 15), [1, 30). [0, 20). and [30, 90), respectively.
Testing Datasets. We evaluate our method on synthetic dataset CelebA-Test and three real-world datasets: LFW-Test, Webshoto-Test, and our proposed WIDER-Tess CelebA- -eet contains 3,000 images selected from the CelebA-HQ dataset [20], where LQ images synthesized under the same degradation range as our training settings. The three real-world datasets respectively contain three different degrees of degradation, i.e., mild (LFW-Test), medium (WebPhoto-Tess) WIDER-TesH LFW-Test consists of the first image of each person in LFW dataset [17], containing 1,711 images. WebPhoto- Test [37] consists of 407 low-quality faces collected from the Internet. Our WIDER-Test comprises 970 severely degraded face images from the WIDER Face dataset [42], providing more challenging dataset evaluate the generalizability and robustness of blind face restoration methods.
and heavy
4.2 Experimental Settings and Metrics
Settings. We represent face image of 512 x 512 3 as a 16 x 16 code sequence. For all stages of training, we use the Adam [23] optimizer with a batch size of 16. We set the learning rate to 8x 10-5 for stages I and II, and adopt a smaller learning rate of 2x10-1 for stage III. The three stages are trained with 1.5M, 200K, and 20K iterations, respectively. Our method is implemented with the PyTorch framework and trained using four NVIDIA Tesla V100 GPUs.
Metries. For the evaluation on CelebA-Tess with ground truth, we adopt PSNR, SSIM, and LPIPS [45] as metrics. We also evaluate the identity preservation using the cosine similarity of features from

---

Symthetic Inpot
DFDNet
PSPRGAN
GLEAN
GER-GAN
GPEN
CodeFormer
Figure 3: Qualitative comparison on CelebA Test. Even though input faces are severely degraded, our CodeFormer produces high-quality faces with faithful details. (Zoom in for details)
Real Input
PSERGAN
GLEAN
GFT-GAN
GPEN
CodeFormeer
Figure 4: Qualitative comparison real-world faces. Our CodeFormer able restore high-quality faces, showing robustness to the heavy degradation. (Zoom in for details)
ArcFace network [8], denoted as IDS. For the evaluation on real-world datasets without ground truth, we employ the widely-used non-reference perceptual metrics: FID [15] and MUSIQ (KonIQ) [22].
4.3 Comparisons with State-of-the-Ar Methods
We compare the proposed CodeFormer with state-of-the-as of-tt--rrt methods, including PULSE [27], DFD- Net [24], PSFRGAN [5], GLEAN [3], GFP-GAN [37], and GPEN [43]. We conduct extensive comparisons on both synthetic and real-world datasets.
Evaluation on Synthetic Dataset. We first show the quantitative comparison on the CelebA- in Table In terms of the image quality metrics LPIPS, FID, and MUSIQ, our CodeFormer achieves the best scores than existing methods. Besides, it also faithfully preserves the identity, reflected by the highest IDS score and PSNR. Additionally, we present the qualitative comparison in Fig. 3. The compared methods fail to produce pleasant restoration results, e.g., DFDNet [24], PSERGAN [5], GFP-GAN [37], and GPEN [43] introduce obvious artifacts and GLEAN [3] produces over-smoothed
Table 1: Quantitative comparison on the CelebA- Test. Red and blue indicate the best and the second best performance, respectively. The result of Code GT is the upper bound of CodeFormer.
Table 2: Quantitative comparison on the real-world LFW-Test. WebPhoto-Tess and WIDER-Test. Red and blue indicate the best and the second best perfor- mance, respectively.
Methods
LPIPSL
RID
MUSIQI
IDS1
PSNR1
SSIM1
Dutaset
LFW-Test mild
WccPhhsto-Tet medium FID.
WIDER Test
Input
0.12
295.73
15.16
0.32
21.53
0.623
Degradation Methods
henvy MUSIQ*
PULSE
72.94
MUSIQ1
MUSI01
DEDNet [24]
0.466
85.15
57.00
0.42
21.24
0.502
Input
137.56
25.05
170.11
19:24
202.06
15.57
20.91
PULSE [27]
64.86
66.98
96.45
66.57
73.59
65.5 36
GLEAN
0.371
59.87
61.59
0.51
21.59
0.508
DFDNet [21]
62.57
67.95
100.68
63.81
57.84
59.34
GFP-GAN [37]
0.391
58.36
67.54
20.37
0.545
PSERGAN
GPEN
0.349
59.70
21.26
0.565
GLEAN (3)
$3.49
66.48
106.63
61.30
47.11
60.68
Codek Fomeerrerrrreo (omrs)
0.299
60.02
75.79
0.60
22.18
0.610
GFP-GAN
49,96
68.95
87.35
68.04
40.59
68.26
54.31
71.94
GPEN
GT
51,40
72.02*
CodeFormer (omes)
52,02
71.43
78,83
70.51
39.06
69.31
7

---

Exp.
Codebook Transformer
Networks
Fix Decoder
NN Code Pred.
Code Lookup
LPIPS, IDS1
Metrics
Code lookup accurag
CodeFormer
Matching
(a)
0.420
0.47
0.20
(b)
0.397
0.51
/
0.15
(e)
0.379
0.52
0.10
(D) (ours, w=l)
/
(2) (ours, w=0)
0.307 0.58
0.05
Table 3: Ablation studies of variant networks and code lookup meth- ods on the CelebA-Test Removing 'Codebook" means the network is a general encoder-decoder structure. "w" is an adjustable coefficient of CFT modules that controls the information flow from encoder.
Degradation levels
Figure 6: Curve comparison on code soquence prediction accuracy.
results that lack facial details. Moreover, all compared methods are unable to preserve the identity. Thanks the expressive codebook prior and global modeling. CodeFormer not only produces high-quality faces but also preserves the identity well, even when inputs are highly degraded.
Evaluation Real-world Datasets. As presented Table our CodeFormer achieves comparable perceptual quality of FID score with the compared methods on the real-world testing datasets with mild and medium degradation, and the best score on the testing dataset with heavy degradation. Although PULSE [27] also obtains good perceptual MUSIQ score, it cannot preserve the identity of input images, as the identity score of IDS and visual results respectively suggested in Table and Fig. 4. From the visual comparisons in Fig. 4, it is observed that our method shows exceptional robustness to the real heavy degradation and produces most visually pleasing results. Notably, CodeFormer successfully preserves the identity and produces natural results with rich details.
4.4 Ablation Studies
Effectiveness of Codebook Space. We first investigate the effectiveness of the codebook space. As shown in Exp. (a) of Table 3, removing the codebook (i.e., directly feeding the encoder features Z₁ to the decoder) results in worse LPIPS and IDS scores. The results suggest that the discrete space of codebook is the key to ensure the robustness and effectiveness of our model.
Superiority of Transformer-based Code Prediction. To verify the superiority of our Transformer- based code prediction for codebook lookup, we compare it with two different solutions, i.e., nearest- neighbour (NN) matching, L..,, Exp. (b), and CNN-based code prediction module, i.e., Exp. (c), that adopts a Linear layer for prediction following encoder E,.. As shown in Table 3, the comparison of Exps. (b) and (c) indicates that adopting code predic- tion for codebook lookup is more effective than NN feature matching. However, the local nature of convo- lution operation of CNNs restricts its modeling capa- bility for long code sequence prediction. In compari- son to the pure CNN-based method, i.e., Exp. (c), our Transformer- based solution produces better- fidelty results in terms LPIPS and IDS scores, well higher accuracy of code prediction under all degrada- tion degrees, as shown Fig. 6. Besides, the supe- riority of CodeFormer is also demonstrated in visual comparisons shown in Fig. and Fig. 9.
Real Input
Nearest Neighhor
CodeFormer
Figure 5: Qualitative comparisons of different
codebook lookup methods.
Importance of Fixed Decoder. Unlike the large dictionary (~3.2G) in DFDNet [24], which aims store massive quantity facial details, we deliberately adopt compact codebook C Nxd with N-1024 and d-256 that only keeps the essential codes for face restoration, which then activate the detailed cues stored in the pre-trained decoder. Thus, the codebook must be used alongside the decoder to fully unleash its potential. To vindicate our design, we conduct two studies: 1) fixing both codebook and decoder, i.e., Exp. (g), and 2) fixing codebook but fine-tuning decoder, i.e., Exp. (e). Table 3 shows fine-tuning decoder deteriorates the performance, validating our statement. This is because fine-tuning the decoder destroys the learned prior that is held by the pre-trained codebook and decoder, resulting in suboptimal performance. Therefore, keep the decoder fixed in our method.
8

---

higher quality
higher Fidelity
Real Input
W 0.2
W 0.4
0.6
W 0.8
W
Figure 7: CFT module is capable to generate continuous transitions between image quality and fidelity.
Flexibility Controllable Feature Transformation Module. Considering the diverse degradation in weal-world LQ face images, we provide controllable feature transformation module (CFT) to allow a flexible trade-off between quality and fidelity. As shown in Fig. 7, a smaller w tends to produce high-qualit, result while larger improves the fidelity. While such flexibility is rarely explored in previous work, here we show that it an appealing strategy to improves the adaptiveness of our method for different scenarios. As shown in Table 3, Exp. (f), i.e., setting the coefficient to 1 increases the reconstruction and identity scores but decreases the visual quality. this work, we trade between the quality and fidelity, and set the coefficient to 0.5 by default.
4.5 Running time
We compare the running time of state-of-the-arr methods [27, 24, 5, 2, 37, 43] and the proposed CodeFormer. All existing methods are evaluated 512² face images using their publicly available code. As shown in Table 5, the proposed CodeFormer has a similar running time as PSFRGAN [5] and GPEN [43] that infer image within 0.1s. Meanwhile, our method achieves the best performance in terms of LPIPS on the Celeb- deet- dataset.
Table 5: Running time of different networks. All methods are evaluated 512 input images using NVIDIA Tesla V100 GPU. indicates the running time is less than 0.1s per test image.
PULSE [27]
DFDNet [24]
PSERGAN
GLEAN
GFP-GAN [37]
GPEN [43]
CodeFormer (Ours)
Time (sec)
48.955
0.179
0.065
0,132 0.371
0.126
0J0S5
0.070
LPIPS)
0.406
0.466
0.395
0.391
4.6 Extensions
Face Color Enhancement. We finetune our model on face color enhancement using the same color augmentations (random color jitter and grayscale conversion) GFP-GAN (v1) [37]. We compare our method with GFP-GAN (v1) [37] on the real-world old photos (from CelebChild-Tes dataset (37]) with color loss. The proposed CodeFormer generates high-quality face images with more natural color and faithful details.
Real Input
GEPGAN (v1)
CodeFormer
Inpot
GFPGAN (v1)
CodeFormer
Figure 8: Visual comparison of face color enhancement the real-world old face photos.
Face Inpainting. The proposed Codeformer can be easily extended face inpainting, and it shows great performance even in large mask ratios. To build training pairs, we use publicly available script [43] randomly draw irregular polyline masks for generating masked faces. We compare our method with two tfate-of-the-ar art face inpainting methods CTSDG [14] and GPEN [43], as well as Nearest-Nei -eighbo matching for codebook lookup. As shown in Fig. CTSDG and GPEN struggle in cases with large masks. Using Nearest-Neighbe matching within our framework roughly reconstructs the face structures, but it also fails in restoring complete visual parts such as the glasses and the eyes. In contrast, our CodeFormer generates high-quality natural faces without strokes and artifacts.

---

Masked Input
CTSDO
GPEN
Nearest Neighbor
Codel'ornes
Figure Visual comparison state-of-the-art face inpainting methods on the challenging cases.
4.7 Limitation
Our method is built on a pre- rre-trained autoencoder with a codebook. Thus, the capability and expres- siveness of the autoencoder could affect the performance of our method. 1) Though the identity inconsistency issue is significantly relieved by the Transformer's global modeling, inconsistency still exists in rare visual parts such accessories, where the current codebook space cannot seamlessly represent the image space. Using multiple scales in the codebook space to explore more fine- tine-grained visual quantization may solution. 2) Although CodeFormer exhibits great robustness in most cases, when it comes to side faces, CodeFormer offers limited superiority to other methods and also cannot produce good results, as failure cases shown in Fig. 10. This expected because there are only few side faces in the FFHQ training dataset, thus, the codebook is unable to learn sufficient codes for this case, leading to less effectiveness in both reconstruction and restoration.
Real Input
GFPGAN
Codefoommer
Real Input
GFPGAN
CodeFormer
Figure 10: Failure cases tend occur on side faces, which could be caused by the limited number side face images in the training dataset of FFHQ.
5
Conclusion
This paper aims to address the fundamental challenges in blind face restoration. With a learned small discrete but expressive codebook space, we turn face restoration to code token prediction, significantly reducing the uncertainty of restoration mapping and easing the learning of restoration network. To remedy the local loss, we explore global composition and dependency from degraded faces via expressive Transformer module for better code prediction. Benefiting from these designs, our method shows great expressiveness and strong robustness against heavy degradation. To enhance also propose controllable feature
the adaptiveness of our method for different degradation, transformation module that allows flexible trade-off between fidelity and quality. Experimental results suggest the superiority and effectiveness of our method.
Acknowledgement
This study is supported under the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, well cash and in-kind contribution from the industry partner(s). is also partially supported by the NTU NAP grant.
10

---

References
[1] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[2] Kelvin CK Chan, Xintao Wang. Xiangyu Xu, Jinwei Gu, and Chen Change Loy. GLEAN: Generative latent bank for large-factor image super-resolution. In CVPR, 2021.
[3] Kelvin CK Chan, Xiangyu Xu, Xintao Wang, Jinwei Gu, and Chen Change Loy. GLEAN: Generative
latent bank for image super-resolution and beyond. IEEE Trunsactions on Puttern Analysis and Machine Intelligence, 2022.
[4] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. MaskGIT: Masked generative image transformer. In CVPR, 2022.
[5] Chaofeng Chen, Xiaoming Li, Lingbo Yang. Xianhui Lin, Lei Zhang, and Kwan-Yee K Wong. Progressive semantic-aware style transformation for blind face restoration. CVPR, 2021.
[6] Yu Chen, Ying Tai, Xiaoming Liu, Chunhua Shen, and Jian Yang. FSRNet: End-to-end learning face super-resolution with facial priors. In CVPR, 2018.
[7] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In CVPR, 2022.
[8] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. AreFace: Additive angular margin loss for deep face recognition. In CVPR, 2019.
[9] Berk Dogan, Shuhang Gu, and Radu Timofte. Exemplar guided face image super-resolutee without facial landmarks. In CVPRW, 2019.
[10] Michael Elad and Michal Aharon. Image denoising via learned dictionaries and sparse representation. In CVPR, 2006.
[11] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high resolutooo image synthesis. In CVPR, 2021.
[12] Jinjin Gu, Yujun Shen, and Bolei Zhou. Image processing using multi-code gan prior. In CVPR, 2020.
[13] Shuhang Gu, Wangmeng Zuo, Qi Xie, Deyu Meng, Xiangchu Feng, and Lei Zhang. Convolutional sparse coding for image super-resolution. In ICCV, 2015.
[14] Xiefan Guo, Hongyu Yang, and Di Huang. Image inpainting via conditional texture and structure dual generation. In ICCV, 2021.
[15] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a time-scale update rule converge to local nash equilibrium. In NeurlPS, 2017.
[16] Xiaobin Hu, Wenqi Ren, John LaMaster, Xiaochun Cao,, Xiaoming Li, Zechao Li, Bjoern Menze, and Wei Liu. Face super-resolutin guided by 3d facial priors. In ECCV, 2020.
(17] Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Leamed-Miller Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In Workshop faces in 'Real- Life Images: detection, alignment, and recognition, 2008.
[18] Younghyun Jo and Seon Joo Kim. Practical single-image super-resolution using look-up table. In CVPR, 2021.
[19] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016.
[20] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In ICLRR 2018.
[21] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
[22] Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. MUSIQ: Multi-scale image quality transformer. In ICCV, 2021.
[23] Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. ICLR, 2014.
[24] Xiaoming Li, Chaofeng Chen, Shangchen Zhou, Xianhui Lin, Wangmeng Zuo, and Lei Zhang, Blind face restoration via deep multi-scale component dictionaries. In ECCV, 2020.
[25] Xiaoming Li, Wenyu Li, Dongwei Ren, Hongzhi Zhang, Meng Wang, and Wangmeng Zuo. Enhanced blind face restoration with multi-exemplar images and adaptive spatial feature fusion. In CVPR, 2020.
[26] Xiaoming Li, Ming Liu. Yuting Ye. Wangmeng Zuo, Liang Lin, and Ruigang Yang. Learning warped guidance for blind face restoration. In ECCV, 2018.
[27] Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. PULSE: Self--uppr photo upsampling via latent space exploration of generative models. In CVPR, 2020.
11

---

[28]
Wengi Ren, Jiaolong Yang. Senyou Deng, David Wipf, Xiaochun Cao, and Xin Tong. Face video deblurring using 3d facial priors. In ICCV, 2019.
[29] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or, Encoding in style: a StyleGAN encoder for image-to-image translation. In CVPR, 2021.
[30] Ziyi Shen, Wei-Sheng Lai, Tingfa Xu, Jan Kautz, and Ming-Hssan Yang. Deep semantic face deblurring.
In CVPR, 2018.
[31] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni-
tion. ICLR, 2015.
[32] Radu Timofte, Vincent De Smet, and Luc Van Gool. Anchored neighborhood regression for fast example- based super-resolution. In ICCV, 2013.
[33] Radu Timofte, Vincent De Smet, and Luc Van Gool. A+: Adjusted anchored neighborhood regression for fast super-resolution. In ACCV, 2014.
[34] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning, NewrIPS, 2017.
[35] Laurens Van Der Maaten. Accelcrating t-sne using tree-based algorithms. The journal of machine learning research, 15(1):3221-3255, 2014.
[36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Maa Polosukhin. Attention is all you need. In NewrIPS, 2017.
[37] Xintao Wang. Yu Li, Honglun Zhang, and Ying Shan. Towards real-world blind face restoration with generative facial prior. In CVPR, 2021.
[38] Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. Recovering realistic texture in image super- resolution by deep spatial feature transform. In CVPR, 2018
[39] Zhouxia Wang, Jiawei Zhang, Runjian Chen, Wenping Wang, and Ping Luo. RestoreFormer: High-quality blind face restoration from undegraded key-value pairs. In CVPR, 2022.
[40] Jianchao Yang. John Wright, Thomas Huang, and Yi Ma. Image super-resolution via sparse representation. IEEE transactions on image processing, 19(1 (9(11):2861228 227-33 2010.
[41] Lingbo Yang. Shanshe Wang. Siwei Ma, Wen Gao, Chang Liu, Pan Wang, and Peiran Ren. HiFaceGAN: Face renovation via collaborative suppression and replenishment. In ACM MM, 2020.
[42] Shuo Yang. Ping Luo, Chen Change Loy, and Xiaoou Tang. WIDER FACE: A face detection benchmark. In CVPR, 2016.
[43] Tao Yang. Peiran Ren, Xuansong Xie, and Lei Zhang. GAN prior embedded network for blind face restoration in the wild. In CVPR, 2021.
[44] Xin Yu, Basura Fernando, Bernard Ghanem, Fatih Porikli, and Richard Hartley. Face super-resolution guided by facial component heatmaps. In ECCV, 2018.
[45] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
[46] Yang Zhao. Yu-Chuan Su, Chun-Te Chu, Yandong Li, Marius Renn, Yukun Zhu, Changyou Chen, and Xuhui Jia. Rethinking deep face restoration. In CVPR, 2022.
[47] Shangchen Zhou, Jiawei Zhang, Wangmeng Zuo, and Chen Change Loy. Cross-scale internal graph neural network for image super-resolution. In NeurlPS, 2020.
[48] Feida Zhu, Junwei Zhu, Wenqing Chu, Xinyi Zhang, Xiaozhong Ji, Chengjie Wang, and Ying Tai. Blind face restoration via integrating face shape and generative priors. In CVPR, 2022.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Sec. 4.7
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] The code is publicly available at: https: //ggtggiiii thub. wat/sschoo/Codeeorroe
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sec. 4.2
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Sec. 4.2
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] It is publicly available.
(c) Did you include any assets either in the supplemental material URL? [Yes]
The code and dataset can be found on our project page.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "WjK18NnVFnW", "reviewer": "Reviewer_pfoY", "summary": "To address the task of blind face restoration, this paper designs a novel network by casting restoration task to the code token prediction task. The authors propose a small and finite codebook space to reduce the uncertainty of image restoration. Upon the learned codebook, they exploit the global interaction by a Transformer, and the controllable modules are employed to balance quality and fidelity. The proposed method is evaluated on both synthetic and real-world datasets and extended to other tasks. The experimental results and ablation studies show that the proposed method is able to restore higher-quality results.", "questions": "*Questions\n1. The training settings of Stage III is unclear, such as, what is $w$ value during training or is it a random value between 0 and 1?\n\n2. The proposed method involves a Transformer to predict the code sequence, and the authors claim that code prediction task by the Transformer could ease the restoration task. But it is unclear if the advantages are introduced by the powerful Transformer. To demonstrate this statement, it would be better to conduct an ablation study that the Transformer predicts the features $\\hat{Z}_c$ directly.\n3. This method is interesting, but I wonder if it has a limitation when applied to generic natural images? From my side, it could be very challenging to learn an essential codebook for a natural image due to diverse scenes and complex textures. What about the authors’ ideas about this? ", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths\n1. The design philosophy of this work is well motivated: building a small discrete codebook to reduce the uncertainty of restoration; treating image restoration as a code prediction task; and adopting global-modeling transformer for better prediction.\n2. The writing of this work is pretty good and it is easy to follow how the highlighted contributions are incorporated into the overall network.\n3. The qualitative results are extensive and promising, especially in the provided supplementary and video demo. The proposed method is also evaluated on other tasks, including face inpainting, color enhancement, and old film enhancement. \n4. I like the controllable design to make a trade-off between quality and fidelity, which is inspiring. A good balance between content preservation and generalization ability is achieved.\n*Weaknesses\nIn general, the paper provides promising performance and sufficient experiments. There is no big flaw. Here, just I just provide some suggestions for improving the quality of the paper.\n\n1. The main power seems to be driven by the learned codebook. In this paper, the authors utilize the codebook with 1024 items. Actually, the performance with the more codes in the codebook is also interesting though the 1024 seems sufficient for face image restoration. Discussion on the number of items could further complete this work.\n\n2. Together with limitation discussion, some failure cases could be provided as well for better understanding.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GaqDNYh_rEb", "reviewer": "Reviewer_ozVV", "summary": "This paper deals with blind face restoration in a learned small codebook space. Using a Transformer to model the global composition of LQ faces, the original restoration task is cast into a code prediction task, which is more robust to degradation. To balance the tradeoff between the quality and fidelity of the results, this work proposes to introduce the controllable modules through the connections between encoder and decoder. The proposed CodeFormer outperforms previous methods on multiple datasets and tasks.", "questions": "1. The proposed network utilizes a learned codebook as a dictionary, which is similar with the design of DFDNet. As far as I know, DFDNet runs very slowly due to the time-consuming feature matching with its multi-scale dictionaries. Additionally, the proposed method also adopts a Transformer module. How about its runtime?\n\n2. I wonder why the authors train the Stage II and III separately? Is it possible to train them jointly to obtain the final model? \n\n3. This paper adopts a large compression rate of 32 when learn the codebook and autoencoder. However the smaller compression rate, such as 16, could have greater reconstruction capability, which may improve the fidelity of outputs. The discussions or study on the choosing of compression rate could be interesting.\n\nMinor typos:\n- In L100, ‘$\\alpha$’ -> ‘$\\theta$’\n- In Eq. (6), should $Z_h$ be $Z_l$?", "limitations": "NA", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper is well organized and nicely written. The presentation of motivation is clear and smooth.\n\n2. Their practice of transforming the image restoration into a code prediction problem is reasonable.\n\n3. The experiments and study are complete and comprehensive. It is also good to provide an interesting video demo. Besides the blind face restoration, the authors also provide an extension of the proposed idea to other tasks such as image inpainting and colorization. The ablation studies are also sufficient to show the efficacy of key components.\n\n4. The visual results are promising, even on heavy degradations.\n\nWeaknesses:\n\nThe learned codebook has been investigated for image generation, as I understand, it is hard to perfectly reconstruct the original HQ faces, which will lead lower fidelity of results when applied to restoration. It would be helpful to provide some concise discussions about the reconstruction capability of codebook and any potential fidelity issues.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7Gb3Z_F59BM", "reviewer": "Reviewer_kois", "summary": "This paper presents a vector quantized codebook based method for blind face restoration. Specifically, the face codebook is first learnt from an encoder-decoder convolutional network. To leverage global context, a transformer-structure neural network is designed to better predict code sequence for highly-corrupted LQ faces. At last, a controllable feature transformation module is trained to adjust the relative importance of the input face (fidelity). The proposed method achieves SoTA performance on synthetic dataset CelebA-Test and comparable performance on real-world datasets LFW-Test, WebPhoto-Test, and WIDER-Test. The ablation study also demonstrates the effectiveness of the proposed modules. ", "questions": "My concerns and questions are:\n\n1. I suggest providing sufficient discussion with some recent literatures, including:\n\n[1] Yuchao Gu et al. VQFR: Blind Face Restoration with Vector-Quantized Dictionary and Parallel Decoder. arXiv preprint arXiv:2205.06803 (2022).\n[2] Yang Zhao et al. Rethinking Deep Face Restoration. In CVPR, 2022.\n\n2. I suggest providing detailed description on the network structure of the encoder and illustrate the design principle.", "limitations": "The limitations of the proposed method have been discussed in the submitted manuscript. I suggest providing quantitative examples for cases where the proposed method fails.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Solid technical design and framework. The proposed CodeFormer framework is properly and elegantly designed, with supportive ablation studies. Remarkably, the proposed method is able to explicitly control and balance fidelity and quality. \n2. Impressive qualitative performances shown in Figure 3, Figure 4, and the supplementary video. State-of-the-art or comparable quantitative performances shown in Table 1 and Table 2. \n3. Clear writing style. Good technical presentation. \n\nConcerns:\n1. Missing related references and disscusion. I suggest to compare and discuss with some recent codebook-based works, e.g., [1] [2]. \n2. The network structure of the encoder in CodeFormer is unclear. The authors should give detailed description and illustrate the design insight.\n3. Writing typos. (1) L144: Unexpected \"?\". (2) L169, Eq (6). Z_h should be Z_l. \n\n[1] Yuchao Gu et al. VQFR: Blind Face Restoration with Vector-Quantized Dictionary and Parallel Decoder. arXiv preprint arXiv:2205.06803 (2022).\n[2] Yang Zhao et al. Rethinking Deep Face Restoration. In CVPR, 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "48-nm_Xzd3z", "reviewer": "Reviewer_BvZ9", "summary": "This paper proposes a transformer-based blind face restoration method. Specifically, it follows VQ-GAN to compress the input features and find the nearest code, and performs autoregression and decoder to generate the synthesized images. To handle corrupted input images, they propose to fix the decoder and introduce a code prediction to ensure output fidelity. Extensive evaluations demonstrate the superior performance of the proposed method over existing approaches.", "questions": "- The proposed architecture follows VQ-GAN, except for the code prediction and controllable feature transformation. I am wondering if the VQ-GAN itself can also achieve similar restoration results.", "limitations": "I do not see any failure cases in the paper. It is essential to show some failure cases for the reader to investigate the failure model to benefit future research.", "rating": 5, "confidence": 1, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The quantitative and visual results are significantly better than the existing methods.\n\nWeaknesses\n- This paper highly follows the architectural design of VQ-GAN. Although with extensive ablation studies, I am wondering if the VQ-GAN can also achieve similar restoration quality.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept