Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a vector quantized codebook based method for blind face restoration. Specifically, the face codebook is first learnt from an encoder-decoder convolutional network. To leverage global context, a transformer-structure neural network is designed to better predict code sequence for highly-corrupted LQ faces. At last, a controllable feature transformation module is trained to adjust the relative importance of the input face (fidelity). The proposed method achieves SoTA performance on synthetic dataset CelebA-Test and comparable performance on real-world datasets LFW-Test, WebPhoto-Test, and WIDER-Test. The ablation study also demonstrates the effectiveness of the proposed modules. 
strengths_and_weaknesses: Strengths:
1. Solid technical design and framework. The proposed CodeFormer framework is properly and elegantly designed, with supportive ablation studies. Remarkably, the proposed method is able to explicitly control and balance fidelity and quality. 
2. Impressive qualitative performances shown in Figure 3, Figure 4, and the supplementary video. State-of-the-art or comparable quantitative performances shown in Table 1 and Table 2. 
3. Clear writing style. Good technical presentation. 

Concerns:
1. Missing related references and disscusion. I suggest to compare and discuss with some recent codebook-based works, e.g., [1] [2]. 
2. The network structure of the encoder in CodeFormer is unclear. The authors should give detailed description and illustrate the design insight.
3. Writing typos. (1) L144: Unexpected "?". (2) L169, Eq (6). Z_h should be Z_l. 

[1] Yuchao Gu et al. VQFR: Blind Face Restoration with Vector-Quantized Dictionary and Parallel Decoder. arXiv preprint arXiv:2205.06803 (2022).
[2] Yang Zhao et al. Rethinking Deep Face Restoration. In CVPR, 2022.
questions: My concerns and questions are:

1. I suggest providing sufficient discussion with some recent literatures, including:

[1] Yuchao Gu et al. VQFR: Blind Face Restoration with Vector-Quantized Dictionary and Parallel Decoder. arXiv preprint arXiv:2205.06803 (2022).
[2] Yang Zhao et al. Rethinking Deep Face Restoration. In CVPR, 2022.

2. I suggest providing detailed description on the network structure of the encoder and illustrate the design principle.
limitations: The limitations of the proposed method have been discussed in the submitted manuscript. I suggest providing quantitative examples for cases where the proposed method fails.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a transformer-based blind face restoration method. Specifically, it follows VQ-GAN to compress the input features and find the nearest code, and performs autoregression and decoder to generate the synthesized images. To handle corrupted input images, they propose to fix the decoder and introduce a code prediction to ensure output fidelity. Extensive evaluations demonstrate the superior performance of the proposed method over existing approaches.
strengths_and_weaknesses: Strengths
- The quantitative and visual results are significantly better than the existing methods.

Weaknesses
- This paper highly follows the architectural design of VQ-GAN. Although with extensive ablation studies, I am wondering if the VQ-GAN can also achieve similar restoration quality.
questions: - The proposed architecture follows VQ-GAN, except for the code prediction and controllable feature transformation. I am wondering if the VQ-GAN itself can also achieve similar restoration results.
limitations: I do not see any failure cases in the paper. It is essential to show some failure cases for the reader to investigate the failure model to benefit future research.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: To address the task of blind face restoration, this paper designs a novel network by casting restoration task to the code token prediction task. The authors propose a small and finite codebook space to reduce the uncertainty of image restoration. Upon the learned codebook, they exploit the global interaction by a Transformer, and the controllable modules are employed to balance quality and fidelity. The proposed method is evaluated on both synthetic and real-world datasets and extended to other tasks. The experimental results and ablation studies show that the proposed method is able to restore higher-quality results.
strengths_and_weaknesses: *Strengths
1. The design philosophy of this work is well motivated: building a small discrete codebook to reduce the uncertainty of restoration; treating image restoration as a code prediction task; and adopting global-modeling transformer for better prediction.
2. The writing of this work is pretty good and it is easy to follow how the highlighted contributions are incorporated into the overall network.
3. The qualitative results are extensive and promising, especially in the provided supplementary and video demo. The proposed method is also evaluated on other tasks, including face inpainting, color enhancement, and old film enhancement. 
4. I like the controllable design to make a trade-off between quality and fidelity, which is inspiring. A good balance between content preservation and generalization ability is achieved.
*Weaknesses
In general, the paper provides promising performance and sufficient experiments. There is no big flaw. Here, just I just provide some suggestions for improving the quality of the paper.

1. The main power seems to be driven by the learned codebook. In this paper, the authors utilize the codebook with 1024 items. Actually, the performance with the more codes in the codebook is also interesting though the 1024 seems sufficient for face image restoration. Discussion on the number of items could further complete this work.

2. Together with limitation discussion, some failure cases could be provided as well for better understanding.
questions: *Questions
1. The training settings of Stage III is unclear, such as, what is $w$ value during training or is it a random value between 0 and 1?

2. The proposed method involves a Transformer to predict the code sequence, and the authors claim that code prediction task by the Transformer could ease the restoration task. But it is unclear if the advantages are introduced by the powerful Transformer. To demonstrate this statement, it would be better to conduct an ablation study that the Transformer predicts the features $\hat{Z}_c$ directly.
3. This method is interesting, but I wonder if it has a limitation when applied to generic natural images? From my side, it could be very challenging to learn an essential codebook for a natural image due to diverse scenes and complex textures. What about the authors’ ideas about this? 
limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper deals with blind face restoration in a learned small codebook space. Using a Transformer to model the global composition of LQ faces, the original restoration task is cast into a code prediction task, which is more robust to degradation. To balance the tradeoff between the quality and fidelity of the results, this work proposes to introduce the controllable modules through the connections between encoder and decoder. The proposed CodeFormer outperforms previous methods on multiple datasets and tasks.
strengths_and_weaknesses: Strengths:
1. This paper is well organized and nicely written. The presentation of motivation is clear and smooth.

2. Their practice of transforming the image restoration into a code prediction problem is reasonable.

3. The experiments and study are complete and comprehensive. It is also good to provide an interesting video demo. Besides the blind face restoration, the authors also provide an extension of the proposed idea to other tasks such as image inpainting and colorization. The ablation studies are also sufficient to show the efficacy of key components.

4. The visual results are promising, even on heavy degradations.

Weaknesses:

The learned codebook has been investigated for image generation, as I understand, it is hard to perfectly reconstruct the original HQ faces, which will lead lower fidelity of results when applied to restoration. It would be helpful to provide some concise discussions about the reconstruction capability of codebook and any potential fidelity issues.
questions: 1. The proposed network utilizes a learned codebook as a dictionary, which is similar with the design of DFDNet. As far as I know, DFDNet runs very slowly due to the time-consuming feature matching with its multi-scale dictionaries. Additionally, the proposed method also adopts a Transformer module. How about its runtime?

2. I wonder why the authors train the Stage II and III separately? Is it possible to train them jointly to obtain the final model? 

3. This paper adopts a large compression rate of 32 when learn the codebook and autoencoder. However the smaller compression rate, such as 16, could have greater reconstruction capability, which may improve the fidelity of outputs. The discussions or study on the choosing of compression rate could be interesting.

Minor typos:
- In L100, ‘$\alpha$’ -> ‘$\theta$’
- In Eq. (6), should $Z_h$ be $Z_l$?
limitations: NA
review_title: null

Meta-review: metareview: This work establishes a face restoration algorithm via integrating and optimizing several existing techniques, including VQ-GAN, Codebook prediction and Transformer. The key innovation comes from a Transformer-based prediction network, named CodeFormer, which may somehow exploit the global contexts helpful for codebook lookup. The experiments are reasonably designed, and the results are convincing. All the reviews agree that the paper is well-written and contains solid contributions, thus I would recommend accepting the paper. 
confidence: Certain
Decision: Accept