Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a pipeline to induce geometrical deformations due to expressions on a person’s 3D face. The existing works either uses single image based 3D reconstruction that manifest the expression present in image or learn a neural representation of the deformation (latent code) which can be used to transfer the deformation due to expression from one face to other. This paper uses direct expression parameters and action units instead of a latent code and predict the deformation on 3D face for an expression instead of manifesting it from image (unlike reconstruction methods).
To this end, the authors use an existing robust single image based 3D face deformation method (Chen et al.) to initialise the texture map and geometry as a displacement map of a 3D face (using Basel Face Model) from an image.  Because there is no ground truth of the new expression, they train it using adversarial loss. They produce the deformation using the Action Units(AU) as input and tries to produce those AU from the output displacement map.
For rendering, the method uses the Neural Texture which is trained with the texture map produced by the Chen et al. The rendering has two components. One is the coarse level rendering which utilises the shape and expression parameters of the Basel Face model and the detail rendering consisting of the information of the displacement map due to the AU obtained through this method. Both are combined to get the final appearance. 
They compared the result with DECA (Siggraph 21) which is a FLAME 3D model based method and shows that the identity, shape and expression rendering is better in the proposed method.

main_review: This paper is based on the Basel Face model and much depend upon initial texture and displacement map from Chen et al. The displacement map is trained for the AU but the AUs get changed for normal animation too (like when a person speak in Neutal emotion).Moreover, it also changes for a same person due to the degree of emotion. Because the Network is learning through the adversarial way through the correct generation of AU from the trained displacement map, the network can learn to deform without considering the person specific characteristics and average out the high frequency details. For example, in Fig 4, the last character’s initial face is in happy mood and the deformation on cheek from Chen et al. is quite good in rendering. But that deformation is not so prominent when this method is rendering happy given that initial frame and seems averaged out. But the AUs could be correct. The rendering seems losing the degree of deformation. To show the efficacy of detail displacement map, it would be better to see the result on a face where many wrinkles are present (like an aged person face) like the Fig 1 of Chen et al. Also, if we get a comparison what Chen et al. produce for an expressive face for a video frame (t) and what this method produces given only the AU for that face after initialising with Chen et al. output from an earlier frame (say t-5), then the method’s effectiveness for producing deformation with only AU  can be conceived better. 
Regarding the rendering, the concept of producing overall texture followed by addition of detailed texture is almost similar to DECA but the method is different here which utilises the power of neural texture map. As the displacement map is changing the normal of the face and Basel face has the linear albedo subspace, utilization of these two to render a face with many wrinkles(due to emotion) without using NTM can be shown in a figure to qualitatively see the effectiveness of NTM.
The comparison with DECA shows the clear advantage of this method. It would be nice to comment about the utility of FLAME vs Basel face model. FLAME’s PCA component and BASEL’s PCA components are different and hence the identity preservation of the DECA can get deteriorate. Is this improvement due to the method or the choice of the underlying face model? Similarly the expression subspace for FLAME is also different and the impact of that will be in the DECA’s rendering. So it may be better for a fair comparison of the method to state the equivalence or the impact of choice of the underlying 3D face model.

summary_of_the_review: The pipeline of the paper can work, but the impact of the method and the novelty is not very clear as it depends heavily on Chen et al. and to some extent like DECA’s principle in rendering. As mentioned in the main review, the dependency of the Chen et al. needs to be critically evaluated as an initialization. Also, the choice of underlying 3D face model and impact of it in the outcome is also needed to understand the method’s utility in the comparisons. In the rendering, it seems the high frequency is getting lost. A few more results with more wrinkles in face would be better to judge the effectiveness of the rendering given a good initialisation from Chen et al.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: They propose FaceDet3D to generate facial expressions with details. To do this, they propose a Detail Hallucination Network to generate the target detail map from the source detail map along with Expression Adversarial Loss [1] and Superresolution Losses. Then they propose a Rendering Network. In addition to Photometric Loss and Expression Adversarial Loss [1],  they add Augmented Wrinkle Loss and Detailed Shading Loss. The result is an image with target expression containing details (mainly wrinkles), comparison with DECA (a previous work) shows they improve FID and FaceID by large margin.

[1] Conditional Image Synthesis With Auxiliary Classifier GANs

main_review: Strengths:
* The motivation is very clear.
* The methodology is well written.
* The use of Age feature and FaceID feature is reasonable and novel
* The idea of separating rendering into two branches is interesting.
* AugW uses the augmented image with wrinkles to enforce the rendering of details.

Weaknesses:
* Lack of ablation study: 1) the “weakly supervise” mentioned in 3.2.1 is not mentioned in the ablation study. 2) The impact of Age and FaceID features is not studied in the ablation. 3) Is the mask necessary in Detail Hallucination Network? 4) the quantitative result from model w/o AugW and model w/o DSL (SEPARATELY) is not provided. 

* Based on Figure 5, 6 and 7, I still observe that quite a few details (i.e., wrinkles) that appear in the input are missing in your generated image. It suggests your model still fails to capture the important and significant details when there are substantial wrinkles on the face.

* Performance study should be conditioned on (broken down according to) the key factors, such as age or emotions.  Obviously, the amount of deformation in the face (wrinkles) will increase with age.  Similarly, certain emotions/AUs result in more fine grained deformation than others.  

* Comparison with DECA seems to be not fair, as you mentioned, DECA uses different morphable models, the reason may be that input is different.

* There is only one compared baseline, DECA. More compared methods could be added.

* It would be better to separate the contributions of the Detail Hallucination Network and Rendering Network. (e.g. Detail Hallucination Network+existing Rendering method)

summary_of_the_review: While the authors design two GANs to generate the detail map and final image along with several loss terms, the authors still need to do a thorough and fair ablation study (how important is the “weakly supervise” mentioned in both Section 3.2.1 and Conclusion? How important is AugW and DSL separately?) and compare with fair previous works (How about comparing with models using the same input?).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes to improve the quality of the 3D reconstructed faces by taking into account the facial expression. Using a set of generative losses some of the details are recovered or hallucinated. The visual examples provided confirm the improvements, however better numerical evaluations and comparisons are required.
main_review: Pos:
+ The method produces faces of higher quality when analyses visually (based on the provided examples).
+ While straight-forward, the proposes losses are sound and appear to work well.

Cons:
- The novelty and scope appears limited and incremental. Perhaps the author could better explain&detail this aspect?
- Little to no comparison against current state-of-the-art on 3D face reconstuction.
- The evaluation metrics used are weak in the context of 3D face reconstruction. While the FID score is a good proxy metric for generated images in general it's unclear how representative it is in this scenario. Furthermore the more established point-to-point errors are missing.
- The problem is quite niche, ideally one will want an approach than can in general synthesize high-fidelity models and texture. How can this method be adapted to a more general case? 
- What is the reason for using the HQ 4096x4096 data if the images are downsampled to 256x256? Is there a benefit over using 256px directly?
- What is the impact of each proposes loss?
summary_of_the_review: The novelty and scope appear limited and the evaluation is incomplete. The paper could be improved by comparison against state-of-the-art and the use of more standard-metrics.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: 1. This paper proposes a new method FaceDet3D that can generate geometric facial details from a single image given any desired target expression. The method contains two components: first pass the input image to the Detail Hallucination Network to infer the geometric facial details of the subject as there expression changes; then a rendering network is used to generate the detailed rendering result using the 3D face geometry information. 
2. The two component models are trained separately using a large scale in-the-wild images and a small video dataset. During the training of the rendering network, the novel Augmented Wrinkle Loss and Detailed Shading Loss were used. 
3. The authors further evaluated the method and compared it to DECA to show that the proposed method could generate plausible facial details for any desired expressions and could render the result photo-realistically.
4. Ablation studies were also done to prove the effectiveness of the components in the proposed method. 
main_review: Strengths:
1. This paper proposed a two-stage pipeline that could generate the facial details of any target expressions and render it realistically using just one input image.
2. A detail hallucination network was proposed and trained. 
3. The Augmented Wrinkle Loss and Detailed Shading Loss were proposed to train the rendering network.
4. The authors evaluated the method quantitatively and qualitatively, also compared it with the DECA method. 
5. The authors further showed that the rendered results are view consistent, proving the effectiveness of the rendering network. 
6. Ablation studies are done to show the effectiveness of the two novel losses.
Weaknesses:
1. The readability of this paper can still be improved. For example, a pipeline figure would help in explaining the overall picture of the proposed method; alpha_s is not in the input of Equation (2) although it appears in Fig 2. 
2. The figures in this paper could be improved to show the details clearly. For example, in Fig 5, it seems that the lighting/shading of the two subjects' results are quite different, making it hard to see the facial details. The same problem also exists for Fig 4&6. Also,I would suggest the authors to show the zoom-in details in the rectangles as in Fig 1 rather than ask the readers to zoom in for details.
3. The proposed method in this paper is not well-supported by experiments. First, it seems that the image quality of the rendered view is much more blurry than the input image and also the previous work [Chen et al. 2019]. Second, since the proposed method is built on [Chen et al. 2019], it would be fair to include this work for comparison as well reconstructed on the expression of the input image. Lastly, there're no evaluations of the usage of the age prediction network and FaceID embeddings. It would be interesting to know that age information make the proposed method outperforms the others in generating results for people across different ages, etc. 
summary_of_the_review: This paper proposed a pipeline FaceDet3D that can generate plausible geometric facial details from a single image given any desired target expression. Novel losses were proposed and the model was trained and evaluated to show its effectiveness. 
However, I think this paper is still under the bar of acceptance as the experiments are not very convincing. Also, the readability of the paper could still be improved.
review_title: null

Meta-review: The reviewers raised a number of major concerns including a poor readability of the presented materials, incremental novelty of the presented and, most importantly, insufficient and unconvincing ablation and experimental evaluation studies presented. The authors’ rebuttal failed to address all reviewers’ questions and failed to alleviate reviewers’ concerns. The authors explain that due to the lack of time they could not complete all experimental studies. A major revision of the paper is needed before the paper can be accepted for publication. Hence, I cannot suggest this paper for presentation at ICLR.
Decision: Reject