Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: 
This paper proposes a method that marries parametric face models and neural radiance fields to achieve view synthesis of a portrait that provides expression control. Another view of this work is extending Park et al.'s Nerfies to support expression control and driving. 

The paper makes three contributions: Firstly, it proposes a dynamic NeRF model that supports certain explicit control. Secondly, the proposed method disentangles the facial expression from the appearance of the background with a simple masking scheme dubbed as spatial ray sampling prior. Finally, the method enables the simultaneous control of viewpoints and facial expressions.
main_review: 
This paper presents a straightforward but sensible method that benefits the best of both worlds: it utilizes the very strong prior of human faces by using a parametric face model and meanwhile uses the powerful NeRF to handle complex appearance, including that of the background. The quantitative experiments suggest that the proposed method outperforms the baseline methods. Qualitatively, it achieves something that prior methods don't: It allows for expression control that Nerfies don't support, and it handles the background better than NerFACE because it disentangles expressions from backgrounds.

Now the weaknesses:

Since the method relies heavily on how well DECA and landmark fitting work, it should be discussed whether there is any measurement taken to recover from DECA's and/or landmark fitting's mistakes. If I understand correctly, there is no such error recovery scheme in the model design. This seems like a major flaw to me. Just to name one potential failure that DECA may suffer from: The camera extrinsics from DECA may not be accurate, and most of us would agree NeRF-like models rely on good camera poses to yield sharp rendering.

The video results look rather blurry to me maybe partially because the model was trained at 256x256. I think for the method to be really useful, the authors should explore why such blurriness still exists with the ray sampling masking scheme and the deformation fields already in place. There are also flickering artifacts that "morph" the person's head when the expressions are driven. 

I believe there is complex dynamics among the expression parameter beta_i, the deformation code omega_i, and the appearance code phi_i: For instance, how does beta_i interact with omega_i? Intuitively, when the expression changes, the deformation field will have to adjust itself to work with the new expression. This dynamics, or rather this entanglement, doesn't sound desirable to me. This deserves at least some discussions, if not experiments.

Equation 6: Why wouldn't the model get into the trivial solution of having a no-op deformation field D? That would give you a perfect zero loss for Equation 6. 

It's strange to have 3.4 ON THE NECESSITY OF A SPATIAL RAY PRIOR as a separate section than 3.3.

There doesn't seem to be any quantitative ablation study. Although we can see qualitatively the disentanglement, it'd be nice to confirm this with numbers.

Fig. 5 shows really similar-quality results between the proposed method and NerFACE in terms of just the reenactment, although I agree the proposed method does perform better overall with fewer artifacts. That said, this observation, together with the fact that the reenacted expressions don't resemble those in the driving frames (e.g., the expressions in Driving Frames 1 & 4 are very different, but they both lead to similar reenacted expressions), casts it into question whether the proposed method controls the expressions well. 

Missing references:
- In the human domain, the authors should cite:
    Neural Light Transport for Relighting and View Synthesis
  which does perform view synthesis of human subjects using a "neural rendering" approach.
- Furthermore, this paper:
    Deep Relightable Textures Volumetric Performance Capture with Neural Rendering
  should also be cited, since it does handle moving subjects.
- I think there should be a section on "Parametric Face Models" that are dedicated to 3DMM, FLAME, etc. since half of the model name "Flame-in-NeRF" lies in that domain. The current Related Work sections focus on just the "NeRF half."

Minor nits:
- Page 2: "Lassner & ZollhÃ¶fer"
- Page 3, grammatical errors: "Nerfies (Park et al., 2021) too work", and then "frame, however it"
- Page 5: "are able generate" & "this phenomena of"
- Fig. 4 seems unnecessarily verbose since Eq. 5 already says the same thing very clearly.
summary_of_the_review: 
This is a paper that clearly explains a straightforward but sensible approach. I judge it to be slightly below the acceptance bar mainly because (1) the method heavily depends on the success of two non-trivial preprocessing steps and has no way to recover from their mistakes, and (2) the complex dynamics among the expression parameters and deformation fields remains unexplored.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The submitted paper focuses on novel view synthesis from portrait video with controllable facial expressions. The authors separate the modeling of the static background from the dynamic foreground by segmenting out the faces (using landmark detection and fitting). The faces are modeled with Nerfies with expression parameters as latent codes, while the static background is essentially modeled with a standard NeRF without conditioning on per-frame expressions. The proposed method was evaluated on 4 subjects with data collected from a phone. The authors also demonstrate an application of the proposed method where videos can be synthesized with consistent facial expressions as a driving video as input.  
main_review: Strengths:
1. The submitted paper demonstrates the interesting use case of controlling video synthesis with a driving frame/video as input. 

Weaknesses:
1. Missing important baseline/citation: HyperNeRF is an improvement over Nerfies and should be considered as a baseline method: "HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields"
2. Quality of synthesized videos: the synthesized videos are low-resolution, blurry, and there are noticeable floaters around human subjects. I don't consider the quality to be on par with Nerfies or HyperNeRF.
3. Limited novelty: the proposed method is a relatively straightforward extension of Nerfies but with masks and facial expressions as the latent code.
4. Limited evaluation: there are only 4 subjects being used as evaluation data
summary_of_the_review: My current recommendation is "3: reject, not good enough". There is limited novelty with important baseline missing (HyperNeRF). The synthesized videos are also blurry with artifacts.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a method for creating animatable portraits within a static scene based on dynamic neural radiance fields. 
The main idea is to fit a 3DMM-like model to a set of portrait frames and use it to rasterize a binary mask which is used to enforce distanglement between the foreground and background when training a dynamic NERF. Both qualitative and quantitative results seem to be convincing.
main_review: 
**Strenghts:**:
- Writing quality. The paper is well-written and is easy to follow.
- Method. The general idea behind the method - to use a spatial disentanglement mechanism to separate effects of conditioning variables - is reasonable and is definitely worth investigating.
- Experimental evaluation. Authors provide a comprehensive comparison to very recent baselines, and show that both qualitatively and quantitatively their method performs slightly better.

**Weaknesses:**

- Method limitations. Authors do hint at a very important issue which is common for animatable models - which is entanglement among different factors of variation among the control signals.  However, the scope of the solution that they propose seems quite limited: effectively the only separation that one gets is between the background and foreground. This means that there is no mechanism for the model to separate independent expression parameters, and even the disentanglement between the viewpoint and expression is not fully guaranteed. It would be interesting to see animation on sequences where a single expression parameter of the time is varied while others are being fixed, and similarly fixing expression vector while varying the view to see if there are any spurious correlations, in order to better understand the scope of this issue. 

- Related work - geometrical prior. The idea of using an explicit geometrical or spatial prior to improve generalization is not particularly novel, even if one only considers the context of neural radiance fields. In particular, most of the work on neural rendering for bodies (NeuralBody, Neural Actor) use SMPL or similar model, which acts as a 3DMM. Although those works do not model the background, ideas presented in those works will likely generalize better as they also encourage spatial disentanglement between different parts of the articulated object. Very similar ideas can be trivially extended to faces data. Moreover, the assumption that the appearance of the background is independent from the expression is not correct - e.g. shadows depend on those? 

- Related work - NerFace. Proposed method is very similar to the existing method - NerFACE. The only difference seems to be in the way the two methods treat background? If that is the main motivation for this work, ideally it should be made more clear. It would also be great if authors describe what would be a difference between their method and simply training a separate NERF model for the background (using pre-computed segmentation masks, and simple ad-hoc blending at inference time). I can see that training a joint model might be beneficial, but ideally this should be demonstrated empirically. 

- Experimental evaluation. There are no video materials that compare proposed method to the baselines, only static comparisons. I would also encourage authors to provide failure cases of their method to have a better idea about the limitations. As mentioned earlier, varying independent control variables - view, individual expression variables - can provide a way to qualitatively evaluate how animatable the model actually is.

**Misc:**
- P5: able to generate
- P8: artefatcs
summary_of_the_review: Although qualitative results of the paper seem convincing, the method itself is very similar to an existing one (NerFACE), while the main claimed novelty - the spatial ray prior - seems to be equivalent to training a separate background model with pre-computed segmentation masks. Thus I am leaning more towards rejecting this paper, but I am eager to reconsider my rating if authors address the concerns raised above.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces a method for novel view synthesis of a dynamic scene with a person in varying facial expression. To enable the controllability of facial expression, a parametric 3D face model is combined with existing neural radiance field-based approaches. To overcome background artifacts (e.g., distortion) while animating, the spatial face prior is added in a form of binary mask. The method is evaluated on four customized videos. 

main_review: Strength:
+ Unlike previous works, the proposed framework enables explicit control of the human facial expression while synthesizing an image from a novel viewpoint.
+ The writing is clear and easy to follow.


Weakness:
- Lack of novelty. The key idea, i.e., combining foreground masks to remove the artifacts from the background, is not new. Separate handling of foreground from background is a common practice for dynamic scene novel view synthesis, and many recent methods do not even require the foreground masks for modeling dynamic scenes (they jointly model the foreground region prediction module, e.g., Tretschk et al. 2021).

- Lack of controllability. 1) The proposed method cannot handle the headpose. While this paper defers this problem to a future work, a previous work (e.g., Gafni et al. ICCV 2021) is already able to control both facial expression and headpose. Why is it not possible to condition the headpose parameters in the NeRF beyond the facial expression similar to [Gafni et al. ICCV 2021]? 2) Even for controlling facial expression, it is highly limited to the mouth region only. From overall qualitative results and demo video, it is not clear the method indeed can handle overall facial expression including eyes, nose, and wrinkle details, and the diversity in mouth shape that the model can deliver is significantly limited. 

- Low quality. The results made by the proposed method are of quite low quality. 1)  Low resolution: While many previous works introduce high-quality view synthesis results with high-resolution (512x512 or more), this paper shows low resolution results (256x256) for some reasons. Simply saying the problem of resources is not a convincing argument since many existing works already proved the feasibility of high resolution image synthesis using implicit function. Due to this low-resolution nature, many high-frequency details (e.g., facial wrinkles), which are the key to enabling photorealistic face image synthesis, are washed out. 2)  In many cases, the conditioning facial expressions do not match that from the synthesized image. From the demo video, while mouth opening or closing are somehow synchronized with conditioning videos, there exists a mismatch in the style of the detailed mouth shape.  
summary_of_the_review: While this paper add a controllability in the facial expression for a novel view synthesis by combining the techniques from previous works, the idea is not novel (which shows many overlaps with many existing NeRF based approaches), and the image synthesis quality is not convincing (low resolution without sufficient high-frequency details, notable jitters, and mismatch between the synthesized facial expression and conditioning ones) when compared to how existing methods have demonstrated. Although the proposed method performs better in several customized scenarios, but those testing examples do not cover diverse scenes that can truly validate the performance. Based on the current demonstration, the ability to control the facial expression is highly limited to a few number of mouth shapes (it's more like mouth control, not facial expression control). Considering the novelty and quality in the context of existing methods, the initial rating is reject.
review_title: null

Meta-review: This submission received 4 ratings, all below the acceptance threshold. The reviewers expressed concerns around overall novelty of contributions and quality of produced results, and also pointed out lack of comparisons with some prior works and gaps in empirical evaluation. The authors responded to most of these comments, but did not convince the reviewers to upgrade their ratings.
The final recommendation is therefore to reject.
Decision: Reject