Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a 3D-Aware Video Generation pipeline. It integrates a GAN model and a time-conditioned implicit field to enable the pipeline to generate 4D videos. The proposed method encodes each video into a 3D content code and a motion code. These two codes, together with the camera pose, determine the video content. In discriminator, the proposed pipeline separately evaluates the quality of each frame and the rationality of the co-occurrence of two given frames in the same video.  
strength_and_weaknesses: Strengths:

1. The proposed architecture successfully combines GAN and Implicit field and is the first work to generate 4D videos with GAN.

2. The proposed method outperforms existing works remarkably in terms of ACD and ID. It suggests that this paper significantly improves the temporal consistency, which can also be concluded from qualitative comparison.

3. The paper is clearly written and easy to follow.

Weaknesses:

1. Some tricks are interesting, but the related experiments are insufficient.

    - Image pre-training seems to contribute a lot to the performance. However, there are few ablation studies about this option. In Supplementary Section E, the authors only provide two groups of figures to compare pre-training and joint training. Authors should provide quantitative results about it.

    - Authors claim that one of the key components of the proposed framework is an image-based discriminator. Can we use a deterioration of video-based discriminator (e.g., let $t_k=t_l$) to replace the image discriminator?

    - More qualitative results about the motion generator should be provided. 

2. In Table 1, StyleGAN-V outperforms the proposed method in aspects of FVD and FID. Does it mean that the implicit field causes many difficulties for the model to fit the original distribution?

3. Authors should clarify more about the differences between DIGAN and the proposed method. In my opinion, the only significant difference is that DIGAN uses INR directly, while this paper exploits Nerf to equip the model with more 3D information. In the architecture design, the split of FG&BG and the motion generator are the new components compared to DIGAN. However, the former component has been applied in the related works \[1-3\]. It is hard to identify the contribution of this paper in its current form.


\[1\] FiG-NeRF: Figure-Ground Neural Radiance Fields for 3D Object Category Modelling

\[2\] LOLNeRF: Learn from One Look

\[3\] NeuralDiff: Segmenting 3D objects that move in egocentric videos
clarity,_quality,_novelty_and_reproducibility: Clarity: well, easy to read and understand

Quality: well

Novelty: not very

Reproducibility: looks OK
summary_of_the_review: An interesting study with some novel designs and extensive experiments. But the novelty should be further justified.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This works presents approach to multi-view-consistent video generation, whereby the camera viewpoint in a generated video can be adjusted (or the same video can be generated from multiple camera viewpoints/perspectives). Furthermore, it is able to achieve this using 2D image/video streams. By using time-conditioned implicit fields, this work extends methods based on neural implicit respesentation to the video domain, crucially without the requirement for explicit 3D data/supervision. They classify their model as a 4D generative model, i.e. a 3-D aware video generator.
strength_and_weaknesses: **Strengths**:

- Clear and well written paper, with a well motivated approach to incorporating 3D structure in video generation models and tasks.
- Great treatise of the capabilities and limitations of the proposed approach.
- Deep consideration of the ethical challenges and threats of this research (and a strong rejection and condemnation of misuse of such work)
- Exhaustive consideration of prior and related works.
- The method appears to be computationally efficient and quite performant when compared to prior art (e.g. Table 1.)
- Extends the use of INRs to 3D video generation using only 2D image/video data. Whereby the viewpoint in the generated video is controllable parameter.

**Weaknesses**:

- The proposed approach is limited to simple datasets with easy foreground/background separation and limited objects in the foreground. This weakness is acknowledged in the paper and is in general a challenge for the broader field.
- The approach is limited to generating 16 frames of videos. Again, this weakness is acknowledged in the paper and is in general a challenge for the broader field.
- Code to reproduce the main experiments is missing.
clarity,_quality,_novelty_and_reproducibility: - **Quality**: This is a high quality article detailing a concrete and unique contribution to the video generation literation.
- **Clarity**: This article is well articulated and clear. It exhuastively covers the proposed approach in a manner that makes it straightforward to understand and replicate the contribution.
- **Originality**: This work is original and unique. It extends the capabilities of current approaches to facilitate a new capability not encountered before in this line of research.
summary_of_the_review: Overall, this is a great and significant contribution to the video generation literature. It is well-polished and provides for a concrete and unique contribution to the field. I support this papers publication to ICLR.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a system to generate multi-view consistent videos. We have only seen 3D GANs on image domain so far and this paper extends it to video domain. There are two core components: a time-conditioned 4D generator and a time-aware video discriminator. Experiments are conducted on 3 datasets ranging from talking faces to moving people. The results are promising and encouraging.
strength_and_weaknesses: **Strength**

- Extending the 3D-aware GAN to video domain is an interesting direction and this paper proposes a legitimate first solution to this problem.
- The proposed two modules time-conditioned 4D generator and time-aware video discriminator make a lot of sense and work well in practice.
- The results are encouraging and the generated videos are view-consistent. 

**Weakness**

- Claiming the time-aware video discriminator to be one of two core contributions seems inappropriate. This module is introduced in DIGAN (Yu et al., 2022) and no further changes are made in this work. The original module is also introduced for video GAN.
- It's a bit strange to me that in Eq.1 the time is used as a multiplier. What's the motivation behind this? When time step is 0, this will make entire feature to be 0-vector. Are the other options such as concatenation or positional encoding tested?
- The writing is messy and inconsistent. For example,  Eq.1 use $\psi^3\circ\psi^2\circ\psi^1$ for MLPs, the mapping network is denoted as $w=MLP(z)$, and Eq.3 use $\phi_{\sigma}$ to denote 2-layer MLP. Another example, there is an extra comma in eq.4 after $\gamma(d)$. Another example, $n$ is defined as a vector above Eq.1 but it's used as a function in eq.1.  These are just few examples. Overall, there is lots of room for writing improvement.
- In the approach it's mentioned there are two training options (section "Training on Image and Video Datasets"). Which performs better? I don't find further discussion or ablation study in sec. 4.3. 
- The visual quality isn't very satisfying. I can see artifacts like extra arms Fig.3. Also, these results are of low-resolution and the local fine details are not recovered well. The quality of the generated faces also falls far behind the image 3D GAN. 
- Even though the evaluation metrics are not perfect for evaluating 3D GANs, the quantitative results are just competitive to 2D methods and sometimes worse (Tab.1).
- Only one 3D static baseline (StyleNeRF) is tested. It's not enough and also StyleNeRF itself isn't the state-of-the-art method.
- No geometry is visualized or shown in the paper. It's important to demonstrate the learned 3D for 3D GANs in my opinion. Does the network learn reasonable structure from the input data?
- I don't understand why Tab.2 and Tab.3 are so incomplete compared to Tab.1. 
clarity,_quality,_novelty_and_reproducibility: **Clarity**

As pointed out in the above section, some of the writing isn't very clear. There remains lots of improvement space especially for the approach section.

**Novelty**

One of the claimed contribution (time-aware video discriminator) isn't novel. For the other contribution (time-conditioned 4D generator), I have some doubts about the way of time conditioning as stated above.

**Quality**

The visual quality isn't very satisfying and I think the experiments are incomplete and needs some further efforts. 

**Reproducibility**

Implementation details are provided in Appendix but no code is submitted. 
summary_of_the_review: I have concerns in multiple aspects of this work including writing, experiments, and technical contribution. It's unlikely that these weaknesses can be properly addressed during rebuttal period. Therefore, I recommend rejection. However, this paper explores an interesting yet challenging direction and I do encourage the authors keeping improving the quality of this work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose 4D generative adversarial networks (GANs) to learn the unconditional generation of 3D-aware videos in neural implicit representations. The experimental results show that the proposed method learns a rich embedding of decomposable 3D structures and motions whose quality is comparable to that of existing 3D or video GANs.
strength_and_weaknesses: **Strengths**

- The paper is clearly written.
- It is the first paper to generate a 3D-aware video in neural implicit representations.

**Weaknesses*

- The technical contribution should be clearly demonstrated at the end of the Introduction.
- According to the results in the supplementary material, the generated videos are quite short and the background images are not consistent among frames.
- The proposed method should be compared to the NeRF-based methods for dynamic scenes, such as D-NeRF.
clarity,_quality,_novelty_and_reproducibility: - The paper is clearly written.
- The contributions are only marginally significant or novel.
- The authors have not provided the code and enough details for reproduction yet.
summary_of_the_review: Please refer to the strengths and weaknesses.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes an approach for 3D-aware video generation, based on implicit representations and adversarial training. The method is evaluated on several datasets, where it approaches or beats relevant baselines (2D video or 3D static image)

The reviewers' evaluation of the paper is mixed, leaning towards negative. The main points are as follows:

Pros:
1. A new task of 3D-aware video synthesis
2. A reasonable solution that works quite well in practice
3. Good presentation, extensive details, and extra experiments in the appendix

Cons:
1. Limited technical novelty: mostly a combination of known components
2. Not super impressive results, the method does not always beat the baselines, 
3. Some missing baselines (D-NeRF, 3D static methods beyond StyleNeRF) and analysis experiments (effect of joint/sequential training, motion vector design)
4. Code not released

Some of the concerns (3, 4) have been addressed in the rebuttal. However, some others (1, 2) haven't quite been addressed (reviewers in a private discussion pointed out that they aren't changing their ratings) 

Overall, it is an interesting paper that has its merit; however, the technical contribution is limited and the results are somewhat mixed. Based on this, at this point, it falls slightly below the acceptance threshold, and I recommend rejection, but I encourage the authors to polish the paper and resubmit to a different venue.


justification_for_why_not_higher_score: 1. Limited technical novelty: mostly a combination of known components
2. Not super impressive results, the method does not always beat the baselines,
Decision: Reject