Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces the first 3D generator that works on 3D datasets without alignment. To this end, conditioning on an off-the-shelf monocular depth estimation approach, the proposed method incorporates generic depth priors to facilitate 3D generation. Moreover, richer camera models are taken into account, ensuring more realistic generation results. Finally, with a distillation mechanism,  a pre-trained image classification network provides further supervision to stabilize and improve training. Experiments on standard datasets validate the effectiveness of the proposed method.

strength_and_weaknesses: **Strength**
- The paper is well motivated as existing methods do not work well on non-aligned data.

- Using an off-the-shelf depth estimation as a generic prior is intuitive, and experiments show that such prior helps improve the generated quality.

- I find this paper easy and enjoyable to read and follow.
 
- The visual generation quality is significantly higher than compared methods.

- The learnable Ball-in-Sphere camera model does have a higher than a standard with 2 dof and works for objects that are not center-aligned.  

- The experiments are carefully and extensively conducted. I appreciate that failed attempts are also mentioned in the appendix, which helps the audience and following researchers better understand this work.

** Weaknesses**

- The usage of an off-the-shelf depth estimator may make the comparison with other depth-estimator-free methods unfair. Plus, how do the performance and generalization of the monocular depth estimator affect the performance? Will the performance improve if trained with ground truth depth (for example if it is trained on a depth estimation dataset or a synthetic 3D dataset)?

- The gradient penalty is widely used and studied in GAN-based approaches, the authors simply extend that to the parameters of the camera model. How good is the method if multiple objects are in the scene? The authors imply that their model works well on "a scene 
 consisting of multiple objects", but I cannot find any experiments verifying this claim.

- The so-called knowledge distillation for discriminator is more like an engineering trick and does not provide much technical insight to me. Moreover, the authors mention that "it can work with arbitrary architectures of the discriminator," but it can only work on a discriminator that produces features with the same dimension as ResNet50. I am also wondering if this L2 KD loss works better than the mostly used KL divergence loss.

- This paper should compare with a 3D photo baseline [a]. Combining a vanilla 2D image generator with such 3D photo generation methods could also produce realistic 3D image synthesis results while it seems to generalize on a larger range of images.  I wonder how the performance differs for these two different generation paradigms. Will the proposed method have a higher multi-view consistency? Will the proposed fail while the 3D photo methods work better in certain situations? Thus, this line of work should also be discussed.

[a] 3D Photography using Context-aware Layered Depth Inpainting. CVPR 2020.


clarity,_quality,_novelty_and_reproducibility: This paper explains its motivation and ideas well. The technical novelty is good. Some technical components in this work are of fair novelty. Many details are provided to ensure reproducibility. However, I found there are many hyperparameters (e.g., in Eq 5) that may make it hard to reproduce the results.
summary_of_the_review: Based on the aforementioned pros and cons, I vote for a weak accept of this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper extends the 3D GAN to more diverse and un-aligned datasets (eg., ImageNet). This problem setting change is significant and very challenging, as we have learned from 2D fields (StyleGAN3--> StyleGAN-XL). This paper introduces several new modules to adapt an existing model EpiGRAF to this new setting including 1) depth-regularized GAN, 2) learnable camera formulation, 3) a knowledge distillation loss. Results on various dataset including ImageNet are promising in both rendering and geometry quality. 
strength_and_weaknesses: **Strength**
- The biggest limitation of existing 3D GAN is the applicable data. This paper extends the 3D GAN to a more challenging setting where large-scale un-aligned dataset is used.
- The introduced components are very interesting and works well. 
   1. The learnable camera formulation is very useful and can be a standard thing for follow-up works in this field. I think this is very important for models object-centric images in the wild. 
   2. The motivation of using depth as regularization makes lots of sense. Even though it's not easy to leverage estimated depth data, the proposed method managed to being beneficial.
   3. The knowledge distillation loss is particularly useful for ImageNet. 

**Weakness**

- My biggest concern is the visual quality. From the provided annoymous link, the generated images suffer from many artifacts. Even though the "flat" issue is addressed compared to EG3D, the visual quality is merely improved. Both of them fall far behind the 2D generators. 
- Even though using depth is resonable, it also makes the framework more complicated. It's quite tricky to do the Adversarial Depth Supervision (ADS) and set the p(d) correct. 
- More 3D GAN baselines besides EG3D (tri-plane) could be tested. It's possible that the "flat" issues are caused by the tri-plane representation.  Therefore, a volume-based or MPI-based 3D GAN could have very different results.
- The loss in Eq.1 is somewhat weird. To me, this loss is minimizing and maximizing the same thing simultaneously. Why does this loss make sense? What does it converge to? What's the optimal value for this task? 

**Doubts**

- Is “Ball-in-Sphere” formulation good enough for any images in-the-wild? What stops us from making it more flexible? For example, why do we need the camera to stay on the hemi-sphere? Why do the loss weights for the three components (pos, fov, lookat) differ so big?
- How are the camera priors decided? In Fig.7 most priors are relatively uniform, but pitch/look-at are more like gaussian. It's unclear to me how to set these priors. Do we need to choose different priors when dataset changed?  Also, it's also unclear to me why residue based method fails so bad.
- Why squared loss for distillation? This is not the standard option for knowledge distillation. Does it outperform standard setting, ie., KL-divergence? 
- Why not applying a depth loss on adaptor output? For example, applying LeReS on the rendered image to get the ground-truth depth and use it to supervise the output of adaptor.

**Writing**

- For 2D methods (StyleGAN2 (with KD)) in Tab.1-2, it's better to change their NFS score from 0 to N/A. 
- The qualitative results from the link could be added to the paper (appendix is fine).
- The paper should be self-contained and thus R1 gradient penalty should be given in Appendix.
clarity,_quality,_novelty_and_reproducibility: **Clarity**

This paper is clear overall. My specific doubts and concerns are mentioned in above section. Appendix is useful and informative. 

**Novelty**

There are mainly 3 introduced modules: 1) depth-regularized GAN, 2) learnable camera formulation, 3) a knowledge distillation loss. All of them are interesting and novel for 3D GAN framework. 

**Quality**

- Despite being novel and promising, the introduced novel modules aren't very easy to use and couple of additional modules (eg., adaptor, camera prior, additional hyper-parameters) need to be leveraged. 
- Also, the quality of the generated 2D images far way behind the state-of-the-art 2D generators. (as shown in 
 https://u2wjb9xxz9q.github.io./) Even though being view-consistent, there exist many obvious artifacts. Nevertheless, this paper is still an exciting first step in this challenging problem setting. 

**Reproducibility**
The paper provides fine implementation details in both main paper and the Appendix. However, it would be still quite challenging to reproduce without the source code since the propose framework is relatively complex.
summary_of_the_review: This paper tries to solve a very challenging task and proposes a legitimate framework consisting several new components. The results are promising and encouraging. I'm supportive overall and my current rating is 6 but not higher because the visual quality of the rendered images are not very satisfying. Also, the introduced framework is a bit complex since it incorporates several sub-modules/networks. 

For the authors: I'm sorry and please point out if I overlooked anything. Happy to chat more from here.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents a new method that learns a 3D-aware generative model on unaligned image datasets like ImageNet. 
The authors propose to use an off-the-shelf depth estimator to guide 3D generator. The estimated depth is concatenated to the real image and feed to discriminator to guide the generated image and depth from the generator. Additionally, a learnable camera model based on the ball-in-sphere geometry is proposed to support the diverse camera settings in unaligned data, which includes position, fov, and lookat parameters. 
The results show that the method can generate realistic images with depth on ImageNet dataset. 
strength_and_weaknesses: ** Strength **

- The use of depth priors is an interesting idea to guide 3D-aware generative model. 


** Weakness ** 

1) The use of the off-the-shelf depth estimator could, however, result in unfair comparisons as the depth estimator relies on another datasets for training. To be fair, the depth estimator has to be trained on the same dataset for the generative model, but this might not be possible on ImageNet. 
The effect of these additional data should be properly discussed. 

    a) Geometry prediction of the original depth estimator should be provided. 
  
    b) Compare a) to the geometry generated by the proposed method. 
  
    c) Compare the geometry generated by different depth estimators, and depth estimators trained on different datasets. Again, this point is not convincing to me still because the use of the additional data. 


2) More results on ImageNet should be presented as claimed in the paper title. For example, out of the 1000 classes on ImageNet, how many categories have the authors experimented? 
To evaluate the geometry quality, the authors might want to refer to real object datasets such as ScanObjectNN, and compare the reconstructed geometry with these objects in point cloud format with FID evaluation. 

    [a] Uy et al., Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data, ICCV 2019.

clarity,_quality,_novelty_and_reproducibility: - 3DGP can be a confusing name given EG3D is a previous work. The authors might want to reconsider this. 

- The pdf I receive has some lines on page 10 that violate the page format. 


summary_of_the_review: The paper itself is an interesting read and provides a step toward making 3D-aware generative models more practical and robust. My concern is that the experiments are unfair and could lead to confusion. I am happy to raise my score if the authors could convince me about the actual performance of the model given the off-the-shelf depth estimator.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to learn a generative model of 3D scenes from large unaligned datasets. The paper argues that previous work only produced good results for well-aligned datasets (e.g. human and animal faces), and that directly applying such methods to unaligned data did not work well. In order to address this problem, the idea is to incorporate an off-the-shelf monocular depth estimator (in this case, LeReS) into their model architecture. In their proposed model, a discriminator aims to discriminate between an true RGBD image (the result of applying LeReS to a real unaligned image) and an RGBD image rendered from their triplane image representation. The discriminator also outputs an embedding vector which aims to represent the semantic content of the image, effectively distilling a feature model (in this case, a ResNet-50) into the discriminator. Additionally, the paper uses a more flexible camera parameterization than previous work: the model predicts a field-of-view and look-at position in addition to the position of the camera on a fixed-radius sphere. 

The paper performs experiments on multiple datasets including a variety of LSUN categories and the full ImageNet dataset. It shows strong results compared to previous work using GANs (e.g. EG3D, EpiGRAF, etc.). 

strength_and_weaknesses: ### Strengths 
* This paper is an empirical work, and the empirical results are relatively strong. 
* There is no reliable quantitative measure of generated geometry, so the paper introduces Non-Flatness Score. NFS is a very simple heuristic, but it does seem to provide at least some insight into the flatness of generated geometry. 
* The NFS values (Table 2) show that for unaligned datasets, previous work (e.g. EG3D) does not learn reasonable depth values, whereas 3DGP does learn reasonable depth values.
* The proposed camera regularization strategy is intuitive. Figure 7 is an illustrative figure in this regard. 
* The inclusion of Appendix D (Failed Experiments) is very nice to see and much appreciated. The field would benefit substantially if sections like these were included in all papers. 

### Weaknesses
* The novelty of this paper may or may not be viewed as a weakness. On the one hand, there are no groundbreaking ideas; the paper uses components that have been used before and combines them to obtain strong results. On the other hand, the exact method has never been used before; nobody else has combined these components in the precise manner that is being proposed here. 
* For the ImageNet experiments, the method uses more time/compute than competing methods (28.2 days vs 18.7 days or 15.9 days; Table 2). Training for a similar amount of time as prior work would allow for a fairer direct comparison.
* When discussing GANs with external knowledge (at the top of page 4), the paper claims that “A similar technique is not suitable in our case as pre-training a generic RGB-D network on a large scale RGB-D dataset is problematic due to the lack of data.” However, it should be possible to train a generic (2D) RGB-D network on existing 2D image datasets by converting the 2D RGB images into RGBD images using monocular depth estimation; the paper effectively uses the same. 

#### Minor Weaknesses
* The method appears to be quite sensitive to hyperparameters (e.g. those on page 7). This is somewhat expected, as other GAN-based methods also tend to be quite hyperparameter-sensitive. 

### Questions:
* Existing 3D monocular depth estimation models are far from perfect. Are the failure cases from the monocular depth estimation network learned and reproduced by the generative network? This is mentioned briefly on Page 6 as motivation for the depth adaptor; it would be interesting to provide some more detail on this point. 
* Do symmetries (e.g. those with respect to the position of the camera on the sphere) cause any issues with learning the camera distribution? 
* With regard to the camera distribution, it seems strange to have the same camera distance for diverse ImageNet categories such as a goldfinch and a church. One can of course partially account for this by predicting different focal lengths, but have you tried learning a more flexible distribution of camera such as the inside of a sphere or a 3D annulus? Additionally, how does the distribution of learned focal lengths for different ImageNet classes look like? 
* Given that the model learns a triplane representation, it should be possible to rotate all the way around the scene. However, all of the examples shown are front-facing views with small-to-medium camera movements. Does the model learn reasonable backsides for objects? For example, the backside of the horses shown at the bottom of Figure 5. It is not critical for the paper to reconstruct the backside of objects. 


clarity,_quality,_novelty_and_reproducibility: The paper is written clearly. Although there is no code provided, the results appear to be reproducible. The novelty of the paper is discussed in the section above. 
summary_of_the_review: This paper proposes a method for learning 3D generative models from large datasets of unaligned images. The paper tackles an important 
topic in unsupervised 3D reconstruction. Although the paper does not have any groundbreaking ideas, it combines ideas and models from the existing literature in a different manner from previous work. It tackles an important and often-overlooked problem (that of generalizing existing methods to real-world datasets) and produces relatively compelling results.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents techniques for learning a 3D generative model given only 2D training data; it goes beyond prior work in that this 2D data does not require foreground/background masks nor (more importantly) does it require the 2D training imagery to depict objects from the same viewpoint.

Strengths: reviewers agreed that the problem addressed by this paper is very important, and the paper makes a significant step beyond prior work by not requiring aligned training images. The proposed method achieves higher-quality visual results than prior work as well. In addition, reviewers believed that some of the technical components contributed by the paper could be useful in other contexts, as well: the learnable camera formulation and the knowledge distillation loss.

Weaknesses: some reviewers commented that the gradient penalty & knowledge distillation loss contributions could be viewed as minor engineering tricks (though this concern was well-addressed in the authors' rebuttal). Reviewers also identified some visual artifacts in the generated results and wanted to see how the method would fare when generating views of all sides of an object, as opposed to just the front (which should be possible with this method). There was also a request to compare against additional 3D GAN baselines beyond EG3D, and some concerns about how well the priors used would generalize to new datasets.

After reading the author responses, three reviewers commented that their concerns had been addressed and were in favor of acceptance.
One reviewer gave an unusually low score (3), given the length of review / magnitude of concerns raised.
justification_for_why_not_lower_score: This is a very challenging problem that is of interest to a large subset of the ICLR community; a paper that makes headway on it merits extra attention. I waffled between "spotlight" vs. "oral," given the initially lower scores that this paper received, but in light of the thorough author response and the final opinions of the reviewers, I think it's reasonable to consider an Oral for this paper. 

I marked "This decision can be bumped down" for the question below this one b/c I wouldn't mind if this paper were downgraded to a spotlight, should the SAC have a different opinion.
Decision: Accept: notable-top-5%