Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: SUMMARY: Unsupervised/Self-supervised generative model for image synthesis using 3D depth and RGB consistency across camera views

CLAIMS:
- New technique for RGBD synthesis using loss in 3D space
- Can disentangle camera parameters from content (I disagree slightly with "disentangle" since you are conditioning on camera parameters in the first place)
- Different generator architectures can be used

METHOD:
Generate RGBD images of 2 different views, have an adversarial loss on the RGB image, have a content loss between RGB1 and warp(RGB2), have a depth loss between D1 and warp(D2)
Equation 5:
- Possibly either "c_{1->2}" needs to be replaced by "c_{2->1}", or "G_{RGB}(z, c_1) - warp(G_{RGB}(z, c_2), c_{1->2})" needs to be replaced by "warp(G_{RGB}(z, c_1), c_{1->2}) - G_{RGB}(z, c_2)" (or am I missing something?)
- Not entirely sure why there is a different "projection" operation, since both "warp" and "projection" are calculated from Equation 3. I understand that "warp" is the combined Rt matrix that is estimated using the two views and Equation 3, assuming that the "d"s are correct. Not sure what "projection" does though, possibly explain it better?

DECISION: Very clearly written paper, simple idea executed well

The paper is clearly written and well organized. It uses a simple idea, and performs sufficient number of experiments to explore the idea. It is not very novel, but the paper shows its applicability with multiple architectures as a bonus.

The figures showed results almost only from their method. It would be great to pick one generator architecture, and elucidate more on the differences between not using their 3D loss and using it. Good attempt though.

ADDITIONAL FEEDBACK:
- Might not be "representation learning", instead it is learning a generative model.
- "3 EXPERIMETNS" -> "3 EXPERIMENTS"
- The appendix should have more details on the equations and the specific formulations of warp  and projection operations
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The submission proposes a technique to learn RGBD image synthesis from RGB images. A distinctive feature proposed by the technique is the user-controllable camera rotation parameters, learned in an unsupervised manner. The technique can be used in conjunction with various models, such as PGGAN, StyleGAN, and DeepVoxels.

This paper provides an interesting approach that can be a useful building block for future investigations.

The main issue I see with the paper is the number of results provided. Only 2 different images are shown per combination of model and dataset, limiting the reader's ability to assess the technique's performance. Would it be possible to provide a large number of results in a supplementary material or appendix?

In my opinion, this may be due to a difference in writing style, but the paper, in general, is slightly hard to read.

The depth in figures 1, 4, 5, 7 and 9 would be easier to read if it was displayed as a colormap (with the corresponding color bar) instead of grayscale. Additionally, a reference sphere would be appreciated near the normal maps shown in fig. 6 to inform the reader of the coordinates system used.

Sec. 3.3 states that “the depth of the background is smaller than that of the face [for DeepVoxels]; however, this does not occur when the proposed loss is used”, however fig. 6 seems to show the contrary. Is it due to the depth discontinuity?

Minor details
- Sec. 3 “Experimetns”: typo.
- Sec. 3.3, “[...] use the 2D CNN”: I would replace “the” by “a”.
- Sec. 3.3, Third paragraph, the first sentence is hard to read.
- Sec. B “the later voxels are ignore[d]”

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: # Review ICLR20, RGBD-GAN

This review is for the originally uploaded version of this article. Comments from other reviewers and revisions have deliberately not been taken into account. After publishing this review, this reviewer will participate in the forum discussion and help the authors improve the paper.

## Overall

The article proposes a method of modifying image-generating networks to also produce depth maps in an unsupervised way by enforcing rotational consistency.

I enjoyed reading this work and I'm recommending it to be accepted. However, first there are some (in my opinion straight-forward) changes that need to be made to this work before I can recommend its publication: 

- The common "Related Works" section is missing and some of the literature is taking place in the introduction. I find this unorganized and I'd recommend keeping the intro shorter and just moving the literature either behind the intro or to the end of the paper.
- Most figures and especially your headline figure (1) suffer from not having the depth normalized and not having a scale to it. The fix for this is simple and two-fold: for each depth image, subtract the minimum value and divide by the range (to normalize it and increase contrast), then write in the caption or as a legend that white is closer to the camera and black is further back.
- 3D vs. 2.5D - If the common geometric definition of "3D" was applied here, the article's title was correct. However, in computer vision and especially 3D vision, the term is commonly used to refer only to models that include full scene geometry, including the occluded backs of objects and the term 2.5D is used to describe assigning depth values to pixels in an RGB image (and therefore only covering the view-dependent front of the object), which I think is the case here. However, this is not a hill that I'll die on so if you insist on that terminology, I won't block acceptance.
- When you first discuss HoloGAN, you mention one of its main downsides being scalability and then proceed to not only explain that but also use a HoloGAN-like architecture in one of your experiments. I'd either remove the scalability argument or justify not just that but also how that's not relevant to your experiments.
- The following phrase occurs multiple times throughout: "camera parameter conditional image generation". I _think_ you're missing a dash between "parameter" and "conditional".


## Specific comments and questions

### Abstract

All good.

### Intro

- Fig.1 normalize image 
- The literature section in intro mentions "For all methods, 3D annotations must be used..." - that's not true. See [Rezende, 2016][1] and [Rajeswar, 2018][2]
- I understand how some literature is required to position your method, but I think it's better to not have the entire literature section in the center of the introduction

[1]: https://arxiv.org/abs/1607.00662
[2]: https://openreview.net/forum?id=BJeem3C9F7

### Method

- 2.1 clear + nicely written
- Figure 2 good, caption a bit too short - figure+caption should be able to stand on their own
- Illustration of Figure 3 nice, except for unclear DeepVoxel part: what's the wavy orange flag stand for?

### Experiments

- You mention K is fixed, but where does the initial K come from? I assume it's just neglected (since it's not important for StyleGAN/PGGAN), but then this needs to be mentioned in the methods sections closer to the formulas dealing with K.
- Figure 4 - the depth maps need to be normalized. All we see here is a grey mush, even worse in Fig. 7
- For ShapeNet cars, the model seems to suffer from not having a reference for the top and bottom of the image - have you tried adding floor/sky?
- Figure 6, the tire marker is a good idea but image still unclear - I recommend slightly less rotation or an intermediate step between generated image and e.g. front view
- For quantitative results/FID: try using Hausdorff or Chamfer distance on the rendered scenes' pixels. We don't care about the goodness of the RGB generation but the depth.

### Conclusion

All good, albeit a bit short.

### Appendix

I don't think I saw any references to the appendix in the main paper.
review_title: Official Blind Review #1

Meta-review: The paper has initially received mixed reviews, with two reviewers being weakly positive and one being negative. Following the author's revision, however, the negative reviewer was satisfied with the changes, and one of the positive reviewers increased the score as well. 

In general, the reviewers agree that the paper contains a simple and well-executed idea for recovering geometry in unsupervised way with generative modeling from a collection of 2D images, even though the results are a bit underwhelming. The authors are encouraged to expand the related work section in the revision and to follow our suggestion of the reviewers.
Decision: Accept (Poster)