Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the task of single image novel view synthesis. The key message is very clear. Methods with explicit 3D projection mechanism, like SynSin, better preserve observed regions, whereas methods with implicit renderer, like GeoFree, do better at outpainting (synthesizing unseen regions). And using both of them in the same framework can complement and reinforce each other and get the best of both worlds.

The paper does a very good job at conveying this message, and presents promising results on both indoor and outdoor datasets to demonstrate the benefit of this joint approach.
strengths_and_weaknesses: ## Strengths
### S1 - Message is clear and makes a lot sense
- The paper delivers the motivation of using both implicit and explicit rendering in single image NVS in a very clear and concise manner.
- The benefits of both worlds can be maintained by defining losses that use the better one to guide the other, ie, features from the explicit branch will guide the ones from implicit branch in observed regions, with the gradient of the former being detached, and the other way around in unobserved regions.
- The resulting model seems to overcome the shortcomings of both approaches, as shown in the comparison results.

### S2 - Clear and concise writing
- I enjoy reading the paper. It's very clear and coherent.

## Weaknesses
### W1 - Results are still a bit unsatisfactory
- First, I was expecting more qualitative comparisons in the supplementary material. There are only four examples shown. They're all good examples with large viewpoint changes. But showing more diverse results will be helpful.
- Based on the limited comparisons provided, it does not seem that the proposed method can fully preserve the benefits of both implicit and explicit methods. The quality of both observed and unobserved (outpainted) regions seems to be slightly compromised.
  1. Observed regions do not seem to be preserved very well, seemingly worse than SynSin and PixelSynth on indoor scenes. I think neither of them has been trained on images of outdoor scenery, so I'm not surprised they both fail on scenery images. The PSNR is also worse than SynSin with larger viewpoint changes.
  2. Outpainted regions also appear worse than GeoFree, although the latter uses an autoregressive framework.
  3. There seem to be obvious artifacts in the synthesized images, eg the center of the generated image in second row of Fig 4 (dinning table).
- Are the PSNR computed only on the seen pixels or on the entire image? Reporting both numbers will resolve the question in 1.
- Have the compared models been finetuned on the same dataset? If not, I do not think the comparison is fair.

### Minor comments
- Line 116: I suppose $f_0$ has a larger channel dimension than input image $I_{ref}$. Use different symbols for their channel dimensions.
questions: 1. Report PSNR both only on seen pixels and on the entire image, if it's manageable during the rebuttal.
2. Finetune the models of the previous methods on the same dataset, if this is not done currently.
3. Show more diverse visual comparisons, maybe in the final version.
limitations: The paper includes a brief discussion on the limitations. The current quality of the results are still quite limited. It would be helpful to provide some failure examples if any.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: There are two main paradigms in the task of single-image view synthesis. The explicit methods use the explicit 3D geometry and thus achieve good results on the reprojected pixels, while the implicit methods leverage implicit 3D inductive bias to implicitly learn the 3D geometry and could complete realistic out-of-view image regions. However, none of them can reach both goals. To mitigate such a seesaw problem, this paper bridges both implicit and explicit geometric transformations in a unified framework. Besides, a novel loss is proposed to fuse the geometry features from implicit and explicit branches. The proposed method can render state-of-the-art results on the novel views and achieve about 100x speedup in rendering compared to implicit autoregressive models.
strengths_and_weaknesses: Pros:
1. Combining the explicit and implicit geometry features in a unified framework is a reasonable and straightforward idea. Since explicit 3D geometry can preserve reprojected image areas and implicit 3D geometry helps to complete out-of-view regions. Meanwhile, the experiments on RealEstate10K and ACID datasets prove the effectiveness of such a combination.
2. The whole paper is succinct and easy to follow.
3. The ablation study in Sec. 4.3 & 4.4 demonstrates the necessity of the proposed Transformation Similarity Loss.

Cons:
1. From the main quantitative results in Table 2, the proposed method only shows superiority in terms of the FID metric. Although using both explicit and implicit geometric transformations, this method still works worse than SynSin. It seems that this method doesn't solve the aforementioned trade-off. For the lower FID score, this method just equips with a better imagination ability, which may be boosted with a more powerful adversarial loss. In lines 164~166, the authors mention they use a new GAN loss. How about the performance of SynSin + this new loss? Such a very important ablation study is not included in this manuscript.
2. The paper only provides the results on RealEstate10k and ACID datasets. How about the results on Matterport3D?
3. Given the better performance on the FID score, this method should compare with another two crucial baselines, Infinite Nature and Look Outside Room. I notice the authors' clarification in the supp. material. If available, the performance comparisons must be provided.
questions: In this manuscript, experiments are not comprehensive. Please refer to the questions proposed in the weaknesses.
limitations: 1. The idea is a little bit straightforward, just like a simple combination of A and B.
2. Experiments are not comprehensive and not enough to demonstrate the effectiveness of the proposed method.
3. Some confusing sentences inside:
- Lines 50 ~ 51: Our approach consists of architecture and loss functions.
- Lines 159 ~ 160: ... generating photo-realistic images. What's the relationship between rendering speed and image quality. The emphasis on "photo-realistic" is unnecessary.
- Line 48: application area -> application areas
...
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new method for single-image view synthesis. The authors point out that existing methods are faced with a "seesaw" problem, i.e., a trade-off between preserving observed contents and synthesizing out-of-view regions. To address this issue, the authors propose a method that shares the advantages of both explicit and implicit geometric transformations, where the former is good at preserving existing contents while the latter is good at synthesizing out-of-view contents. A transformation similarity loss is proposed, which motivates the explicitly rendered feature and implicitly rendered feature to learn the best from each other. The pipeline is a feed-forward model without an autoregressive module, thus significantly faster than other autoregressive models. Experiments on two datasets show that the proposed method achieves obvious improvements over other implicit or explicit baselines both qualitatively and quantitatively. Ablation studies are carefully conducted to verify the effectiveness of the proposed transformation similarity loss and the dependency between explicit and implicit renderers.
strengths_and_weaknesses: Strengths:

1. This paper points out the trade-off between explicit and implicit geometric transformation-based methods, and proposes a hybrid model that integrates the best of both, which is new and reasonable. The synergy effects of explicit and implicit renderers are well demonstrated in experiments.  

2. The proposed transformation similarity loss is novel to me. Its effectiveness is clearly verified in Tab.4 and Fig.5, which shows that it helps the explicitly rendered feature and implicitly rendered feature to benefit from each other.  

3. The proposed method shows clear improvements over other baseline approaches in quantitative and qualitative evaluations. Results in Fig.4 show that it better preserves existing contents and synthesizes out-of-view contents.  


Weaknesses:

1. I feel that the "seesaw" problem is somewhat exaggerated. In my humble opinion, a simpler and more straightforward way to solve this problem is to reproject or warp existing pixels to the target view based on the depth, and then use a generative model to synthesize the out-of-view pixels conditioned on the warped pixels, like inpainting. In this way, it naturally preserve existing contents and synthesize out-of-view contents. InfiniteNature [23] is designed in this way but is not compared. If this design can address the "seesaw" problem, then the contribution of this work is not strong. Otherwise, the authors should add some discussion to explain why.   

2. Following the previous point, the authors mentioned in the supplementary that InfiniteNature [23] is not compared because the training code is not available. However, to my knowledge, the testing code is already available online and comparison is possible. For example, even GeoFree [40] compares with InfiniteNature. I suggest adding InfiniteNature (or a warp+inpainting method) as a baseline if there is no strong reason why this cannot be done.  

3. If explicitly rendered feature and implicitly rendered feature are complementary, then why not just manually fuse the two features according to the out-of-view mask or send the mask as an additional input to the final decoder so that the decoder can learn to fuse them?  In this case, the two features do not have to mimic each other. Thus, I doubt if the transformation similarity loss is necessary. I suggest adding this experiment as an ablation study.

Some typos and grammar issues:  
Line 53: a novel loss function that explicit features improve ... -> a novel loss function that motivates explicit features to improve ...  
Line 116: should $I_{ref} \in \mathcal{R}^{H\times W \times C}$ be $I_{ref} \in \mathcal{R}^{H\times W \times 3}$?  
Line 135: resulting in a space complexity is reduced from ... -> resulting in a space complexity reduction from ...  
Line 207: Therefore, We -> Therefore, we  


Considering all the strengths and weaknesses, I rate borderline reject now. I may change my rating if my concerns in weaknesses are addressed.

=============

I have read the authors' feedback. My major concerns on InfiniteNature and feature fusion are well addressed. Thus, I increase my rating to weak accept.

questions: 1. I suggest adding InfiniteNature (or a warp+inpainting method) as a baseline if possible.

2. I suggest manually fusing the two features according to the out-of-view mask or sending the mask as an additional input to the final decoder so that the decoder can learn to fuse them.

3. Authors may consider adding some video results, which would be more intuitive for view synthesis tasks.

4. Authors may comment on any weakness listed above if they disagree.
limitations: It seems that for reprojected regions, some mismatch to the input image can still be observed in the generated results. Thus, the "preserving reprojected contents" objective is still not fully fulfilled and has room for improvement.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a non-autoregressive architecture for single-image view synthesis that combines both explicit and implicit methods. Two parallel renders extract explicit feature maps and implicit feature maps correspondingly, and the proposed transformation similarity loss encourages the consistency between two feature maps in the reprojected region and the out-of-view region. 
strengths_and_weaknesses: Originality: The idea of combining the explicit and implicit methods for single-image view synthesis is interesting. Also, the non-autoregressive framework reduces the inference time significantly. Some existing methods like GeoFree[40] also compare both explicit and implicit geometric transformation, but they didn’t combine the advantages of explicit and implicit methods, so the paper is sufficiently novel.

Quality: Overall the proposed method is well-described.
- As mentioned in the paper, the PSNR is a good metric for evaluating the preservation of reprojected pixels on small changes only. Is it possible to evaluate it for reprojected regions only? In table 2, the Synsin[50] has very competitive PSNR performance (even better for both medium and large split); in Figure 4, the Synsin also shows a better ability to preserve projected contents (e.g. the carpet near the door in 1st row and the droplight in 2nd row), so the explanation in Line 230-232 is not very convincing to me.
- Maybe it is worth moving the ablation study on the Type of Set Attention in supp. to the main paper since the design of the encoder didn't be discussed in the experiments section, though it seems a novel contribution.

Clarity:  The paper is well-written and the motivation is clear. Some parts of the experiments can be explained better:
- The definition of the explicit methods is not clear to me. For example, the Tatarchenko [46] is defined as an explicit method, but the way they parametrize the pose is very similar to the proposed Implicit renderer (angles only, no explicit warping introduced to the features to generate the image).
- It’s good to evaluate methods on different sizes of viewpoint changes. Is it possible to also show the qualitative results respectively?  Currently, Figure 4 is not clear since it only has the input source image and doesn’t show the target image, so it’s difficult to know if they are examples of small or large viewpoint changes.

Significance: The idea of rendering two feature maps in parallel can be general and it is possible to extend it to other frameworks.


questions: - In Figure 4, there are check-board artifacts for your method (e.g. the 2nd row), while the autoregressive methods don’t have that. Do you think it can be caused by the upsampling in the decoder? 

limitations: The limitations are discussed in Sec 5.
review_title: null

Meta-review: metareview: The reviewers found the approach of combining implicit and explicit methods for single-view synthesis both interesting and novel. All four reviewers recommended accepting the paper. You addressed many of the remaining questions/concerns in your rebuttal and in the author-reviewer discussion. Please, go through the reviews once more and make sure you have included the changes in the final version of the paper.

Update: given the extent of revisions required, we agreed that this would require another full evaluation of the paper due to the substantial changes that would need to be made to this paper. We hope that the rebuttal would help to guide subsequent submissions of this manuscript
confidence: Certain
Decision: Reject