Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposed CoCo-INR, an implciit neural representation trained with sparse multi-view images leveraging on prior information from pre-trained image prototypes/features.

Specifically, two attention modules called codebook attention and coordinate attention are proposed to introduce most revelant scene priors into the learning process of neural implicit representations, so that few images are adequate to render realistic images as well as high-quality geometric information. Three datasets of different complixities are used to demonstrate the effectiveness of method.
strengths_and_weaknesses: Strengths

1. The idea of integrating VQ codebook into NeRF training looks interesting and leads to promising geometry recontruction given few images on three public datasets.

2. The experimental set-up of ablations and vislusations validate the effectiveness of codebook attention.

3. Though there are some missing details (mentioned in the weakness part), the overall writing is good and easy to read.

Weaknesses


1. How to select a subset of training views are not clearly represented and only the number of views are listed. Are subset views evenly sampled are consecutively sampled while leaving the rest as testing views? 

    If possible, it would be good to see the camera trajectory or distributions along with rendering results to get better understand the method.

2. Similar to previous points, as cross-attention mechanism are adopted to propagate information from observed to unobserved ones, it would be interesting to see the extrapolated view synthesis in addition to interpolation.

3. One major point missing in my opinion is the comparison to other baselines introducing learnable/pre-trained priors into training process, such as PixelNeRF (CVPR2021) and GRF (ICCV2021).

    How does the codebook prior using VQ compared to stratigies adopted in PixelNeRF or GRF, where CNN feautes are used to augment local priors so that few views could be used to enable view synthesis. I think it is necessary to add related comparisons and discussions since currently all selected baselines does not take any external priors.

4. Another interesting thing is to explore is how the richness of priors affect performance of CoCo-INR? For example,the number of embeddings M and the size or Codebook, for a specific given scene, presumably a limited number of prototypes would be activated while others are not. Will a meaningful subset reduce the overall computation without obvious performance degradation? 

5. As Coco-VolSDF and CoCo-NeRF are used to model foreground and background respectively, following NeRF++. It would good to see how the decomposition looks like because it is not clear to readers how the CoCo-NeRF hanles background regions. In Figure 2 and 3, there are some noisy patterns in the first two images while the rest shows a clean backgroud geometry.Is the geometry (normal) only computed from foreground regions? Is the grey region computed from background NeRF? How does the attention mechism helps the background region?


6. Minor Issues.
In Sec3.2 of supplement, is it claimed that only 3 extreme views are used in Fig.2 while the captions says 8 images are used. Please clarify how many images are used?


questions: 1. One important missing experiments are about other strategies in introducing scene priors like PixelNeRF and GRF. Existing experiments only consider baseline methods without external piros so the comparison is not that fair in my point of view.


2. Further demonstrate on how CoCO-VolSDF and CoCo-NeRF separately represent the full scene and individual improvements among baseline methods are expected.

3. More ablation on hyper-parameters are welcome.

4. Please consider adding more details about the training set-ups, e.g., how the subset views are selected. Furthermore, depending on how the training subset are selected, it would be good to show if it is possible to infer images given extrapolated view points.

limitations: 
Some limitations are mentioned in the supplement material. Related to that, does the method scale to larger or unbounded scenes well? Is the cluterred background of these scenes hamper the performance of CoCo-INR? It woukd be good to add some analysis here.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper presents a method to utilize a non-scene-specific ImageNet-pretrained codebook for learning neural 3D representations, such as NeRFs. The codebook attention module transforms the codebook into a scene descriptor, and the coordinate attention module concatenates it with the positional encoding/features. The intuition is that doing so encourages the network to learn the semantic correlation between the input point (e.g. R^3) and the scene, enabling "extrapolation".
strengths_and_weaknesses: Strength: Demonstrates that the codebook method from VQGAN and VQ-VAE can be applied to 3D representation learning, lifting 2D ImageNet-based features to 3D. While the architecture may seem incremental, the extension to 3D is noteworthy.


Weakness: No quantitative ablation study. The "per-scene learnable features vs. fixed codebook" and "impact of codebook attention" experiments are both important yet entirely qualitative. Improvement from VolSDF could have been due to better initialization, more computation, and so on (as usual).  It would also be helpful to empirically evaluate novel views vs. observed views.
questions: How did you get the baseline MLP features (e.g. compared in Figure 4)?
limitations: The paper does not address limitations and societal impact.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper focus on improving the training efficiency of coordinate based representations by reducing the number of camera views needed during training. To accomplish this, the authors proposed a codebook attention module and a coordinate attention module to inject prior knowledge into implicit representations. Experiments on novel view synthesis are conducted and improvements are shown over baseline methods on some public datasets.
strengths_and_weaknesses: Strengths:

- The idea of bringing prior knowledge into implicit representation is novel and interesting.

- The paper is well written and easy to read.

Weaknesses:

- Although we do observe improvements over other baselines, it seems like the changes are relatively small. For example, in fig.2 the results of VolSDF and the presented methods are giving similar rendering results and in table.1 the quantitative results are tied.

- Lack of some ablation studies for a more thorough analysis of the proposed method. For example, the presented methods added two new modules to baseline implicit representations. And when doing comparisons with baseline methods, it is hard to tell whether it is because the addition of the new module brings the improvement or it is probably the network has more weights and could fits better. It could be better if more details are provided, for instance, the number of weights for the presented method and the baselines.
questions: - In Table 1. the quantitative results from the presented method and the VolSDF are tied on DTU dataset. However, it seems like in fig 2. the qualitative results are having a larger gap there. I am wondering what is the cause for this?

limitations: - Although prior knowledge are used in this work, it seems like the model is still trained specifically to one scene and is hard to generalize to novel scenes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes the utilization of a pre-trained codebook within an implicit neural 3D representation. This is enabled by using two attention modules termed codebook attention and coordinate attention, which together form the proposed Codebook Coordinate Attentional Implicit Neural Representation (CoCo-INR). The model is evaluated on the DTU, BlendedMVS, and H3DS datasets and compared with NeRF, UNISURF, and VolSDF using either "sparse views" (16-32 views) or "few views" (5-8 views). CoCo-INR shows better performance than the baseline methods in the conducted experiments.
strengths_and_weaknesses: **Strengths**

- The idea of combining a pre-trained codebook with (cross) attention modules is very interesting.
- The paper has high visual quality due to well-arranged figures and tables.
- For the presented experiments the model shows an improvement over the compared-against models (VolSDf, UNISURF, NeRF).

**Weaknesses**

- The comparison wrt. the SOTA is only done on "sparse views" and "few views". While these settings are sensible for evaluation of the proposed approach, it is crucial to also evaluate with pre-existing evaluation protocols. Currently, the reader cannot tell if the improved performance originates from an actually improved model, or because the model and benchmark were designed simultaneously.
-  The codebook idea is very interesting, however, it is difficult to understand what this module adds to the network and where the improved performance originates. Please also see the questions below.
- The ablation study is only qualitative and could contain further ablations, please see the questions below.
- Understanding the proposed architecture could be easier. I found Figure 1. in the supplementary material very helpful because it shows the actual computation the model is doing. In contrast, I found Algorithms 1 & 2 to be less helpful because they mostly reproduce the attention algorithm.

Overall, the paper proposes an interesting idea that seems to improve results. If the evaluation wrt. the SOTA is improved, the ablation study is expanded by more ablations and quantitative results, and the clarity is improved I am willing to upgrade my rating.
questions: - What is the intuition behind using the VQGAN pre-trained codebook? These codes are extracted from 2D images, but the paper tackles 3D reconstruction. Would it be better to use a pre-trained codebook from a 3D network?
- There are two coordinate attention modules in the "Neural Renderer" part of the pipeline, does this mean that the codebook is more important for generating the color?
- It would be interesting to include further ablations, for example:
  - reducing the number of MLP layers (maybe even to 0) to see the effect of only using the introduced codebook/coordinate attention modules
  - reduce the number of utilized codes from 16384 to maybe 8192, 4096, and 1024 by random sampling to investigate the effect of the codebook size
  - using a randomly initialized codebook to understand the effect of the VQGAN pre-trained codebook
  - exclude the "skip connection" in the "Implicit Neural Representation" (Suppl., Fig. 1, upper part) that concatenates the output of the coordinate attention module ($\widehat{X} \in R^{N \times 39}$) with the query coordinates after embedding ($X \in R^{N \times 39}$). This would also help to show how only the codebook-based module works.
  - Use more coordinate attention modules in the "Implicit Neural Representation" and fewer in the "Neural Renderer"
  
limitations: The limitations are addressed in the supplementary material.
review_title: null

Meta-review: metareview: This paper focuses on improving the training efficiency of coordinate based representations by reducing the number of camera views needed during training. To accomplish this, the authors proposed a codebook attention module and a coordinate attention module to inject prior knowledge into implicit representations. The intuition is that doing so encourages the network to learn the semantic correlation between the input point and the scene, enabling "extrapolation" to far away views.

The reviewers appreciated the idea of the paper and how it improved image reconstruction quality across various number of views.  They raised concerns regarding lack of experiments with models that perform conditioning to pixel features, e.g., PixelNerf, to generalize across scenes, as well as lack of ablative quantitative experiments to evaluate the architectural contributions and lack of a limitations section. The rebuttal submitted by the authors includes experiments with PixelNerf but does not state mention the number of views used, and does not describe how the method “ours (with scan 51 priors)” is obtained.  The second experiment where PixelNerf is trained on DTU  and tested on BlendedMVS is not a fair experiment and we do expect pixelNerf to fail there given there is no test time adaptation through gradient descent at the test scene as is the case with the present method. The authors are encouraged to move all rebuttal experiments to the main paper, and thoroughly explain the experimental setup they used. 
Overall, the paper is not very clearly written. Specifically, the reader  learns only at the end of the implementation detail section that a separate network is trained per scene.  Reviewer q1CY mentions: “Although prior knowledge are used in this work, *it seems like* the model is still trained specifically to one scene and is hard to generalize to novel scenes.”
By not comparing or contrasting the proposed approach to cross-scene generalization works the reader is left to wonder what is the generalization capabilities of the proposed model with varying number of input views.
The authors are encouraged to clarify these points in their final version. 

confidence: Less certain
Decision: Accept