Coordinates Are NOT Lonely - Codebook Prior Helps Implicit Neural 3D representations
Authors: Fukun Yin
Wen Liu
Zilong Huang
Pei Cheng
Tao Chen
Gang YU
Venue: NeurIPS 2022 Accept (2022)
Keywords: Coordinate Learning
Implicit Neural Representation
Inverse Rendering
Neural Radiance Field

Abstract: Implicit neural 3D representation has achieved impressive results in surface or scene reconstruction and novel view synthesis, which typically uses the coordinate-based multi-layer perceptrons (MLPs) to learn a continuous scene representation. However, existing approaches, such as Neural Radiance Field (NeRF) and its variants, usually require dense input views (i.e. 50-150) to obtain decent results. To relive the over-dependence on massive calibrated images and enrich the coordinate-based feature representation, we explore injecting the prior information into the coordinate-based network and introduce a novel coordinate-based model, CoCo-INR, for implicit neural 3D representation. The cores of our method are two attention modules: codebook attention and coordinate attention. The former extracts the useful prototypes containing rich geometry and appearance information from the prior codebook, and the latter propagates such prior information into each coordinate and enriches its feature representation for a scene or object surface. With the help of the prior information, our method can render 3D views with more photo-realistic appearance and geometries than the current methods using fewer calibrated images available. Experiments on various scene reconstruction datasets, including DTU and BlendedMVS, and the full 3D head reconstruction dataset, H3DS, demonstrate the robustness under fewer input views and fine detail-preserving capability of our proposed method.

Coordinates Are NOT Lonely Codebook Prior Helps Implicit Neural 3D Representations
Fukun Yin
Wen Liu 2+
Zilong Huang²
fkyin210m.. fudan. fudan edu. liuwen. steven. 19940gmail.....m comamaail.oommmmm com
zilong. hwwng20208gmail.commmmmm com
Pei Cheng
Tao Chen
Gang YU2 iskicy6gmail com
peicheng@tencent.. com
eetchenffudan edu, cn
'ccoool of Information Science and Technology, Fudan University, China
2 Tencent PCG, China
Abstract
Implicit neural 3D representation has achieved impressive results in surface scene reconstruction and novel view synthesis, which typically uses the coordinate- based multi-layer perceptrons (MLPs) to learn continuous scene representation. However, existing approaches, such Neural Radiance Field (NeRF) [5], and its variants [16 26 29), usually require dense input views (i.e. 50-150) to obtain decent results. To relive the over-dependence on massive calibrated images and enrich the coordinate-based feature representation,
explore injecting the prior information network and introduce a novel coordinate-ban CoCo-INE for implicit neural 3D representation. The cores of our method are two attention modules: codebook attention and coordinate attention. The former extracts the useful prototypes containing rich geometry and appearance information from the prior codebook, and the latter propagates such prior information into each coordinate and enriches its feature representation for With the help of the prior information, our method can render 3D views with more photo-realistic appearance and geometries than the current methods using fewer calibrated images available. Experiments on various scene reconstruction datasets, including DTU and BlendedMVS (28), and the full 3D head reconstruction dataset, H3DS [9] demonstrate the robustness under fewer input views and fine detail-preserving capability of our proposed method.
into the coordinate-base
model,
object surface.
Introduction
3D reconstruction aims recover the underlying scene geometry and appearance from of posed images. It is fundamental problem in computer vision and graphics with wide usage in 3D object surface reconstruction and scene novel view synthesis. Recently, implicit neural representation (INR) with coordinated-basee learning has been driving significant advantages in producing high- fidelity 3D shapes
[6, 26 29 and photo-realistic images [I5] 2 With the per-scene optimized
multi-layer perceptrons (MLPs), such methods usually require a dense set of calibrated views to
'Joint first authors.
'Corresponding author. This work was done when Fukun Yin was an intern at Tencent PCG.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Implicit Neural Representation
Codebook Attention Module
Leomable
Duery
Embeddings
Layers
Coordinate Attention Module
SDT se girt
Goondrate
Quary Coordinates Enocddrr
ga (2.0)
Global Geometry 1
Codebonk Artention Module
Neural Renderer
Cadebnox
Self
Qvery Embedsing:
learnable
MUP Layer
Coordinate Attention Module
Eribodungg Coondivatss & ge123
rate 9o20s
gions
Corver
Cross
Color Feature ce RM
Global Geometry Secture
Figure 1: The architecture of the previous and our proposed coordinates network. Our cores two attention modules: codebook attention and coordinate attention. The former extracts the practical prototypes from the initial codebook, and the latter injects the prototypes into each coordinate.
optimize high-quality continuous 3D scene representation. However, their performance would degrade dramatically when the number of input views is very limited, because only spatial coordinate information is insufficient for the network to characterize continuous 3D scene with few input views. It is practical to overcome this drawback by introducing additional high-level semantic descriptors of the context scene as the prior information, to enrich the feature representation of each coordinate and compensate for the missing views when only sparse views are provided.
Fortunately, learning the prior descriptors prototypes from a large scale dataset is long-studied topic from the early Bag-of- Words (BoW) in image classification, scene recognition [4] and visual SLAM [T], Vector Quantized (VQ) codebook in data compression [18. the recent neural discrete codebook representation in image synthesis [23 20 3]. These methods learn and maintain repre- sentative codebook, with each codeword denoting a specific prototype from the dataset for serving as
prior to encode new images or scenes, However, how to leverage and correlate the codebook prior with the implicit neural representation is still an unreached area.
This paper investigates a novel coordinate-based deep model that injects the prior codebook informa- tion into the implicit neural 3D representation. To achieve this, we first design a codebook attention module to extract the valuable prototypes from the prior codebook for each scene. These prototypes might contain rich geometry or appearance information, with parts of them being shared and others being exclusive for different views. The shared prototypes in the codebook guarantee the network to generate con-consi results. In contrast, the exclusive prototypes for each view bring additional information for the network to enrich its feature representation and produce photo-realiste rendering results and decent geometry by leveraging fewer posed images. Then, we equip each coordinate with the extracted prototypes to improve its feature representation via coordinate attention module. They are both the attention- based modules, as depicted in Fig. [ The former queries the per-scene relevant prototypes from the codebook prior via the learnable embeddings, and the latter enables each coordinate to query representative features from the prototypes. With the help of prototypes, each coordinate is not alone, resulting in more robust representation and improving the quality of rendering images.
We perform extensive experiments on different scene reconstruction benchmarks, including DTU [9] BlendedMVS (28) and the full head reconstruction benchmark H3DS [19]. Our approach outperforms current state-of-the- th-the-art methods [5
16 29 under a sparse set of input views. We summarize the contributions as follows: 1) we propose a novel coordinate- based network that incorporates codebook prior for implicit neural representations, which builds a connection between the coordinates and the codebook prior information for the first time; 2) with the help of the codebook prior, our method could improve the feature representation of each coordinate and is more robust in producing realistic- looking novel views with fewer images; 3) extensive experiments demonstrate the effectiveness of our CoCo-INR, including the robustness in generalizing to novel views under few inputs images and

---

the capability to produce more photo results under sparse input views. The code is available uttttps:///ithuu cww com//tuuunnin//oooooeeaaaaa
2 Related Work and Background
Codebook Representation. Bag-of- Bfgrof-Words (BoW) is popular codebook representation method, which has been widely used in document recognition [[3] and image classification [22]. BoW model usually contains the following steps: a) extracting the local features; b) learning the language or visual vocabulary; c) quantizing features by the vocabulary; d) representing a document or an image by the frequencies of words. Recently, Vector Quantization (VQ) has driven advantages in feature representation learning, ranging from the image tokenizer in transformer architecture [25] to the latent codes in generative models [23] 20 3. VQ-VAE 23 proposes a
framework to train an image-level VQ model from a large-scale dataset. VQGAN 3 further improves the codebook representations by using more photo-realistic losses, such as the perceptual loss [00 and the generative adversarial discriminator [5]. Though the VQ codebook provides a powerful tool for generative models, whether it can be explored for implicit neural representation is still unknown.
Neural Scene Representation and Rendering. Implicit neural 3D function is leading the develop-
ment of the shape and appearance reconstruction of a scene [15 16 26 29]. It models the scene as two functions via two neural networks: a geometry function f R + R maps the spatial location E - to its shape value, and a color function _ IR2 + R maps coordinate x E R and view direction d E IK² to its color value. According to the shape representation differences, we can roughly categorize them into neural surface fields [6, 26 [29 and neural radiance fields [5] 14 2
The former define the object surface as the zero-level set of its signed distance function (SDF) 26 or occupancy [16), shown as follows:
5= E (=)()) 03.
(1)
While the latter uses generic volume density a, and the object surface could be any level set of its density function without constraints, leading to noisy shape reconstruction 29
Both of these representations enable to render an image via the volume rendering. For each ray + fd, we sample N points {x;} along the ray with (t,). Here, E IR3 is the location of the camera and IR3 is the viewing direction. The color Ö of each ray can be approximated follows:
N
N
Σ Tia;; Σ
(1
j=1
where T; is the accumulated transmittance along the ray, defined T, = 111 oz).
The way to calculate the ag varies among different methods. In NeRF [13), a; = exp c (-) o, is the density value at Iy, and 8; = f - t.. In UNISURF [16), a, is the predicted occupancy probability. In NeuS (26) and VoISDF [2 a o-function is utilized to map the SDF to the volume density probability to calculate the og, The former chooses a logistic density distribution, and the latter uses cumulative distribution function of the Laplace distribution the o-function, respectively. These methods can produce photo-realistic images and precise shapes with a dense set of calibrated views. However, their performance will degrade quickly when only sparse or few views are available.
3 Methodology
Our goal is to design a new coordinate-basee network for implicit neural 3D representation, which utilizes the codebook prior information to improve the feature representation of each coordinate. To demonstrate the effectiveness of our proposed Codebook Coordinate Attentional Implicit Neural Representation (CoCo-INR), we apply it to the task of multi-view scene reconstruction and novel view synthesis. Since our method relies on a representative codebook prior, we employ the same setup of VQGAN model 3 and use powerful pre-trained model on the ImageNet dataset [2I], which contains a codebook & E IR F with E prototypes, E R To make a fair comparison with the previous methods for multi-view scene reconstruction and novel view synthesis, we follow the same pipeline with VolSDF (29] including the geometry representation, sampling strategy, and all training losses, but just replace its MI.P-based network with our proposed Codebook Coordinate Attentional Implicit Neural Representation (CoCo-INR).
3

---

3.1 Codebook and Coordinate Attention
We now present the details of our proposed CoCo-INR, As shown in Fig. it contains two attentional modules: codebook attention and coordinate attention. The former queries the per-scene relevant prototypes from the codebook prior via the learnable query embeddings, while the latter injects the prototypes' information into each coordinate and enriches its feature representation.
Codebook Attention. Since a global pre- traitrd codebook might contain irrelevant prototypes for a specific scene, the codebook attention is going to extract the representative prototypes from the codebook for this scene. Given a pre-trained codebook & E ERD where E is number of prototype vectors, and D is the dimension of each prototype vector ei, we define M as the learnable query embedding vectors {q1. 92....,
and each embedding vector 9: E I queries the scene relevrtlevant prototype (0) E IR$ from the global codebook & via a cross-attention mechanism. We will arrive (0)) ). We then apply self-attention modules on the initial peneeprototypees g to improve their feature representations further and =M 22,....
..., qM
at the initial scene-relevant prototypes, g {=] -20)
J. which is inspired by the previous Perceiver 8 transformer architecture. We describe the procedure our codebook attention module in Algorithm. Coordinate Attention. Through the aforementioned codebook attention module,
obtain the final scene-relevant prototypes 2 = {21. 2 -2,-...2dd
Algorithm 1 Codebook Attention Module.
Require: a global pre-trained codebook, & = (&+ E RR E 1,2, EJ, and the learnable query embedding vectors, Q = {q E R 1,2, M).
Ensure: Z = {2
E R 1,2, M), the output prototype vectors;
1: Q <- (Jq(g)) E B 1,2,
1,2, ..., MM # Sq is the query linear projection;
2: K + (fx(e:) E
...., E5} # fx is the key linear projection;
3: V < -[f(ee) E E E} # fvv is the value linear projection;
4: < Pross-Attention(2. V) Softmax QK" V # the selected prototypes from the global Jag
codebook :;
5: 2 + (2) (0)) 1, M] fo is the output linear projection;
6: for to L perform L Self-Attention layers improve prototypes' representation;
7: # #5 (1) ((() and f: are the query, key, and value linear projection, respectively. (1 8: 2(1) t- Self-Attentio (f()) (()ioon((((((())))))) (2(-1)), (2(--))) s( (( (2(1-1)))
(2)-1)),
9: end for
return 2 - - (1) E M) final scene-relevant prototypes.
will arrive at series of representative scene-relevant prototypes, Z (2(2), 12 (1)
The coordinate attention module going to gradually inject the representative prototype information into each point via T cross-attention modules, and consequently results in more representative feature vector for each coordinate. The coordinate attention module the bridge that connects the coordinate and the codebook prior whose details are shown in Algorithm.
3.2 Codebook and Coordinate Attention for Implicit Neural 3D Representations
We now present the details of proposed CoCo-INR for multi-view scene reconstruction and novel view synthesis. We follow the NeRF++ 30 framework, which separately models the foreground and background neural representations. For the foreground, follow the neural surface field with VoISDF [29] and replace its MI Plsss network with our proposed codebook and coordinate attentional modules, and arrive
VoISDD. Similarly, we neural radiance field to model the background scene and arrive at our CoCo-NeRF.
CoCo- VolSDF contains two CoCo-based networks: geometry network, fa(x) (s, predicts the SDF of the object E RR and global geometry feature 2 E 2256, 2) an appearance network, Cg (ar, n, d, =) E RR, conditions on spatial location, surface normal, viewing direction, as well as global geometry codes, and then predicts the radiance (color). CoCo-NeRF has a similar architecture, except that the geometry network predicts the density value, and the appearance network only conditions the global geometry feature and the viewing direction as the inputs.

---

Algorithm 2 Coordinate Attention Module.
Require: a series of scene-relevant prototypes. g(4) - (zz E li E 1,2, MJ and a set of
query coordinates (points), X' {a; E RC 1, 2, .... N}:
Ensure: X = { Ri 1,2,,... ..., J.. the output coordinate features;
1: x110) + {I; E R .I,,...,NN N}
2: # perform 7 Cross-Attention layers to propagate the prototypes information into each coordinate.
3: for t = 1 to T do
4: Q - (x)) Rd
i 1,2, ...,....... NN # (()) is the query linear projection:
5:
K + (f) ER)
1, M) f is the key linear projection layer;
V + (z) E RR 1,2 ..., M # f( the value linear projection layer;
6:
X10) + Cross Attentintt Q, V) Softmax OK* ) # bring prototypes for coordinates;
7:
Vil X11 (x.") (2) E C ..,, N fot is the output linear projection in attention; (T)
8:
9: end for
return X t {og E :ROii E 1.2, .... N} final output coordinate features.
3.3 Training Losses
To train our CoCo-INR, we minimize the difference between the rendered colors and the ground truth colors without 3D supervision. Specifically, we optimize our neural networks by randomly sampling batch of pixels and their corresponding rays in world space, and this results in a triplet set = [[C,, Og, d E 1, ...,, K}. where K is the total number sampled pixels/rays, Cş is its pixel color, og and dę are the camera location and the ray direction from the camera origin to the pixel points in the world space. The loss function is defined as
L Legh + ALask-
(3)
Here, Lygh is color rendering loss, which enforces the rendered pixel color O₂ computed by Equa- tion. 2 to be similar to the ground truth pixel color Ck, formulated follows:
1 K
Legb = Elc - Cxll1-
K
R=1
Since the gradient of SDF satisfies the Eikonal equation [ = apply this Eikonal term on the sampled points (N in total) to regularize the geometry network f by
N
Lesk Σ (oVVll 1)a.
(5)
3.4 Implementation Details
We use the pre-trained VQGAN model on the ImageNet dataset (11 with a codebook & I 016384x256. In each CoCo block, the number of learnable query embeddings is 256, and each embedding has a dimension e; E 128 We apply one cross-attention block the codebook attention module, followed by three self-attention blocks. In the coordinate attention module, we perform one cross-attention operation in the geometry network, and perform once or two times cross-aad attentiono operations in the appearance network. All attention modules are the transformer (5 style with multi- head attention mechanism (with four heads), Layer Normalization (Pre-Norm) 1. Feed-Forwarde Network (25) and GELU activation 66. The dimension of the key/value and Forwaro layer is 128 and 256, respectively. The subsequent fully connected hidden layers use Softplus activation with 8 100 in the geometry network, and apply the ReLU activation between hidden layers and Sigmoid for the output in the appearance network.
To model the high-frequency details, we use positional encoding 15 on spatial location as maximum frequency of 6 and that of view direction as 4. We follow the same hierarchical sampling strategies with the етrоr bound and geometric initialization Stochastic Gradient Descent method is used for parameter optimization with a fixed learning rate of 0.0005. The number of sampled rays/pixels is 1024 and the À of Leak is 0.1. Our method is built with Pytorch [17 framework. Each scene is trained on a single Nvidia V100 GPU device for around 3-14 hours, depending on the number of input views.
VolSDF [29]. Adam [12] based
5

---

4 Experiments
4.1 Experimental Setup
Datasets. We conduct the experiments on three publicly available and challenging multi-view scene reconstruction datasets: DTUU [9. BlendedMVS (28) and H3DS [19]. The DTU and BlendedMVS are the datasets that contain real objects with different materials, appearances, and geometry. Each scene on the DTU dataset has 49 64 posed images with the resolution of 1600 1200, and there are 31 to 144 calibrated images with the resolution of 768 x 576 in the BlendedMVS dataset. Following the previous works 29 16]. we perform experiments on 15 challenging scenes of the DTU dataset and 9 scenes of the BlendedMVS dataset. The H3DS dataset contains 23 scenes of high-resolution full-head 3D textured scans with diverse hairstyles and challenging lighting environments. There are 64 calibrated images in 360 views.
Baselines and Evaluation Metries. We choose the recent saae-of-the--. art implicit neural 3D rep- resentation methods, including NeRF [13], UNISURF [16], and VoISDF [29] as the baselines. To evaluate the performance of each method in scene representation, we use three common metries: Peak Signal to Noise Ratio (PSNR), Structural Similarity (SSIM) [27]. and LPIPS [31] on novel view synthesis. Higher PSNR and SSIM mean a better performance, while a lower LPIPS means better.
Settings with Different Numbers of Views. Since the previous methods like UNISUIRF [16] and VolSDF [29] use all images of each scene train the model and report the rendering results on the training views, this is unreasonable to measure the generalization performance in the novel views. To better evaluate the performance of each method under sparse set of input images, in all datasets, we only use half (or less than half) of the provided images (16-32 in total) of each scene as the training set and set the left views as the testing set. Same with UNISUIRF [66] and VoISDF (29), do not use the object mask inputs and the mask loss. We further conduct the experiments with fewer training views (5-8 in total) to compare the robustness of each method.
4.2 Performance Comparison
To comprehensively evaluate the performance, we compare our method with baselines on DTU, BlenedMVS, and H3DS datasets under different numbers of training views.
Setting with Sparse Views (16-32 in total). The quantitative results under this setting are reported in Table I Our CoCo-based method outperforms others in terms of PSNR, SSIM, and LPIPS when sparse views are available. Also, we further illustrate the qualitative comparison with other methods in Fig which demonstrates that our method produces a more realistic-oooking novel view synthesis and more precise normal results concerning geometry.
It can be seen from Fig 2 that our method shows robustness and detail recovery ability in the case of sparse views. NeRF's [15] estimated volume density could not generate accurate surface recon- struction and novel views with large-scale changes. UNISURF [16] relies on multiple views to learn details and distinguish foreground and background and is sensitive to noise (The H3DS dataset [19 has challenging lighting environments due to the use of flash). Both ours and VoISDF [2] have shown generalization for a novel view. However, DTU failed in the scene where the foreground and background were indistinguishable (the second column in Fig. 2). because each point is independent and cannot obtain semantic information. Our CoCo-based method queries the per-scene relevant prototypes from the codebook prior via the learnable query embeddings, to deal with scenes with similar colors and features. More importantly, propagating prior information into each coordinate will enrich its feature representation for a scene or object surface, and improve the network's 's ability to model the fregefrryeency information. Consequently, our method can preserve the details better, such as the holes (the first column), reflections (the fourth column), hair, and expressions (the last two columns) in Fig. 2
Setting with Few Views (5-8 in total). To further demonstrate the generalization and the robustness of our method under a few input views, we conduct the experiments and make comparisons with VolSDF [29]. which has demonstrated generalization in sparse views. As Table. shoos in this setting, our method has significantly improved over the baseline method, VoISDF, with a higher PSNR, SSIM, and LPIPS, respectively. Also, it is apparent from Fig. that our method is more robust to produce better rhoto-realistic novel views with more detailed geometry normal than the baseline method, which demonstrates that the codebook prior information could indeed be a compensation for

---

BlenedMiVS Scan B
BlenedMVS
H3DS
Scan 43
Scan 118
Scan 9
3410gaf/sdeb5127 saCaff60ecdeccee
Ground Truth
Render
NeRF
Rondar
UNISURF
Normal
VolSDF
Normal
Ours
Normal
Figure 2: Qualitative results (zoom-in to view better) each method on the DTU, BlendedMVS, and H3DS datasets under sparse views (16-32 in total). We both visualize the novel view synthesis and the surface normal each method.
Table The results of different methods the DTU, BlendedMVS. and H3DS datasets with sparse views (16-32) and without object masks of each scene. 1 means the higher, the better.
DTU
BlendedMVS
H3DS
PSNR
SSIM
LPIPS
PSNR1
SSIM
LPIPS
PSNR
SSIM
LPIPS]
NeRF 15
22,162
0.790
0.306
16.523
0.667
0.272
21.185
0.861
0.144
UNISURF [6
23.549
0.823
0.348
14.749
0.649
0.299
17.095
0.816
0.190
VoISDF 29
26.609
0.839
0,309
18.942
0.747
0.213
23.922
0.898
0.110
Ours
26.738
0.852
0.298
19,594
0.764
0.201
25.279
0.911
0.098
the limited input views. To investigate the potential reason why CoCo-base based coordinate network could render better realistic-looking views under fewer input views than the MLP-based network, we compare the feature distribution of sampling points.
We take two patches from different areas of the foreground and visualize their geometry features and appearance features by the t-SNE (24) method, Fig. 4 We can see that our proposed CoCo- based network results in more discriminative features for each coordinate, which indicates that the codebook prior could enrich each coordinate's 's feature representation. On the contrary. MLs-dased networks, such as VolSDF, cannot distinguish the features different points in both geometric and appearance spaces. In other words, MLP-based networks cannot extract necessary features for surface reconstruction and new views generation, leading to errors in geometry normal or blurring and color distortion.
Setting with Extremely Limited Views (3 views). In this part, we attempt a challenge: only use three training views to train our CoCo-based network. In this challenge, three training views mean

---

DTU Scan 122
BlenedMiVS Scan 1
BlenedMMSS
H305
H3D5
Scan 63
Scan 5
142a6613401e42d6 5oDe871Badb80226
Ground Truth
Render
VolSDF
Normal
Render
Ours
Normal
Figure 3: Qualitative visualization results (zoom-in for the best of views) the DTU, BlendedMVS, and H3DS datasets with few views (5-8). We both show the novel view synthesis and the surface normal of the cfate-of-the-ar method, VoISDF 29 and our method.
I-SNE of Geometry Features the Sampled Points
Our CoCo-based Network
MIP-based Network
t-SNE Appearance Features the Sampled Points
Our CoCo-based Network
MIP-based Network
Figure 4: The visualization learned features of the sampled points, by our CoCo-based network and the MLP-based network under few input views. Both geometry features and sampled appearance features of all coordinates are illustrated.
that is difficult for the network to learn background information, so we use masks to make the network more focus on the foreground. We conduct experiments on the DTU dataset and compare with mask-based VolSDF [29]. As shown in Table 3 our method has significantly improved over VoISDF with higher mean PSNR, SSIM, and LPIPS respectively. Especially for the LPIPS, VoISDF nearly 25% higher than our method, which means that the new view images generated by our method perform better the semantic level. We visualize part of the experimental results in Fig. [ and it can be seen that our CoCo-INE can still render satisfactory RGB images and surface features under the extremely limited 3 training views.

---

Scan 40
Scan 55
Scan 63
Scan 65
Scan 69
Scan 105
Scan 106
Scan 114
Ground
Truth
Ours
Render
Ours
Normal
Figure 5: Qualitative visualization results (zoom-in for the best of views) on the DTU dataset with extremely limited views.
Novel
Per-scene Learnable Codebook
Pre-trained Codebook
Figure 6: Results of our method with different codebooks with views available on the H3DS dataset.
Ground Truth
Without Codebook Attention
With Codebook Attention
Figure 7: Results of our method with and without codebook attention module on the DTU dataset.
Table 2: The results between VoISDF
29
and ours, on the DTU, BlendedMVS, and H3DS datasets
with few views(5-8) and without object masks of each scene. means the higher, the better.
PSNRT
SSIM
LPIPS..
PSNRT
SSIM
LPIPS.]
PSNRT
SSIMI
LPIPS
VoISDF 29
18,692
0.747
0.362
13.918
0.624
0.317
18.939
0.846
0.166
19.624
0.757
0.346
14.417
0.298
19.084
0.847
0.165
Table 3: The results of different methods on the DTU dataset for each scene with 3 training views.
Method VoISDF (29
PSNR
SSIM
LPIPS..
13.929
0.522
0.491
Ours
14.761
0.624
0.396
Table 4: Per-scene learnable codebook fixed pre-trained codebook with few views (5-8). Experi- ments on the first three seans of the DTU dataset(scan24, scan37, and scan40). 1 means the higher, the better.
Method
PSNRT
SSIM
I.PPSS
Per-scene Learnale Codebook
13.315
0.525
0.298
Ours (Fixed Pre-trained Codebook)
15.622
0.576
0.283

---

4.3 Ablation Studies and Analysis
Per-scene Learnable Codebook vs. Fixed Pre-trained Codebook. To verify the efficacy of the codebook prior, we perform controlled experiments on both a per-scene learnable codebook and fixed pre-trained codebook from large-scale dataset, ImageNet [[]] by VQGAN model B.. The comparison result is shown in Tad Fig- 6 Our method with per-scene learnable codebook does not introduce additional information and would fail under a few input views, which further indicates the importance of representative codebook in our system.
Impact of Codebook Attention. To demonstrate its efficacy, we conduct an ablation study of our CoCo-NIR with and without the codebook attention module. The comparison result in Fig. 7 shows that the codebook attention improves the robustness of our system when the scene has many homogeneous textures, which reflects that the codebook attention could extract scene-relevant prototypes (such as shapes or textures) for the coordinates.
Table 5: Different number of codebook items with few views (5-8). Experiments the first three scans the DTU dataset(scan24, scan37, and scan40). 1 means the higher, the better.
Method
PSNR1
SSIM
LPIPS
8192 items
15.463
0.567
0.287
4096 items
14.042
0.550
0,288
2048 items
14.762
0.565
0.285
1024 items
13.948
0.548
0.300
Ours(16384 items)
15.622
0.576
0.283
Impact of Codebook Size. We report the results of different numbers of codebook items, ranging from 16384, 8 8192, 4096, 2048 to 1024 in T It shows that the performance tends to decrease with the reduction of the codebook items.
Conclusions and Limitations
We introduce CoCo-INR, novel framework for implicit neural 3D representations. It surpasses the previous ML.Ps-based implicit neural network by building connection between each coordinate and the prior information, propagates them into each coordinate, and consequently, more representative feature for each coordinate. With the help of prior information, our framework could produce photo- realistic novel view synthesis and 3D shapes with fewer posed images available in the task of scene reconstruction.
Our framework is flexible and generic. In future work, our framework has the potential ability to handle cross-modalities implicit neural representations such as text-to-imageSSDF/Neke connection between the language prompts and coordinate. Moreover, if we replace the per-scene learnable query embeddings in codebook attention modules with the image/pose paggeppseedeeenden feature tokens, our framework could also be extended to a generalizable network across scenes rather than the current per-scene optimization.
via the
Acknowledgements
This work is supported by National Natural Science Foundation of China (No. 62071127 and 62101137), Zhejiang Lab Project (No. 2021KHOAB05).
10

---

References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv: 2016.
arXiv preprint
[2] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855-5864, 2021.
[3] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for nigh-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12873-12883, 2021.
[4] Li Fei-Fei and Pietro Perona. A bayesian hierarchical model for learning natural scene categories. In 2005 IEEE Computer Society Conference on Compuder Vision and Puttern Recognition (CVPR'0S)) volume 2. pages 524-531. IEDE, 2005.
[5] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems 27, pages 3672-2680. Curran Associates, Inc., 2014.
[6] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), arXiv preprint arXiv:1606.08/4 2016.
[7] Andrew Jaegle. Sebastian Borgeaud, Jean- Baptistt Alayrac, Carl Doersch, Catalin Ionescu, David Ding. Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107. 140999507 2021.
[8] Andrew Jaegle. Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman. and Joao Carreira. Perceiver: General perception with iterative attention. In International Conference Machine Learning, pages 4651-4664, PMIR, 2021.
[9] Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs. Large scale multi-view stereopsis evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 406-413, 2014.
[10] Justin Johnson, Alexandre Alahi, Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pages 694-711. Springer, 2016.
[11] Ayoung Kim and Ryan Fustice. Combined visually and geometrically informative link hypothesis for pose-graph visual slam using ofg-of-words. In 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1647-1654. IEEE, 2011.
[12] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 14Xiv:1412669800 2014.
[13] Guy Lebanon, Yi Mao, and Joshua Dillon. The locally weighted bag of words framework for document representation. Journal of Machine Learning Research, 8(10), 2007.
[14] Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 7210-7219, 2021.
[15] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In European conference on computer vision, pages 405-421. Springer, 2020.
[16] Michael Oechsle, Songyou Peng, and Andreas Geiger. Unisurf: radiance fields for multi-view reconstruction. Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 5589-5599, 2021.
Unifying neural implicit surfaces and
[17] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[18] Sharon M Perlmutter and Robert M Gray. A low complexity multiresolution approach to image compression using pruned nested rre-structured vector quantization. In Proceedings of 1st International Conference on Image Processing, volume pages 588 592. IEEE, 1994.

---

[19] Eduard Ramon, Gil Triginer. Janna Escur. Albert Pumarola. Jaime Garcia, Xavier Giro-i Nieto, and Francesc Moreno-Noguer. Hid-net: Few-shot high-fidelity 3d head reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5620-5629, 2021.
[20] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelitg images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
[21] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, ct al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-25) 11222, 2015.
[22] Pierre Tirilly, Vincent Claveau, and Patrick Gros. Language modeling for bag-of- words image categorization. Proceedings of the 2008 international conference on Content-based image and video retrieval, pages 249-258, 2008.
[23] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[24] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11). (), 2008.
[25] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[26] Peng Wang, Lingje Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106. 10689, 2021.
[27] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions image processing. 13(4):600-612, 2004.
[28] Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang. and Long Quan. Blendedmvs: A large-scale dataset for generalized multi-view stereo networks. In Proceedings of the JEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1790-1799, 2020.
[29] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34, 2021.
[30] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint 2700:2200,. 2020.
[31] Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
12

---

(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "Yjz6V0xYRjL", "reviewer": "Reviewer_k1QM", "summary": "The paper proposes the utilization of a pre-trained codebook within an implicit neural 3D representation. This is enabled by using two attention modules termed codebook attention and coordinate attention, which together form the proposed Codebook Coordinate Attentional Implicit Neural Representation (CoCo-INR). The model is evaluated on the DTU, BlendedMVS, and H3DS datasets and compared with NeRF, UNISURF, and VolSDF using either \"sparse views\" (16-32 views) or \"few views\" (5-8 views). CoCo-INR shows better performance than the baseline methods in the conducted experiments.", "questions": "- What is the intuition behind using the VQGAN pre-trained codebook? These codes are extracted from 2D images, but the paper tackles 3D reconstruction. Would it be better to use a pre-trained codebook from a 3D network?\n- There are two coordinate attention modules in the \"Neural Renderer\" part of the pipeline, does this mean that the codebook is more important for generating the color?\n- It would be interesting to include further ablations, for example:\n  - reducing the number of MLP layers (maybe even to 0) to see the effect of only using the introduced codebook/coordinate attention modules\n  - reduce the number of utilized codes from 16384 to maybe 8192, 4096, and 1024 by random sampling to investigate the effect of the codebook size\n  - using a randomly initialized codebook to understand the effect of the VQGAN pre-trained codebook\n  - exclude the \"skip connection\" in the \"Implicit Neural Representation\" (Suppl., Fig. 1, upper part) that concatenates the output of the coordinate attention module ($\\widehat{X} \\in R^{N \\times 39}$) with the query coordinates after embedding ($X \\in R^{N \\times 39}$). This would also help to show how only the codebook-based module works.\n  - Use more coordinate attention modules in the \"Implicit Neural Representation\" and fewer in the \"Neural Renderer\"\n  ", "limitations": "The limitations are addressed in the supplementary material.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The idea of combining a pre-trained codebook with (cross) attention modules is very interesting.\n- The paper has high visual quality due to well-arranged figures and tables.\n- For the presented experiments the model shows an improvement over the compared-against models (VolSDf, UNISURF, NeRF).\n\n**Weaknesses**\n\n- The comparison wrt. the SOTA is only done on \"sparse views\" and \"few views\". While these settings are sensible for evaluation of the proposed approach, it is crucial to also evaluate with pre-existing evaluation protocols. Currently, the reader cannot tell if the improved performance originates from an actually improved model, or because the model and benchmark were designed simultaneously.\n-  The codebook idea is very interesting, however, it is difficult to understand what this module adds to the network and where the improved performance originates. Please also see the questions below.\n- The ablation study is only qualitative and could contain further ablations, please see the questions below.\n- Understanding the proposed architecture could be easier. I found Figure 1. in the supplementary material very helpful because it shows the actual computation the model is doing. In contrast, I found Algorithms 1 & 2 to be less helpful because they mostly reproduce the attention algorithm.\n\nOverall, the paper proposes an interesting idea that seems to improve results. If the evaluation wrt. the SOTA is improved, the ablation study is expanded by more ablations and quantitative results, and the clarity is improved I am willing to upgrade my rating.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QkfhKWvtpU3", "reviewer": "Reviewer_q1CY", "summary": "This paper focus on improving the training efficiency of coordinate based representations by reducing the number of camera views needed during training. To accomplish this, the authors proposed a codebook attention module and a coordinate attention module to inject prior knowledge into implicit representations. Experiments on novel view synthesis are conducted and improvements are shown over baseline methods on some public datasets.", "questions": "- In Table 1. the quantitative results from the presented method and the VolSDF are tied on DTU dataset. However, it seems like in fig 2. the qualitative results are having a larger gap there. I am wondering what is the cause for this?\n", "limitations": "- Although prior knowledge are used in this work, it seems like the model is still trained specifically to one scene and is hard to generalize to novel scenes.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The idea of bringing prior knowledge into implicit representation is novel and interesting.\n\n- The paper is well written and easy to read.\n\nWeaknesses:\n\n- Although we do observe improvements over other baselines, it seems like the changes are relatively small. For example, in fig.2 the results of VolSDF and the presented methods are giving similar rendering results and in table.1 the quantitative results are tied.\n\n- Lack of some ablation studies for a more thorough analysis of the proposed method. For example, the presented methods added two new modules to baseline implicit representations. And when doing comparisons with baseline methods, it is hard to tell whether it is because the addition of the new module brings the improvement or it is probably the network has more weights and could fits better. It could be better if more details are provided, for instance, the number of weights for the presented method and the baselines.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "F28JDvevpA", "reviewer": "Reviewer_Jhwv", "summary": "This paper proposed CoCo-INR, an implciit neural representation trained with sparse multi-view images leveraging on prior information from pre-trained image prototypes/features.\n\nSpecifically, two attention modules called codebook attention and coordinate attention are proposed to introduce most revelant scene priors into the learning process of neural implicit representations, so that few images are adequate to render realistic images as well as high-quality geometric information. Three datasets of different complixities are used to demonstrate the effectiveness of method.", "questions": "1. One important missing experiments are about other strategies in introducing scene priors like PixelNeRF and GRF. Existing experiments only consider baseline methods without external piros so the comparison is not that fair in my point of view.\n\n\n2. Further demonstrate on how CoCO-VolSDF and CoCo-NeRF separately represent the full scene and individual improvements among baseline methods are expected.\n\n3. More ablation on hyper-parameters are welcome.\n\n4. Please consider adding more details about the training set-ups, e.g., how the subset views are selected. Furthermore, depending on how the training subset are selected, it would be good to show if it is possible to infer images given extrapolated view points.\n", "limitations": "\nSome limitations are mentioned in the supplement material. Related to that, does the method scale to larger or unbounded scenes well? Is the cluterred background of these scenes hamper the performance of CoCo-INR? It woukd be good to add some analysis here.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. The idea of integrating VQ codebook into NeRF training looks interesting and leads to promising geometry recontruction given few images on three public datasets.\n\n2. The experimental set-up of ablations and vislusations validate the effectiveness of codebook attention.\n\n3. Though there are some missing details (mentioned in the weakness part), the overall writing is good and easy to read.\n\nWeaknesses\n\n\n1. How to select a subset of training views are not clearly represented and only the number of views are listed. Are subset views evenly sampled are consecutively sampled while leaving the rest as testing views? \n\n    If possible, it would be good to see the camera trajectory or distributions along with rendering results to get better understand the method.\n\n2. Similar to previous points, as cross-attention mechanism are adopted to propagate information from observed to unobserved ones, it would be interesting to see the extrapolated view synthesis in addition to interpolation.\n\n3. One major point missing in my opinion is the comparison to other baselines introducing learnable/pre-trained priors into training process, such as PixelNeRF (CVPR2021) and GRF (ICCV2021).\n\n    How does the codebook prior using VQ compared to stratigies adopted in PixelNeRF or GRF, where CNN feautes are used to augment local priors so that few views could be used to enable view synthesis. I think it is necessary to add related comparisons and discussions since currently all selected baselines does not take any external priors.\n\n4. Another interesting thing is to explore is how the richness of priors affect performance of CoCo-INR? For example,the number of embeddings M and the size or Codebook, for a specific given scene, presumably a limited number of prototypes would be activated while others are not. Will a meaningful subset reduce the overall computation without obvious performance degradation? \n\n5. As Coco-VolSDF and CoCo-NeRF are used to model foreground and background respectively, following NeRF++. It would good to see how the decomposition looks like because it is not clear to readers how the CoCo-NeRF hanles background regions. In Figure 2 and 3, there are some noisy patterns in the first two images while the rest shows a clean backgroud geometry.Is the geometry (normal) only computed from foreground regions? Is the grey region computed from background NeRF? How does the attention mechism helps the background region?\n\n\n6. Minor Issues.\nIn Sec3.2 of supplement, is it claimed that only 3 extreme views are used in Fig.2 while the captions says 8 images are used. Please clarify how many images are used?\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-Wn3zThW2Q5", "reviewer": "Reviewer_tT43", "summary": "The paper presents a method to utilize a non-scene-specific ImageNet-pretrained codebook for learning neural 3D representations, such as NeRFs. The codebook attention module transforms the codebook into a scene descriptor, and the coordinate attention module concatenates it with the positional encoding/features. The intuition is that doing so encourages the network to learn the semantic correlation between the input point (e.g. R^3) and the scene, enabling \"extrapolation\".", "questions": "How did you get the baseline MLP features (e.g. compared in Figure 4)?", "limitations": "The paper does not address limitations and societal impact.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: Demonstrates that the codebook method from VQGAN and VQ-VAE can be applied to 3D representation learning, lifting 2D ImageNet-based features to 3D. While the architecture may seem incremental, the extension to 3D is noteworthy.\n\n\nWeakness: No quantitative ablation study. The \"per-scene learnable features vs. fixed codebook\" and \"impact of codebook attention\" experiments are both important yet entirely qualitative. Improvement from VolSDF could have been due to better initialization, more computation, and so on (as usual).  It would also be helpful to empirically evaluate novel views vs. observed views.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept