CoFie: Learning Compact Neural Surface Representations with Coordinate Fields
Authors: Hanwen Jiang
Haitao Yang
Georgios Pavlakos
Qixing Huang
Venue: NeurIPS 2024 poster (2024)
Keywords: Neural Surface Representation
Shape Auto-Decoding
Coordinate Field.
Primary area: machine_vision

Abstract: This paper introduces CoFie, a novel local geometry-aware neural surface representation. CoFie is motivated by the theoretical analysis of local SDFs with quadratic approximation. We find that local shapes are highly compressive in an aligned coordinate frame defined by the normal and tangent directions of local shapes. Accordingly, we introduce Coordinate Field, which is a composition of coordinate frames of all local shapes. The Coordinate Field is optimizable and is used to transform the local shapes from the world coordinate frame to the aligned shape coordinate frame. It largely reduces the complexity of local shapes and benefits the learning of MLP-based implicit representations. Moreover, we introduce quadratic layers into the MLP to enhance expressiveness concerning local shape geometry. CoFie is a generalizable surface representation. It is trained on a curated set of 3D shapes and works on novel shape instances during testing. When using the same amount of parameters with prior works, CoFie reduces the shape error by 48% and 56% on novel instances of both training and unseen shape categories. Moreover, CoFie demonstrates comparable performance to prior works when using even 70% fewer parameters. Code and model can be found here: https://hwjiang1510.github.io/CoFie/

CoFie: Learning Compact Neural Surface Representations with Coordinate Fields
Hanwen Jiang Haitao Yang Georgios Pavlakos Qixing Huang
Department of Computer Science, The University of Texas at Austin iang, pavlakos, huangqx) wtw.....................ca..a..e utexas.edu
Abstract
This paper introduces CoFie, novel local geometry-aware neural surface rep- resentation. CoFie is motivated by the theoretical analysis of local SDFs with quadratic approximation. We find that local shapes are highly compressive an aligned coordinate frame defined by the normal and tangent directions of local shapes. Accordingly, we introduce Coordinate Field, which composition coordinate frames of all local shapes. The Coordinate Field is optimizable and is used to transform the local shapes from the world coordinate frame the aligned shape coordinate frame. It largely reduces the complexity of local shapes and benefits the learning of MLP-based implicit representations. Morcover, we intro- duce quadratic layers into the MLP to enhance expressiveness concerning local shape geometry. CoFie is a generalizable surface representation.
trained on curated set of 3D shapes and works novel shape instances during testing. When using the same amount of parameters with prior works, CoFie reduces the shape error by 48% and 56% novel instances of both training and unseen shape categories. Moreover, CoFie demonstrates comparable performance to prior works when using even 70% fewer parameters. Code and model can be found here: https://wwjiangl 1510 wtw.:aaggn.......gg.ooooooooo iithub
1 Introduction
In the realm of geometry modeling, neural implicit shape representations have become powerful tool [33,7 7, 4, 13, 41, 39, 2]. These representations typically use latent codes to represent shapes and employ multilayer perceptions (MLPs) to decode their Signed Distance Functions (SDFs). Early works in this field use a single latent code to represent an entire shape [33]. Nevertheless, the decoded SDFs usually lack geometry details. To improve the shape modeling quality, recent approaches have introduced local-based designs [4, 26, 42J. By decomposing an entire shape into many local surfaces, the shape modeling task becomes effortless local surfaces are simpler geometry which are easier to represent. Despite the progress, the local -wcaee design significantly increases the number of parameters, as each local surface represented by one or even multiple latent codes. Thus, proposing a neural surface representation that is both accurate and compact is necessary.
To achieve this goal, we argue it is important understand the properties of local surfaces. Following prior works [31, 49, 10, 44], we approximate the local geometry with quadratic patches [9] and perform analysis. Results show the feasibility of fitting the geometry of specific category of quadratic patches. In detail, the quadratic patches are aligned with the coordinate system defined by the normal, principal directions, and principal curvatures of quadratic patch [9. 31]. However, when the quadratic patches are not aligned - they are freely transformed with random rotations and translations in 3D, mimicking real local surfaces the optimization will be easily trapped into local
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Local Patones
Spod Compaerry
Transform
Aägned
Figure 1: CoFie is local geometry-aware shape representation. (Left) CoFie divides a shape into non- overlapping local patches, where each local patch is represented by MLP-based Signed Distance Function. (Right) CoFie introduces Coordinate Field, which attaches a coordinate frame each local patch. transforms local patches from the world coordinate system to an aligned coordinate system, reducing shape complexity.
minima. This analysis reveals the difficulty of jointly recovering transformation information and geometry of local patches.
Based on the analysis, we propose CoFie, novel local geometry-aware neural surface representation. The key insight of CoFie is decomposing the transformation information of local shapes from its geometry. As shown in Fig. I, we associate each local surface with a learnable coordinate frame, which forms Coordinate Field. We use the Coordinate Field transform all local surfaces into an aligned coordinate system, reducing their spatial complexity. Thus, the geometry space of local surfaces becomes more compact, where the MLP-based neural SDFs are easier to learn.
An important design aspect is how to represent the Coordinate Field. Departing from the implicit- based representations, we
explicit representation. Specifically, the coordinate frame of each local surface is parameterized by rotation and a translation, forming a 6 Degree-of-Freedon pose. Moreover, we initialize the rotation using the estimated normal, principal direction, and principal curvature of a local surface. This design makes CoFie local geometry-awee aware and facilitates the learning of Coordinate Fields.
To better represent local surfaces' geometry, we introduce quadratic layers to the MLP. Prior works typically employ ReL-U-based MLP with shallow layers and limited hidden size [33, 4J. Thus, the MLP is piece-wise linear [25] and cannot represent the distribution of local surfaces well. We demonstrate simple quadratic layer improves the geometry modeling capability.
CoFie a generalizable shape representation. After training on curated dataset, it can represent arbitrary shapes that belong to any novel category. We evaluate CoFie on novel shape instances from both seen (training) and unseen categories, encompassing both synthetic and real shapes. Results show that Colie outperforms prior arts, reducing the chamfer distance by 50% instances from both seen and unseen categories. Moreover, Colie achieves comparable results with prior work using 70% less parameters. In addition, we demonstrate that CoFie, which uses single shared MLP for all shapes, achieves comparable results with methods that overfit a specifie model for each testing shape.
2 Related Work
Implicit Shape Representations. Implicit shape representations are state-of-the-art in encoding shape geometric details [33, 41, 39, 37, 53, 48, 52, 45, 28, 15, 55, 16, 34]. To improve the shape modeling capability, researchers inject local-aware designs. For example, DeepLS [4] integrates voxel grids and local MLPs to decode geometric shapes. Another line of work explores hierarchical
representations where the local surfaces are divided unevenly [30, 47, 26, 42, 46, 40], leveraging Octree, For example, Multilevel Partition of Pnity [30] (MPU) blends parametric implicit surface patches into global implicit surface. DOGNet uses dual-octree designs for neural MPU. The contribution of CoFie is perpendicular these methods. CoFie still works on evenly divided voxels. However, instead of resolving fighffreuuency details of local shapes by using higher local resolution, CoFie proposes the Coordinate Field to reduce the spatial complexity. This is motivated by the analysis result that local geometric shapes are highly compressive under suitable coordinate frames.

---

The idea of coordinate field is related to several existing approaches. For example, MVP [27] introduced oriented boxes for 3D face synthesis. However, in our setting, the variations in geometry and topology are much more significant than those of 3D human faces. Another relevant work is LDIF [18], which transforms 3D point in the local coordinate system of each primitive to decode the iso-value of each shape. However, LDIF uses fixed coordinate frame for each primitive. In contrast, the coordinate field varies spatially in CoFie and can be optimized, allowing us to capture detailed variations of the parts flexibly. Moreover, CoFie is based on a rigorous analysis of the expressivity of SDF and SDF learning. A follow-up approach [56] uses a warping field to transform a 3D point into canonical space of specific categories. In contrast, CoFie is category -agnost,,,,,,iic, benefiting from the use of local shapes.
On the learning side, many approaches show that MLPs are expressive and that their performance depends on the loss of training. For example, SAL [1] and SALD [2] show the importance of integrating normal losses to capture geometric features. SIREN [39] introduced other regularization losses to improve the quality of implicit representations learned. Although these approaches focus on local shape details, CoFie focuses network design using coordinate frames. The CoFie approach
is orthogonal to the encoding schemes.
Hybrid 3D Representations. Each 3D representation has fundamental advantages and limitations from the machine learning and representation perspective. For example, implicit representations allow flexible topologies. whereas explicit representations are easier to edit. Therefore, hybrid 3D representations, which aim add the strength of different 3D representations for representation learning, have received lot of attention. The main stream in hybrid 3D representations sequentially applies hybrid 3D models [24, 50, 12, 11, 38, 29, 8, 51]. For example, GRASS [24] combines a part-based representation to capture geometric structures of 3D shapes and a volumetric representation
per part to capture geometric details of the parts. DSG-Net [50] employs part-based deformations to capture geometric details of the part. Other examples [12. 11, 38, 4, 14, 5, 19, 20] combine explicit graph, mesh, voxel and triplane representations with implicit volumetric representations to encode geometry details. CoFie is relevant to this series of approaches, where it combines voxel grids to encode global shapes and an implicit representation to decode local geometric details. The novelty of CoFie is that the local module employs coordinate frame representation and enforces the prior knowledge that the local shape is roughly a low-complexity polynomial surface in the coordinate system defined by normal and principal directions.
Coordinate Field Optimization. The task of computing the proposed cell-based coordinate field is related to the problem of feclor-field and frame-field design meshes, where we want to ensure that the coordinate field is smooth and consistent among adjacent cells, and where we want the normal and tangent directions of each coordinate frame to align with the local fitting results if the fitting results are highly confident. This problem was studied [35], which introduced global optimization framework to compute global vector field triangular mesh. Several more recent approaches have developed improved formulations for vector field optimization [17, 22] and extensions to frame field optimization [32, 36]. We refer to [43] for surveys on this topic. Rather than solving global optimization problem to compute the coordinate field, the learning of the coordinate field in CoFie is driven by learning compressive MLP.
Analysis of Fitting SDFs of Local Patches
In this section, we provide an analysis of fitting local surfaces. Following prior works [31, 49, 10, 44], simplify local surfaces as quadratic patches. Additionally, we note that some works approximate local surfaces with linear patches [23, 47]. However, to handle the geometry details, it usually requires extremely high [47] or infinite resolution [23] during local surface partition. Approximating local surfaces with quadratic patch is more practical.
3.1 Importance of Non-linearity
quadratic surface patch can be represented by f(u, v) = (u, 1 (au² + cv² + 2buu)), where u² + x2 < ,2 for locality, and a, b, e are parameters for controlling the shape of the quadratic patch. The following proposition characterizes the SDF of a point p to f.
v,
3

---

Proposition 1 For each point p v.2) in the neighborhood of the origin the signed distance function from p f(u, 1) can be approximated as
d(p, f(u, x - = czs² + 2bry).
(I)
where the approximation omits third-and-higher order terms in Y and 2.
Proof: See Appendix A.
Prop. suggests that the SDF is non-linear. However, shallow MLP using ReLU activation is piecewise linear, where the ReLU activation functions essentially decompose the input space into subspaces and the function in each subspace is still linear. This motivates the use of quadratic layers instead of linear layers (Sec. 4.2).
To hold generality, in Appendix B, also analyze the local surface that not be simplified as single quadratic patch, i.e. sharp edges as the intersection two quadratic patches.
3.2 Difficulty of Fitting Transformation Information
We demonstrate the difficulty of recovering the transformation information of quadratic patches during geometry fitting.
Aligned Quadratic Patches. Same as the previous section, we define the SDF of quadratic local patch as 2 1 {ax² cy² + 2bxy), where the quadratic patch axis-aligned. Consider set of 2
S n from this quadratic patch, where (æz, Vs. 2;) is the location of point p;. d, the SDF value, and n is the number of samples. To fit the surface from the samples, we solve the optimization problem as
samples {((;, Vt. 2,). d.),
arg min a.b.c
Σ (z i=1
laa2 + cył + 26x,y1) - di)
which is a convex problem that has a unique global optimal.
Unaligned Quadratic Patches. Consider transforming the quadratic patch with a random rigid transformation (R, t). This quadratic patch is not axis-aligned. In this case, the SDF function is given by z' (ax' + cy²² + 2hx'y') where (a', v','') =') R(x, y, 2) + t. To fit the surface from the samples, we solve the optimization problem as
n
Σ (et 1--2(ax
[axx + +zy22 2bx(//) d.)²
arg min R,
R
Y/
+
24
In this case, (3) becomes non-convex and has local minima. We defer detailed characterization of the local minima of (3) to Appendix C.
In general, this non-convex problem makes geometry fitting non-trivial. It motivates the of the Co- ordinate Field to explicitly model the transformation information and disentangle the transformation information of local patches from its geometry (Sec. 4.1).
CoFie
In this section, we introduce details of CoFie, including its representation (Sec. 4.1), MLP architecture (Sec. 4.2), and its learning scheme (Sec. 4.3).
4.1 CoFie Representation
As shown in Fig. 2, CoFie is based on a hierarchical representation, with coarse and fine- ine-grained geometry. the coarse level, it represents a shape with voxels. In detail, for a shape S, it divides the space that contains the shape into V x V x V non-overlapping voxel grids, where V is the resolution of the voxel grids. subset of voxels that intersect with the shape surface will be valid and CoFie only consider the valid sparse voxels to ensure its efficiency.

---

Data Preparation
Model
Oi Nermalast Scila)
Truning Shapes
Local Shapes
Local Verday
Local Shapes
Train
Fointx (Nrti
Transtormed Point,
jAggned Coord
Prodiction
Optimized Parameteess
HLP
Local Stape Celie
HILP
BOR
Coondaan
Farttion
Partion
Prudiction
Cytirissd Panometes
SGP Lile
HLP
A
Figure 2: Overview of CoFie. CoFie represents a shape using hybrid representation of voxels/cells and local implicit functions. (Left) For preparing the data for training the MLP-based local implicit functions, we split the training shapes into local shapes and initialize their coordinate frames using PCA. (Right) During training. point will be transformed the aligned coordinate all local shapes using coordinate frame. The MLP takes the transformed point and the latent code the local shape to predict its SDF value. During testing, we fix the MLP, optimizing the latent codes and coordinate fields of valid cells.
At the fine-grained level, for each valid voxel v, we use an implicit representation to encode the geometry details for the local surface inside the voxel. Specifically, we use MLP--basd neural SDFs. Each voxel has latent code 2, representing the local geometry and we use the MLP g to decode the SDF values. For point æ, its SDF value contributed by the voxel is
f v) gd (Zv: ze),
Ey (nu; bu The xx (x=-).) o.),
where (o., ne (.)) parameterize the coordinate frame of voxel Ideally, og, n. and ty are the origin, normal direction, and tangent direction of the local surface, respectively.
Intuitively, for decoding the SDF value, we transform the point from the world coordinate system to the shared coordinate system for all local surfaces. o, forms the translation between the two coordinate system, and (n., fv) form the rotation.
The final SDF value is then given by
f(x) Evey EnE w(r, Evey w(x
(5)
where V is the set of all valid voxels, w(x:, v) is the weight assigned for the voxel with regard to point æ. In practice, use w(æ v) if v, and w( v) otherwise, Finally, the surface of a 3D shape is defined as the union set of local surfaces in its valid voxels V.
4.2 CoFie MLP Architecture
Following the common practice of MIP,, define
g0(x, z) = Ar o 01-1 o oo (x,
where g - >> m is a layer with trainable parameters 0₁ and where & is an activation function. Denote as the output in layer I, i.e., 20 z). A common strategy is to set each " as linear function, i.e.,
9.t (x-1) A bi.
(6)
where 0, = (A₁, b.). Furthermore, Φ is chosen as the ReLU layer, i.e., d0z1) max (21, 0) where the max operator applied clement-wise. This strategy is widely used in prior works [33, 7].
However, in Sec. 3.1, we demonstrate the SDF function has non-negligible quadratic components locally and its incompatibility with MLPs with linear layers and ReLU activation. Therefore, instead, we model the quadratic components with quadratic layers. We let the top k layers of g to be quadratic functions, where 1. The quadratic layer can be formulated as
81 (zI-1) = Tizi + A + b
(7)
where Tj E 11 Xm is tensor, and 0 (Ti, Ar, b.).
5

---

We can understand the trade-offss between the use of linear layers (Eq. 6) and the quadratic layers (Eq. 7) as follows. With the same latent dimensions my, the quadratic layers have many more parameters than the linear layers. Therefore, with the same network size, we have to use fewer layers or smaller latent dimensions for quadratic layers. This will limit the capability of the network instead. In practice, setting k = 1 leads to the best performance.
4.3 CoFie Learning Scheme
Problem Setup. Following DeepSDF-series, we perform shape auto-decoding [33, 4]. The task assesses the capability of models fit/represent given shapes. During both training and inference, the input is points sampled freely in space with their ground-truth SDF values. The output is the neural SDF. Additionally, we notify that the task different from shape reconstruction from point cloud inputs, or so-called shape auto-encoding, which is studied in [54, 8, 29].
Moreover, CoFie is a generalizable shape representation. It is trained a curated dataset with multiple shapes. Once trained, the MLP can be used to represent or decode the SDF of any incoming shapes. We note the setting of generalizable shape representation different from overfitting shape, where an MLP is specialized for each shape.
Training and Inference. We follow the protocol of the shape auto-decoding task [33, 4]. We train CoFie with a set of shapes denoted as S = [St, < i < n). For each shape, we perform voxelization (Sec. 4.1) and train CoFie with valid local shapes. We denote the set of valid local shapes of shape S; as V. Following [33, 39, 4], we collect a set of point samples P. = (p', d²) in the neighborhood of each voxel V. where p' and di denote the position of the sample and the SDF value of p The point samples are sampled in free space and are not necessary to be on-surface points. For each local shape in voxel v, we associate with latent code y and the coordinate frame (O uy by). Then the training objective can be formulated as
ΣΣ Σ 8, te 1=1 VEV. (p) ,.iEEP
arg min
l1g (p', zv) - d'Il1.
(8)
where pi: - (n by ny X t.)) o.). In this step, we jointly optimize the MLP, the latent codes, and the coordinate field for all training shapes. Intuitively, it trains the MLP to represent training shapes and optimize the compatibility between the MLP, latent codes and the coordinate fields.
During inference, we freeze the MLP g We optimize the latent code and the coordinate field for a single target shape time. It is formulated
arg min (f vEV (p" ....PP
Σ
Σ
(p',)), d'Ila
Besides, we use the regularization term over the inferred latent codes following [33, 4].
Shape Consistency Boundary of Voxels. If we sample the points P, within each voxel v. Eq. and Eq. optimize the local geometry within each voxel independently. This may lead the non- smooth and inconsistency surface the boundary of voxels. To solve this, follow [4] to expand receptive field of each voxel by sampling points from their neighbouring voxels.
Coordinate Field Initialization. Eq. 8 has many unwanted local minima, especially for optimizing the coordinate field. Thus, a good initialization of the coordinate fields ensures the compactness of local shape at early stage of training, and facilitates the learning of MLP. Motivated by the analysis in Sec. 3.2, we use estimated normal and tangent directions to initialize the coordinate fields. In detail, compute the derivatives of SDF values these point samples and perform PCA to get them. Besides, o, is initialized as the center of the cell. We find that this initialization is important to reduce errors (Sec. 5.2).
Experiment
This section presents an experimental evaluation of CoFie. begin with the experimental setup and then present the results and ablations.

---

Figure 3: Diveristy and quality of meshes that CoFic can represent. The results include both novel instances from ShapeNet training categories (top left), instances from ShapeNet unseen categories (bottom left), and real shapes from the Thingi dataset (right). We visualize the shapes with surface normal better show their geometry. Please see the appendix for comparisons with ground-trun
Implementation Deatils, We use latent code of size 125 for all cells. The MLP is composed of layers where the first 4 layers are linear layers and the last layer is quadratic. The hidden channel size is 128. We use the voxel grid size of 32 x 32 x 32. During training, we use 12 shapes for each batch. For each shape, we sample 3000 voxels that intersect with the surface of the shape (with return). We sample 24 points for each cell for training. and each point is sampled within 1.5 times the radius of the voxel to ensure boundary consistency between cells. We the Adam optimizer [21] with learning rates 5e 4. le 3, and 1e 3 for the MLP, coordinate fields, and latent codes. We train with 150000 iterations and reduce the learning rates by half for every 20000 iteration. During inference, we use a learning rate of 5e 4 for 800 iterations. Reconstructed meshes are obtained by performing Marching Cubes with 128 resolution by default. We use the quaternion representation for the rotation matrix of the coordinate frames. We train on GPUs with 24GB memory for day.
Training and testing data. We train CoFie 1000 shape instances sample from ShapeNet [6] of chairs, planes, tables, lamps, and sofas (200 instances for each category). We test CoFie with three test sets for comprehensive analysis of CoFie: i) 250 novel instances from the 5 training ShapeNet categories; ii) 250 novel instances from 10 unseen ShapeNet categories; iii) 24 meshes from the Thingi dataset [57]. which captures real scenes. The test set i) checks how CoFie fits the training distribution. Test sets ii) and iii) are used to test the generalization capability of CoFie on novel shapes that observe different structures with training shapes.
Baseline Approaches We compare our CoFie with three types of methods: generalizable methods, which use a single MLP to represent multiple shapes; shape-specific methods, which train MLP for each testing shape. Generally, the latter genre demonstrates better performance as the MLP model can be trained to overfit single testing shape. Both the two types of methods performs shape auto-decoding. Besides, we also report results for a statc-of-ther the-the-rtt shape auto- encodenn method. We note that it is reference method while the result is not directly comparable.
Note that CoFie is generalizable method for shape auto-decoding. We include more details for baselines as follows.
DeepSDF [33] is generalizable shape auto-decoding method using a global latent code to represent one shape.
DeepLS [4] is a generalizable shape auto-decoding method using local-basee representations. DeepLS is a direct comparable baseline.
NGLOD [40] a shape-specific method for shape auto-decoding, achieving -heee- performance. For a fair comparison with CoFie, we use the level of detail as 3, keeping the number of parameters the latent codes in the same magnitude as our CoFie.
3DS2VS [54] is generalizable shape auto-encoding method. It employs transformers to predict the shape latent code, rather than getting it by optimization (shape auto-decoding) The input is on-surface point clouds.
7

---

Table 1: Shape errors on novel instances ShapeNet training categories. We report chamfer distance (10 and highlight the best.
DeepLs
Shape Eioor
Novel Instances of Seen Shape Category
chair
lamp
plane
sofa
table
mean
3DS2VS
9.11
10.9
1.68
8.76
13.7
DeepSDF
5.69
15.1
7.51
4.08
6.64
7.84
DeepLS
7.70
6.57
0.83
2.54
2.18
3.91
40
120
14C
Latent code langth
CoFie (ours)
2.35
3.13
0.80
2.44
1.41
2.05
Figure 4: Trade-off between accuracy
model size (notified by the radius of circles).
DeepSDF
DeepLS
Ours
GT
NGLOD
Ours
GT
(Zoom-in)
Figure 5: Comparison with prior works. (Left) Results of generalizable methods, where CoFic demonstrates better capability for modeling geometry details, (Right) Compare with the per-shape-base method NGLOD. We note that NGLOD is a shape-speed specific method that overfits one MLP on one testing shape.
Besides, we also compare with state-of-the-aae shape auto- auto-encoding (point cloud reconstruction) methods. We note these methods are used as reference for understanding the model performance. They are not directly comparable.
We train Deep DeepL.,, and CoFie using the same dataset for fair comparisons. NGLOD is trained on each test shape. All methods receive the same inputs during inference.
We report the mesh reconstruction error as the chamfer- L2 distance between the reconstructed and ground-truth meshes. We sample 30000 points compute the chamfer distances. The meshes are normalized into a unit scale.
Evaluation Metries
5.1 Experimental Results
Qualitative Results. As shown in Fig. 3, CoFie demonstrates strong surface representation capability. The details of geometry are maintained well. The results on out-of-distribttion shapes from unseen categories are comparable to the training categories.
Performance on Training Categories. As shown in Table 1, CoFie outperforms baselines by large margin. In detail, the average chamfer distance of CoFie is 1.86 (48% relatively) smaller than the best baseline DeepLS. Moreover, we provide more detailed comparison with DeepLS, as shown in Fig. We observe that CoFie consistently better than DeepLS with different latent code and MLP size. Specifically, CoFie with latent code size 48 achieves slightly better performance compared with Deepl.S with latent code size 128. Note that the number of ML.P parameters for the former about 15% for the latter.
8

---

Table 2: Shape errors instances of the ShapeNet novel categories. We evaluate the chamfer distance (10%
Unseen Shape Category
cabinet
phone
bus
guitar
clock
bottle
mug
washer
rifle
3DS2VS
16.4
12.7
21.9
24.1
2.4
10.5
10.6
9.3
26.7
25.3
16.0
DeepSDF
12.3
6.87
6.92
18.4
11.8
10.6
4.54
10.83
6.17
15.7
10.4
DeepLS
9.74
5.77
2.09
7.22
0.63
4.30
12.7
7.28
18.8
4.13
7.27
CoFie (ours)
4.19
3.09
1.86
3.66
1.23
3.57
4.48
2.58
4.23
2.88
3.18
Table 4: Ablation study (0) Base performance; (1) coordinate field and its initialization methods; (2) using quadratic MLP: (3) full performance. We use resolution 128 to get reconstructed meshes.
Table 3: Results Thingi meshes. We evaluate the chamfer distance (10") with marching cube resolution 256. Note that NGLOD trained on each test shape, while CoFie uses a shared MLP for all shapes generalizable method.
Coord. (CF)
Settings
Emor
Use CF Geo-Aware Inil.
Linear e Quad.
(0)
X
3.91
Unseen Thingi Shapes
X
Generalizable
Total MLP Size
Shape Erer
(1)
NGLOD
X
24 x0.2MB
1.04
3.01
(2)
DeepSDF
0,2MB
3,68
x
X
3.70
CoFie (ours)
0,2MB
1.87
(3)
2.05
Performance on Unseen Categories. We compare CoFie with previous generalizable methods on ShapeNet unseen categories and the state-of-ther afrt per-shape-based method the challenging real scans. We provide visualization results Fig. 5.
ShapeNet Unseen Categories. As shown in Table CoFie achieves better generalization on 9 out of 10 novel shape categories. We also observe that the performance gap between CoFie and prior works is larger in the unseen categories, showing the strong generalization capability of CoFie.
Thingi Real Shapes. As shown in Table 3, CoFie achieves comparable results with NGLOD, We note that NGLOD is per-shape-baset method, which trains model for each shape and performs better naturally. In contrast, CoFic trained on ShapeNet shapes.
5.2
Ablation Study
As shown in Table 4, we experiment with CoFie variants to validate the effectiveness of our coordinate field and MLP designs.
Coordinate Field and Initialization. As shown in Table 4 (1), using coordinate fields with different initialization strategies can both reduce the shape error. detail, when using axis-aligned coordinate field initialization, where all coordinate frames are initialized as the world frame, the shape error reduced slightly from 3.91 to 3.45. The result demonstrates the difficulty frames. In contrast, when using geometry-aware initialization, i.e., initializing local frames with estimated normal and tangent directions of local shapes, the shape error is reduced to 2.33, observing 40% improvement.
optimizing coordinate
MLPP Design. As shown Table 4 (2), using a quadratic layer as the last layer of the MLP observes 0.9 (23% relatively) reduction of shape error. As the use of the quadratic layer introduces additional parameters, we compare it with a variant for a fair comparison. In detail, we compare it with a linear MLP with additional layer (6 layers in total), where the two MLPs have the same amount of parameters because the output channel size the last layer 1. The result shows that increasing number of linear layers only reduce the shape error slightly.
Moreover, Table 4 (3) demonstrates the combination of the introduced techniques can jointly reduce the shape error.
6 Conclusions and Future Work
This paper has introduced CoFie, novel neural surface representation. is based on the theoretical results of using ReLU-based MLP to encode geometric shapes. The results strongly motivate

---

the use of local coordinate frames, which encompass the coordinate fields, to transform a point before decoding its SDF value using an MLP. This leads to a hybrid representation combined with coordinate frames associated with local voxels. The experimental results show strong generalization behavior CoFie in new instances for shape reconstruction, which significantly outperforms previous generalizable methods and achieves comparable results to shape-specifie methods.
Limitations. One limitation of CoFie is that is based on local shapes and cannot be used for the shape completion task. Different from DeepSDF, which learns global shape priors and can fill the large missing components in the input, CoFie is restricted to observable parts. We plan to incorporate more global priors into CoFie. Besides, with fixed cell resolution, the local shape analysis is broken when a local cell intersects with thin structures. We plan to extend it with adaptive local cell resolutions.
Broader Impact. CoFie is a neural surface representation, which have the potential to be used for 3D) reconstruction and generation.
Acknowledgment.
Q. H. would like to acknowledge NSF IIS 2047677 and NSF IIS 2413161
References
[1] Matan Atzmon and Yaron Lipman. SAL: sign agnostic learning of shapes from raw data. In CVPR, 2020.
[2] Matan Atzmon and Yaron Lipman. SALD: sign agnostic learning with derivatives. In ICLR, 2021.
[3] Tristan HumentaadoAAmsstrorr Stavros Tsogkas, Sven Dickinson, and Allan D Jepson. Representing
3d shapes with probabilistic directed distance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 19343-19354, 2022.
[4] Rohan Chabra, Jan Eric Lenssen, Eddy Ilg, Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard A. Newcombe. Deep local shapes: Learning local SDF priors for detailed 3d reconstruction. In ECCV, 2020.
[5] Eric R Chan, Connor 7 Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, al. Efficient geometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16123-16133, 2022.
[6] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang. Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository, 2015.
[7] Zhiqin Chen and Hao Zhang. Learning implicit fields generative shape modeling, CVPR, 2019.
[8] Yen-Chi Cheng, Hsin- Ying Lee, Sergey Tulyakov, Alexander G Schwing, and Liang- Yan Gui. Sdfusion: Multimodal 3d shape completion. reconstruction, and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4456-4465, 2023.
[9] Manfredo P. do Carmo. Disferential geometry of curves and surfaces. 1976.
[10] Roderik GF Erens, Astrid MI Kappers, and Jan Koenderink. Perception local shape from shading. Perception & psychophysics, 54:145-156, 1993.
[11] Lin Gao, Jie Yang. Tong Wu, Yu-Jie Yuan, Hongbo Fu, Yu-Kun Lai, and Hao Zhang. Sdm-net: Deep generative network for structured deformable mesh. ACM Trans. Graph. 2019.
[12] Georgia Gkioxari, Justin Johnson, and Jitendra Malik. Mesh R-CNN. In ICCV, 2019.
[13] Amos Gropp. Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization for learning shapes. In JCML, 2020.
[14] Vicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Irm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.0e 1.04200 2023.
[15] Jiahui Huang, Zan Gojeic, Matan Atzmon, Or Litany, Sanja Fidlor, and Francis Williams. Neural kernel surface reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4369-44-44 4379, 2023.
[16] Ka-Hei Hui, Aditya Sanghi, Arianna Rampini, Kamal Rahimi Malekshan, Zhengzhe Liu, Hooman Shayani, and Chi- Wing Fu. aake-a-shape: a ten sen-million-scal 3d shape model. In Forty-first International Conference on Machine Learning, 2024.
10

---

[17] Wenzel Jakob, Marco Tarini, Daniele Panozzo, and Olga Sorkine- Hornung. Instant field-aligned meshes. ACM Trans. Graph., 2015.
[18] Chiyu "Max" Jiang, Ayneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießnnrr and Thomas A. Funkhouser. Local implicit grid representations for 3d scenes. In CVPR, 2020.
[19] Hanwen Jiang, Zhenyu Jiang. Kristen Grauman, and Yuke Zhu. Few-view object reconstruction with unknown categories and camera poses. arXiv preprint arXiv:2212.04492 2022.
[20] Hanwen Jiang, Zhenyu Jiang, Yue Zhao, and Qixing Huang. Leap: Liberate sparse-view 3d modeling from camera poses. arXiv preprini 23X1v:3100004100, 2023.
[21]
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[22] Felix Knöppel, Keenan Crane, Ulrich Pinkall, and Peter Schröder. Globally optimal direction fields. ACM Trans. Graph. 2013.
[23] Ravikrishna Kolluri. Provably good moving least squares. ACM Transactions on Algorithms (TAL.G), 4(2):1-25, 2008.
[24] Jun Li, Kai Xu, Siddhartha Chaudhuri. Ersin Yumer, Hao Zhang, and Leonidas Guibas. Grass: Generative recursive autoencoders for shape structures. ACM Trans, Graph., 2017.
[25] Ruiyuan Lin, Suya You, Raghuveer Rao, and C-C Jay Kuo. Constructing multilayer perceptrons as piece- wise low-order polynomial approximators: a signal processing approach. arXiv preprint arXiv:2010.07877 2020.
[26] Qinqing Liu, Peng-Shual Wang. Chunjiang Zhu, Blake Blumenfeld Gaines, Tan Zhu, Jinho Bi, and Minghu Song. Octsurf: Efficient hierarchical voxel-based molecular surface representation for protein- affinity prediction. Journal of Molecular Gruphics and Modelling, 105:10786: 2021.
[27] Stephen Lombardi, Tomas Simon, Gabriel Schwartz, Michael Zollhoefer, Yaser Sheikh, and Jason Saragih. Mixture of volumetric primitives for efficient neural rendering. ACM Trans. Graph., 2021.
[28] Yujie Lu, Long Wan, Nayu Ding, Yulong Wang, Shuhan Shen, Shen Cai, and Lin Gao. Unsigned orthogonal distance fields: An accurate neural implicit representation for diverse 3d shapes. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20551-20560, 2024.
[29] Paritosh Mittal, Yen-Chi Cheng, Mancesh Singh, and Shubham Tulsiani. Autosdf: Shape priors for 3d completion, reconstruction and generation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 306-315, 2022.
[30] Yutaka Ohtake, Alexander Belyaev, Mare Alexa, Greg Turk, and Hans-Pete Scidel. Multi-level partition of unity implicits. ACM Trans. Gruph., 2003.
[31] Yutaka Ohtake, Alexander Belyaev, Mare Alexa, Greg Turk, and Hans-Peter Seidel. Multi-level partition of unity implicits. ACM SIGGRAPH 2003 Papers, 2003.
[32] Daniele Panozzo, EnriCO Puppo. Marco Tarini, and Olga Sorkine- Frame fields: Anisotropic and non-orthogonal cross fields. ACM Trans. Gruph., 2014.
[33] Jeong Joon Park, Peter R. Florenee, Julian Straub, Richard A. Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR, 2019.
[34] Dmitry Petrov, Pradyumn Goyal, Vikas Thamizharasan, Vladimir Kim, Matheus Gadelba, Melinos Averkiou, Siddhartha Chaudhuri, and Evangelos Kalogerakis. Gem3d: Generative medial abstractions for 3d shape synthesis. ACM SIGGRAPH 2024 Conference Papers, pages 1-11, 2024.
[35] Nicolas Ray, Wan Chiu Li, Bruno Lévy, Alla Sheffer, and Pierre Alliez. Periodic global parameterization. ACM Trans. Graph., 2006.
[36] Nicolas Ray, Dmitry Sokolov, and Bruno Lévy. Practical 3d frame field generation. ACM Trans. Graph., 2016.
[37] Tianchang Shen, Jun Gao, Kangxue Yin, Ming- Yu Liu, and Sanja Fidler. Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. Advances in Neural Information Processing Systems, 34:6087-6100 61011 2021.
[38] Tianchang Shen, Jun Gao, Kangxue Yin, Ming- Yu I Liu, and Sanja Fidler. Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. In NewrIPS, 20211
[39] Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In NewrIPS, 2020.
[40] Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real- time rendering with implicit 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11358 11367, Washington, DC, USA, 2021. IEEE
11

---

[41] Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. NeurIPS, 2020.
[42] Jia-Heng Tang, Weikai Chen, Jie Yang. Bo Wang. Songrun Liu, Bo Yang. and Lin Gao. Octfield: Hierarchical implicit functions for 3d modeling. arXiv preprint arXiv:211.01007. 2021.
[43] Amir Vaxman, Marcel Campen, Olga Diamanti, David Bommes, Klaus Hildebrandt, Mirela Ben-Chen Technion, and Daniele Panozzo. Directional field synthesis, design, and processing, In ACM SIGGRAPH 2017 Courses, 2017.
[44] Timothy P Wallace, Owen Robert Mitchell, and Keinosuke Fukunaga. using local shape descriptors. IEEE Transactions on Pattern Analysis and Machine Intelligence, (3):310- 323, 1981.
shape analysis
[45] Li Wang, Weikai Chen, Xiaoxu Meng, Bo Yang, Jintao Li, Lin Gao, et al. Hsdf: Hybrid sign and distance field for modeling surfaces with arbitrary topologies. Advances in Newral Information Processing Systems, 35:32172-32185. 2022.
[46] Peng-Shuai Wang. Yang Liu, and Xin Tong. Dual octree graph networks for learning adaptive volumetric shape representations. 2022.
[47] Peng- Soug-Shuii Wang. Chun-Yu Yu Sun, Yang Liu, and Xin Tong. Adaptive o-enn: A patch-based deep represen- tation 3d shapes. ACM Transactions on Graphics (TOG), 3716) 2018.
[48] Zixiong Wang. Yunxiao Zhang, Rui Xu, Fan Zhang, Peng-Shus Shuai Wang. Shuangmin Chen, Shiging Xin. Wenping Wang, and Changhe Tu. Neural-singuulr--eer hessian: Implicit neural representation of unoriented point clouds by enforcing singular hessian. ACM Transactions on Graphics (TOG), 42(6): 1-14, 2023.
[49] Ying Xiong, Ayan Chakrabarti, Ronen Basri, Steven J Gortler, David W Jacobs, and Todd Zickler. From shading to local shape. IEEE transactions on pattern analysis and machine intelligence, 37(1):67-79 2014.
[50] Jie Yang, Kaichun Mo, Yu-Kun Lai, Leonidas J. Guibas, and Lin Gao. Dsg-net: Learning disentangled structure and geometry for 3d shape generation. ACM Trans. Graph., 2022.
[51] Lior Yariv, Omri Puny, Oran Gafni, and Yaron Lipman. Mosaic-sdf for generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4630-4639, 2024.
[52] Jianglong Ye, Yuntao Chen, Naiyan Wang, and Xiaolong Wang. Gifs: Neural implicit function for general shape representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 12829-12839, 2022.
[53] Tarun Yenamandra, Ayush Tewari, Nan Yang, Florian Bernard, Christian Theobalt, and Daniel Cremers. Fire: Fast inverse rendering using directional and signed distance functions. In Proceedings of the JEEE/CVF Winter Conference on Applications of Computer Vision, pages 3077-3087, 2024.
[54] Biao Zhang, Jiapeng Tang, Matthias Niessner, and Peter Wonka. 3dshape2vecset: A 3d shape representation for neural fields and generative diffusion models. ACM Transactions on Graphics (TOG), 42(4):1-16,
2023.
[55] Longwen Zhang, Ziyu Wang, Qixuan Zhang, Qiwei Qiu, Anqi Pang, Haoran Jiang, Wei Yang. Lan Xu, and Jingyi Yu. Clay: controllable large-scale generative model for creating high-quality 3d assets. ACM Transactions on Graphics (TOG), 43(4):1-20, 2024.
[56] Zerong Zheng, Tao Yu, Qionghai Dai, and Yebin Liu. Deep implicit templates for 3d shape representation. In CVPR, 2021.
[57] Qingnan Zhou and Alec Jacobson. ThingilOk: A dataset of 10,000 3d 3d-printing models, 2016.
12

---

Appendices
A Proof of Prop.
Let (u, v) be the parameters of the closest point of p (1 on f(u, v). We have the following constraints on (u, w):
(p- f(u, 8)) f,(u, v) = 0
(10)
(p - u(), (2, = 0
(11)
Note that
f.(u, v) (1, 0, au bb)" f f,, v) = (0, 1, bu + cv)
Ignoring quadratic-and--hiher order terms in u.U.e, y, and in (10) and (11), we have
(x u) + 2 zau be) = 0
(12)
(y v) + z (bu cv) ~ 0
(13)
This leads
u
1 - az
- bz
v
=
bz
1 oz
y
11
+
a
b
y/
2.
(14)
The normal direction (u, u) is
f (u, x f f (u, v)
nw,
If (u, v) x f (u, w)]l
(au + bw), (bu + cr), 1)T
(15)
/1+ + (au + bu) (bu + cu)2
The signed-distance function of p to f(u, v) is given by
d(p, f (u, v)) (p f fu, o))² n(u, v).
(16)
Substituting (15), (14) into (16) and ignoring third-and-higher terms in u, U, y, 2, have
d(p. f (u, v)) x -(x - u)(au + bu) - (y - v)bu + cv)
+ (2
2 (au² + 2buc + co²))
1
= 2 2 (au² + 2buv + cu²)
B Representing Sharp Edges as Quadratic Patches
We consider the intersection of two quadratic patches where the intersection is along the y-axis. In this case, we can define the surface patch f(u, v) (u, v, fl, D))T where
f(u, v) = {
(alu² + 2 + 2b + Eiu
5
(17)
(azu² + cu + 2b2uv) + 6z otherwise
In (17), we do not have any linear term in v, so that the normals to these two patches at (0, 0, 0)7 are in the plane. In addition, the coefficients in front of x² are identical, these patches stitch along w = 0.
The following proposition provides approximation to the SDF function of fu, v).
13

---

Proposition 2 For each point p V.2) " in the neighborhood of the origin the signed distance function from p f(u, B) can be approximated as
(aix² + c12² 2bbbbyy) e1
x $ lazo? + c1y² + 2b2ry) 62 otherwise
d(p, f(, v))
(18)
The proof is very similar to that of Prop. 1. When x 2 0, the parameters (u, v) of the closest point satisty 0, and vice versa. Therefore, the proof applies the description in Section A.
C Local Minima of (3)
We will show that there are nontrivial local minima due to symmetries induced by the rotation group. However, those local minima do not recover the underlying ground-truth shape. As a result, they force the network learn the wrong patterns from the data. For simplicity, we focus on the 2D setting. The extension to 3D is straightforward.
In 2D, assume that the underlying curve is (I, koz²). SDF samples given by (æ, kor² + y. y) where x ~ p. ~ Consider the 2D rigid pose parameters0, ty. ty. Let be the curve parameter. Our goal is to optimize parameters 0, tz: ty. & to minimize the following L reconstruction loss:
r(k, tz. tg. 0) = E E sin( + cos(t) + 4) + ty zmpyyng
k cos (f)r sin(0) + tx)2 v) 2
Clearly, (Ko, 0, 0, 0) is global minimum of The following proposition shows that there is another local minimum of r.
Proposition 3 Suppose and are independent, and
E
z~p
Then (-ko, 0,2c, TT) is a critical point of where = E In addition, it is a local minimum of r if assume
you
E mmpp
r
E x zmp
O,
|z].
We defer the proof of Prop. 3 to Appendix C.I. Prop. 3 shows that there is non-trivial critical point whose parameters depend on the sampling pattern. As neural network training mostly uses first- order methods that can be trapped into critical points, this means that without careful initialization, the network will memorize non-shape-related patterns from data, and significantly impairs the generalization ability of the resulting network.
Proof of Prop. C.1
Denote
Ux, y, k to tyrtyy sin(0) + cos + + ty
k cos(O) - (k) + y) + tx)
is easy to check that
1(x, y, -ko 0, 2c,
2c,T) T) 2c 2y.
(19)
14

---

The first-order gradients of with respect to k, tvs ty. 0 are given by
0l
(z, y, ko. 2e, πT) =²²
(20)
0k
01
(I,, y/ - 0, 0,,,, IT) = -
(21)
01
(z, y/ 0o.0.22 0,2,,, 17
(22)
0ty
0l
(1r U. Ko, 0,20, nt) -x kgg -kooy-
(23)
08
Therefore, we have
Ör
Øl
0k (-ko, 0, 2c, m)
E
(2, y, -u. 0, 2c, x))(1, -ko, O, 2c, T
a-ppy- ak
E E (2c 22y = O,
a-paya
and
( (- ko, 0,2c, = E
a~ppyr Ol, (x, 1, - kn. 0, 2c, (I, -ko, 0, 2c, m)
01.
E (2c 2y) k: 0. p~ppy~
and
dr Ot.
E Øl
0,
(x. y, ku 0, 2,,,, -oo 2c, a))
x--py-q 0t.
E E (2c 2y) 0,
w~ppri
and
h- 0, 0, = E E Øl rt.
0r
(x. y, - ko. 0, 2c, Mx - Ko, 0, 2c, m)
x-ppyrp 0t
E E (2c 2y) (x + x + koxy) = 0.
=
x~ppyー
This means that (- 0, 2c, π) is a critical point r. To show that it indeed a local minimum, we study the second-order derivatives of We begin with the second-order derivatives of 1. They are
021
cr, 9: ko, 0, 2c, T) 2ko + y)² + y - k
020
811
(x, y. - ko. 0, 22,,,.c,0 T) = 2ko + 43)
aldt,
8c1
(x, y. -oo 0, 0,,, 2ko.
02t
and
821
02k (r, - ko, 0, 2c, T) =0
821
(x, y, Ko, 0, 2c, T) 2x
8P1
(r, ko, 0,2c, 1) = 2xr(ko0² + +z)
0k00
and
01
021
(I, 8. - ko. 0, 2c, T) =
(x, - -ko, 0, 0c, x) 0
02t
at Ōk
821
821
(x, -ko, 0, 2c, T
(x, v. - Ko. 0, 2c, T 0 00,
01
15

---

Note that Ya, B [k, tz. ty. f).
8+,
(-ku, 0, 2,, x)
dad5
012
E E p>ppy~n
[(I, y, -ko, 0, 2c, T)
(z, y. 1ko-ko, 0, 2c, T)
dad3
01
01
(x, y. - ko. 0, 2c, πT)
(x, y. - No. 0, 2c, T)
aa
03
Denote
V
E x²
V
E=
zmp
gru
We have
02r
( ( 0, 2c, 1t) V22 2koc(V + 2V,) + (2 (2+ V
026
-4hi
V ạ (2V V2 - 24 + 1/
02r
(- Ko: 0, 2c, π) = ko (V22 + V + kocV? + +(e² - V29)
0001
80r ( ko: 0, 22, m) = k2
021
and
@r
(-ko. 2c, 1t) Vid
8h
(-kg. 0, 2c, k) = -o = 0
0kOtz
82r
(-ko. 0, 2c, Rt) Vi + koc) + +gV.
0k:0
and
02r (-ko, 0, 20, π) = 1,
Pt, y
Pr
(-ko, (-ko,0,2c, 2c, = -V2
Pr
(- ko, 0,20, ④)) 0,
Öt
8r
(-kg,
Ko. 0, 2c, x)
0t,
remains show that
8r. (- No. 0,2c, ④)
Pr
Pr.
028
gtt
0,2c,))
(000t.
(-ho, 0, 2c, m)
(24)
and
82T
8r (
02T akit y
kg. 0, 2c, T)
ko, 0, 2c, m)
(- O, 2c, 2))
(25)
22h
g2t
16

---

The difference between the left and right-hand
hand sides of (24)
(v2 2koc(V + +2+ 2
-V - - V
+ Ag(221 + V222) + 2 kgV.) - (V2 + +1zv2)²
(kocV? + +4(c² v;,)) - 2(2 + +/y:2' (NocV? + -4(c² - V29))
0g (Mo - 1/2 ( (V2(cc4c333 - 26
(6 (6 (1,+5V220 5V -c²) - -4k-y
2k) 16(c² )2
As < I, the above quantity is above zero if
V2ve V2
which can be derived from Cauchy inequality. (25) is equivalent to
V22 V2²
which can be derived from the Cauchy inequality.
D More Results
We include more visualization comparisons. show the comparison with generalizable methods and scene-specific methods in Fig. 6 and Fig. 7, respectively. We also include a failure case CoFie in Fig.
Visualization
Quantitative Results. We include more comprehensive comparison with generalizable shape auto-encoding (GAE) and shape-specific outo-decoding (SSAD) methods for understanding the performance of our model. Again, we note CoFie performs generalizable shape auto-decoding (GAD) and is not directly comparable to these models.
Table 5: Performance on ShapeNet 10 novel cat- egories. Specifically, the reported 3DS2VS [54] and NKSR [15] are trained on the full set of the training categories.
Table 6: Performance on Thingi shapes. Note that SSAD methods take long time for infer- ence, e.g. NGLOD and UODFs take 105 and 300 minutes, respectively. contrast, CoFie takes 10 minutes.
contrast, the reported nu- mebrs in the main paper use subset of 1000 instances for training.
Setting GAD DeepSDF
Method
CD (le-4)
gloU
Setting
Method
CD
gloU.
10.4
83.1
GAD
DeepSDF
9.79
87.1
GAD
DeepLS
7.27
96.2
GAD
DeepI.S
3.68
97.4
GAD
CoFic
3.18
98.3
GAD
Colie
1.87
99.0
GAE
3DS2VS (full set)
9.30
94.8
SSAD
NGLOD
1.04
99.3
GAE
NKSR (full set)
4.24
96.9
SSAD
UODFs
0.932
99.4
17

---

DeepSDF
DeepLS
Ours
GT
Figure 6: Compare with the generalizable methods DeepSDF and DeepLS on ShapeNet shapes. show two images for each method, one for the overall shape quality, and a zoom-in detail check.
18

---

Ours
NGLOD
GT
Figure 7: Compare with the shape-specific method NGLOD on Thingi shapes. We show two images for each method, one for the overall shape quality, and a zoom-in detail check.
Ours (2 Views)
GT (2 Views)
Figure 8: Analysis the failure case. CoFie still struggles to represent extremely detailed geometry parts.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We include the experiments to validate every claim.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see the last page of the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
We have included the theoretical analysis in the supplementary.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have included all experiment details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification:
We committed to releasing code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wwtpccccpppcccaeccccccccccccctlt S ssisssslicyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We have included all experiment details.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our method outperforms the baselines with significant margins. Visualization also verifies the contributions.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
22

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have included the details in the experiment details section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : http///eee.../eeeeee cc. httpp//tcccccpcppicc//////c///// cc/pppllittttthhhttiiieeeeeeee ceehhicsCuieelneee ines?
Answer: [Yes]
Justification: Our paper does not contain any harmful results. It is neutral research.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the broader impacts on the last page of the paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our paper on shape representation does not have the risk of being misused. Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided relcasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
ShapeNet: Custom (non-commnn commmcommeraal)
Thingi: CC BY-NC 2.0 license
DeepSDF: MIT License
3DLatent2 Custom (non-commercial)
DeepLS: MIT License
NGLOD: MIT License
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We will release the code with documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "70Vd0upa8z", "reviewer": "Reviewer_LpYZ", "summary": "The paper proposes a local prior based method where the model is trained on a dataset and learns a prior over local patches of the shapes, and then at test time reconstructs patches by optimising to minimise the reconstruction error w.r.t. the trained model's prior. While this has already been done in the literature, they propose to learn a coordinate transformation for each local region and initialise the coordinate transformation based on the geometry in that region. They also use a quadratic layer at the end of the network, inspired by their analysis of representing local patches by a quadratic surface.", "questions": "- The method is called coordinate field, but it is not actually a true coordinate field: the coordinate transformation is learned discretely as parameters for each voxel, not continuously throughout the domain?\n- Usually shape autodecoding uses regularization on the latent parameters so then when doing test time latent optimisation the latent parameters can be initialized to zero and be \"close\" to many possible shapes. It seems the proposed method has no regularization, is there a reason for this?\n- Line 219: \"compute the derivatives of the SDF\" however at test time you don't have access to the full SDF (or do you?), so how to you get well oriented normals for initialization?\n- Would like more information about what SDF supervision exactly is given at test time. Is it just the surface points, or is assumed the SDF is known at test time as well?\n- Would like timing performance of test time optimization (general ballpark and if this compares to other methods)\n- Line 243: two types of methods or three types of methods?\n- Missing ref on line 114, line 128 and line 197", "limitations": "- Yes, limitations have been discussed adequately.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Good ablation study showing the benefits of each component. Interestingly the geometric initialization is very important for performance improvement, suggesting that the latent based model has trouble training efficiently.", "weaknesses": "- A lot of things are not made clear (asked in the questions) which makes it hard to understand the method\n- As mentioned in the strengths, it seems that the benefit of this method is that the coordinate field (and especially its init) it enables the latent based model to train better, rather than being a better representation. This could be explored more deeply. Also it makes the question below (\"It seems the proposed method has no regularization, is there a reason for this?\")", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the positive comments and your acknowledgment of the thoroughness of our ablation study. We will address your comments as follows.\n\n---\n\n**[W2, Benefit of Coordinate Field]** We kindly note that the coordinate field is a part of our representation, as the coordinate field is involved in the computation of decoding shape surfaces. It makes the learning more efficient, improving the geometry modeling capability, and helping us learn the MLP in a more compressed local patch geometry space.\n\n---\n\n**[Q1, Coordinate field]** A field is defined as follows in Wikipedia: “In science, a field is a physical quantity, represented by a scalar, vector, or tensor, that has a value for each point in space and time.” Our coordinate field is a discrete field, where the coordinate frame value of a 3D point is associated with the local patch it belongs to. We note that this discretization is the key to defining local patches and enabling the geometry-aware coordinate field and its initialization.\n\n---\n\n**[Q2, Regularization]** Thanks for pointing this out. We use the same regularization as previous works that penalize the latent code values, following the standard regularization term in DeepSDF and DeepLS. We didn’t include the term in the loss function as it is standard. We will revise the text for clarity.\n\n---\n\n**[Q3, Well-oriented normal]** Great question! We clarify that at test time, we don’t have full SDF but SDF point samples (following DeepSDF, DeepLS, NGLOD, etc), and the estimated normal can be imperfect. Thus, as we mentioned in equation 9, we optimize the latent code and the coordinate frame jointly, which makes our model more robust to noise in the estimated normals. \n\n---\n\n**[Q4, Supervision at test time]** During testing, we follow practices from DeepSDF, DeepLS, and NGLOD. We sample points (with known SDF values), including points close to the surface, which is a standard point sampling method for shape auto-decoding. Thus, the points are not on-surface points, but also we don’t have access to the full SDF (we only have point samples of the SDF). In evaluation, we compare with DeepSDF, DeepLS, and NGLOD inference on the same data with our method.\n\n---\n\n**[Q5, Inference time]** We achieve an inference time of 10 minutes in the setting of *optimizing a single shape*, which is much faster than the baseline NGLOD (105 min). Moreover, our method benefits from the capability of *inference on a batch of shapes* by sampling batched local shapes from all of them. This property further improves the efficiency of our method. NGLOD does not support batched inference since it optimizes a non-regular grid which is different for each test shape.\n\n---\n\n**[Q6, L243]** Thank you for catching that! We meant three types of methods: 1) generalizable shape auto-decoding methods, 2) shape-specific shape auto-decoding methods, and 3) generalizable shape auto-encoding methods, which are used as a reference and are not directly comparable (as discussed in L247). We will clarify this in the revision.\n\n---\n\n**[Q7, missing references]** Thanks for the detailed feedback. These references include previously mentioned works including [22,38,8,34,30]. We will revise the references in revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "raW2IrHvhg", "reviewer": "Reviewer_V5fn", "summary": "Paper proposes CoFie — 3D shape representation as a set of latents arranged on a regular voxel grid. Each latent encodes an oriented local quadratic patch. This local oriented patch defines local SDF in a local coordinate frame which is decoded via conditional MLP for which the last layer is quadratic: it defines a bilnear form on input vector instead of linear mapping). SDF value for query point x is decided based on the SDF value of local implicit function belonging to the nearest voxel and transformed from local coordinate frame to global. \n\nAuthors train proposed methods and baselines on a single random subsample of 1000 ShapeNet shapes (5 categories, 200 shapes each) and evaluate on holdout ShapeNet shapes and out-of-distribution shapes from Thingi10K. Proposed method outperforms baselines trained in the same limited data regime. The same ShapeNet model is also evaluated with the Neural Geometric Level of Detail method and DeepSDF fitted in the shape overfitting scenario.", "questions": "— What is the input for the model for auto-encoding settings? Is it a point cloud? Or model back propagates into learnable latent code like DeepSDF? If it is a point cloud, what encoder model is used? If model back propagates into latent code, how come it is generalizable?\n\n— How were baselines trained? Were they trained on the same inputs as the proposed method? Did all methods use the same SDF sampling? As far as I know, 3D2VS uses occupancy as supervision. Was it trained with SDF or occupancy supervision for the paper? \n\n— What motivated size 1000 for a training dataset? Majority of modern shape auto-encoder train either on full ShapeNet of full ShapeNet categories (e.g. ~6000 ShapeNet chairs). If 5 different subsets of ShapeNet of size 1000 would be selected (same categories), how high would be the variance in evaluation across these subsets? I have strong suspicion that results shown int Tables 1 and Tables 2 might not generalize across training subsamples. \n\n— Have you tried using Fourier embeddings of input coordinates instead of quadratic layers? This might be a very strong alternative since it helps coordinate based models to encode high-frequency details very efficiently.", "limitations": "— Proposed method represents the shape as a set of latents arranged on a regular latent grid and thus scales cubically with latent grid resolution. This limits method’s ability to efficiently encode topologically challenging shapes. \n\n— Method requires a lot of compute to fit 1000 shapes (1 day on 4GPUs with 24GB VRAM). This might be due to fact that authors use 32^3 latent grid (SDFusion for example uses 8^3). This severely limits scalability of the proposed method even for training on full ShapeNet, not to mention larger datasets like Objaverse.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "— Proposed representation is more efficient compared to regular latent grids and is comparable to surface latent grids (3DILG) in terms of latents needed to represent shape; \n— Method figure is clear and well done and quantitative results are well formatted;\n— Qualitative results look comparable to NGLOD which is a strong baseline for shape overfitting setting;\n— Design choices are clearly ablated (Table 4);", "weaknesses": "— Proposed method seems to be very similar to AutoSDF and SDFusion: both methods utilize a regular latent grid that encodes local SDF that is used to infer the global SDF. Main difference seems to be in a local SDF decoder that is quadratic (last layer) for proposed method and also encodes SDF in the local coordinate frame (SDFusion and AutoSDF use global coordinates). Overall, it is fine as long as proposed method compares to these similar methods but evaluation of the paper is limited (see below). \n\n— Similarly to other regular latent grids methods (e.g. SDFusion, AutoSDF), this model scales cubically with respect to grid resolution and thus might not be well suited for representation of topologically challenging shapes with thin parts. \n\n— Evaluation of the paper is extremely limited. For point cloud reconstruction (auto-encoding), proposed methods and baselines are trained on 5 ShapeNet categories with 200 shapes per category (overall 1000 shapes). This evaluation is not enough to support the claim that CoFie “can represent arbitrary shapes that belong to any novel category” (LL52-53). These results might only indicate that CoFie might be better representation in low-data regime but in this case evaluation should be done similar to few-shot learning setting: all models should be trained on several different subsets of ShapeNet (e.g. 5) and average/std of test errors should be reported. I also want to note that 3D2VS trained on full ShapeNet is an extremely strong baseline that achieves almost perfect reconstruction quality for some categories (like airplanes). Also, the paper only uses Chamfer distance for evaluation and ignores other common measures like IoU and surface F-Score. \n\n\n— The choice of baselines is also limited. DeepSDF is a 2019 paper that uses a very simple Pointnet (not even PointNet++) encoder. DeepLS is a 2020 paper. 3D2VS seems to be the only recent baseline for 3D point cloud reconstruction. It is a very strong baseline but authors have not used pretrained models and trained their version on limited data instead. Given the fact that 3D2VS uses a very high capacity attention-based decoder, this training regime might not be a fair comparison because attention-based models often struggle in low data regimes. For the same reason, 3D2VS might not be a good baseline if the goal of the paper is to show that their representation is compact. In this case, 3DILG might be a more suitable and strong baseline (see below) because it uses a simpler local patch encoder (Pointnet), so it might work better in a limited data regime. Since the model utilizes local surface patches similarly to AtlasNet (see links below), it also can be a strong baseline, especially in low data setting. \n\n— Evaluation in an overfitting setting  does not seem methodologically correct. : NGLOD and DeepSDF overfit to single shapes while CoFie was pretrained trained on 1000 shapes. These are completely different settings: overfitting regime measures capability of the model to efficiently fit one shape with preservation of geometric detail, and fitting and evaluation on collection of shapes evaluates the ability of the model to generalize to unseen shapes. This experiment should either be done if fully overfitting setting (similar to NGLOD paper) or in a reconstruction setting (see concerns above). \n\n— Quadratic MLP contribution seems a little bit weak to me. It looks like it is basically equivalent to linear MLP being run on quadratic feature expansion and in this case commonly fourier embeddings of points can be a very strong alternative (this is not tested). \n\n— Paper writing can be significantly improved in clarity. For example, paper does not specify input to auto-encoder for baselines. Is it point cloud or mesh? If it is a point cloud, what is the density of sampling and what sampling was used (FPS, random, poisson disk sampling, etc). How many codes were used to train 3D2VS? 512? What were hyperparameters for baseline training? On the other hand, paper spends a lot of space describing relatively common knowledge like MLP with ReLU activations (LL172-177). Another example is equation (5). On first glance it appears as weighted average, but since authors use voxel indicator function as weight, it means that for each query point x, global implicit function is based only on implicit function of nearest voxel grid latent. \n\n— Some statements in the paper are not correct. For example, the paper states that 3D2VS “employs transformers to predict the shape latent code” (LL258-260) . This is not factually correct: 3D2VS does not represent shape as one vector, it uses latent code clouds (usually 512 codes per shape). \n\n— Representation seems to be computationally expensive. Training on 1000 shapes takes one day on 4 GPUs with 24 GB VRAM. \n\n— Some relevant work is missing\n\nAutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation https://arxiv.org/abs/2203.09516 \n\nSDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation\nhttps://arxiv.org/abs/2212.04493 \n\n4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks https://arxiv.org/abs/1904.08755 \n\n3DILG: Irregular Latent Grids for 3D Generative Modeling https://arxiv.org/abs/2205.13914 \n\n\nAtlasNet: A Papier-Mâché Approach to Learning 3D Surface Generation https://arxiv.org/abs/1802.05384", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed feedback, and your acknowledgement of the efficiency of our model, the clarity of our writing and figure, and our clear ablation study. We will address your comments and some **factual errors** in your comments as follows.\n\n---\n\n**[W1, AutoSDF and SDFusion]** The mentioned papers are significantly different from our work. First, AutoSDF and SDFusion have a different target from ours. In detail, AutoSDF and SDFusion target to learn the latent space of SDF represented shape, which enables them to perform downstream applications of shapes that belong to specific categories, e.g., chairs and airplanes. In contrast, our work focuses on shape representation, where we prove that using coordinate fields with better MLP designs can represent the shape of general categories better with both experimental and theoretical evidence. Moreover, AutoSDF and SDFusion use shape auto-encoding to learn the latent space. Differently, our paper works on shape auto-decoding (as mentioned in L189 and L243), or termed as auto-decoder in DeepSDF and DeepLS paper. As discussed in L248, shape auto-encoding works can not be directly compared with our method. **At the same time, we agree with your comment that we both use grid-based SDF. However, this is very common and standard in related research and we have discussed them in Sec.2, including [4,21,36,19,32,35,30]. We never claim that using the grid-based SDF is one of our novelties.** We are happy to add the discussion of the mentioned paper in the revision for more clarity.\n\n---\n\n**[W2, Complexity]** **Our method doesn’t scale with cubical complexity.** As illustrated in Fig 2, our representation only focuses on grids that contain the shape surface. Thus, we actually use **sparse voxels**, and those voxels that don't contain surfaces are discarded. When using the resolution of 32x32x32, only 6% percent of the grids are valid. When using the resolution of 64x64x64, only 1.2% of voxels are valid, where the complexity is only 1.6 times of resolution 32 and is far less than a cubical factor of 8. This property is similar to DeepLS.\n\n---\n\n**[W3, Evaluation]** The effectiveness of our evaluation is acknowledged by other reviewers, including “convincing evaluation” (Kq6M), “good ablation study” (LpYZ), “Experimental results demonstrate strong generalization capability of CoFie, outperforming previous methods in terms of accuracy and efficiency”, and “the results on two datasets show great potential” (58nR).\n\nTo further address your comments, first, we would kindly remind you that CoFie works on the shape auto-decoding task (optimization-based) as we mentioned in L189, L198, L247, and L249. CoFie doesn’t work on the auto-encoding task (feed-forward-based) you mentioned.\nSecond, regarding our claim of “represent arbitrary shape”, we have verified the claim in our experiment in Sec 5, and it is appreciated by other reviewers, including “strong generalization capability” (4hsy), “outperform previous generalizable methods” (Kq6M), and “show great performance on both synthetic and real data” (58nR). As we mentioned in L277, we train the model on 5 ShapeNet categories, and we test the model on 10 novel ShapeNet categories and also novel real objects from novel categories. Results in Table 2 and Table 3 are comparable to Table 1, demonstrating the strong generalization capability. Conceptually, our method doesn’t have any category-specific designs, and the local patch-based method is naturally generalizable to any shape [4].\n\nFinally, we would like to address your concerns regarding the dataset size. First, using a thousand instances for training is typical in previous works [24,4]. We strictly follow the setting in DeepLS [4]. Second, to further address your concern, we report the result of 3DShape2Vec using the full set of the 5 categories in Table 1 below. We alternatively mention that 3DS2VS is a shape auto-encoding method and it is not directly comparable to shape auto-decoding methods including DeepSDF, DeepLS, NGLOD, and our method. We have mentioned this in L248. Shape auto-decoding generally has a much better performance than shape auto-encoding. With more data, the shape representation accuracy of 3DS2VS is still not as good as shape auto-decoding methods, especially in novel categories.\n\n---\n\n**[W4, baselines]** We respectfully disagree with your points. **You mentioned that DeepSDF uses a PointNet encoder, but actually there is no encoder in DeepSDF as it performs shape auto-decoding (or termed as auto-decoder). By searching “PointNet” in the DeepSDF paper, we don’t find any matches in the method section.** Please let us know if there is any following work that modifies DeepSDF with a PointNet encoder. This is also true for DeepLS. The mentioned 3DILG paper works on shape auto-encoding, and actually the SoTA of shape auto-encoding is 3DS2VS and the performance of 3DS2VS is reported in Table 3 of 3DS2VS paper, which outperforms 3DILG. Thus, we didn’t compare with 3DILG. Again, we want to emphasize that shape auto-decoding methods and auto-encoding methods are not directly comparable. We would be happy to compare with more shape auto-decoding methods if we missed any.\n\n|     | 3DS2VS (1000 shapes) | 3DS2VS (full set) |  Ours  |\n|:---:|:-----------:|:-------:|:------:|\n| CD  |    16.0     |  9.3 | **3.2** |\n\n--- \n\nWe will continue to address the question in the official comment reply.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pv49UOYY99", "reviewer": "Reviewer_4hsy", "summary": "This paper introduces CoFie, a novel neural surface representation method designed to efficiently learn and represent complex 3D shapes. CoFie addresses the challenges of existing methods by introducing a coordinate field that optimizes the representation of local shapes, significantly reducing shape errors and improving the efficiency of parameter usage. The method is based on a hierarchical representation with coarse and fine-grained geometry, utilizing MLP with quadratic layers to enhance the expressiveness of the model. Experimental results demonstrate CoFie's strong generalization capabilities in shape reconstruction tasks, outperforming previous methods in terms of accuracy and efficiency.", "questions": "I noticed that the article mentions, \"We train CoFie on 1000 shape instances sampled from ShapeNet [5] consisting of chairs, planes, tables, lamps, and sofas (200 instances for each category).\" Why not train on the entire ShapeNet?", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "CoFie uses a coordinate field to transform local shapes into an aligned coordinate system, reducing spatial complexity and making the learning of MLP-based implicit representations more efficient. By incorporating quadratic layers into the MLP, CoFie enhances the model's ability to capture local shape geometry, improving the quality of shape modeling. CoFie is trained on a curated dataset and can represent arbitrary shapes from novel categories, demonstrating strong generalization capabilities.\nThe theoretical proofs in the paper are sufficient and the articulation is clear.", "weaknesses": "1.CoFie's performance in detailing is strongly related to the cell size, which hinders its ability to represent fine-grained details. \n2.Representations based on nonoverlapping local patches may not favor expressing rich colors, a capability that implicit representations possess. \n3.It excels in displaying regular shapes, such as furniture, but is weak in showcasing shapes with rich curves.\n4. Line 197, citation is missing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive comments, and your acknowledgment that our theoretical proof is sufficient and the articulation is clear. We will address your comments as follows.\n\n---\n\n**[W1, grid size]** Our method has demonstrated good performance in modeling detailed geometry as shown in Figure 3, Figure 6, and Figure 7, including thin structures, non-smooth local surfaces, as well as edges and corners. Alternatively, we note that our method can be easily extended to high resolution without increasing the complexity a lot. Our method works on valid voxels that intersect with the shape surfaces, which represent only 6% of all voxels when using a resolution of 32x32x32. This property makes our method efficient. When extending to a resolution of 64x64x64, only 1.2% of voxels are valid, where the complexity is only 1.6 times of resolution 32 and is far less than a cubical factor of 8.\n\n---\n\n**[W2, Representing color]** Since we follow the common setting in implicit representation works [4,24,30,31,29,3,27,49,37], representing color is currently out of the scope of our work. Moreover, as shown in Figure 3, Figure 6, and Figure 7, our method can represent the local shape boundary between voxels accurately. This is strong evidence that it can also work for color. Moreover, as mentioned in L211, we improve the smoothness of the surface across different voxels by expanding the \"receptive filed\" if each voxel. This can also contribute to better color modeling capability.\n\n---\n\n**[W3, Representing rich curves]** We kindly note that the reported number on Thingi is from the model trained on ShapeNet. Thus, its comparable performance with the shape-specific NGLOD demonstrates the power of CoFie on complex shapes.\n\n---\n\n**[W4, Reference]** Thanks for the detailed feedback. We will revise the reference.\n\n---\n\n**[Q1, Training set]** Training on 1000 shapes of ShapeNet is a common setting in shape auto-decoding works. We strictly follow the setting in DeepLS for using 1000 shapes. Please see more details in the experimental setup section of the DeepLS paper. Besides, we want to clarify that the 1000 shapes contain 3 million local surface samples, which is sufficient for training. This property demonstrates the data-efficient nature of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hcCG3cH0u5", "reviewer": "Reviewer_Kq6M", "summary": "### Motivation\n- In the realm of implicit 3D shape representations, local-based solutions [4, 19, 32] (which decomposes the target shape into a set of local surfaces to model) provide higher accuracy but at the cost of a higher number of parameters to optimize. \n- The authors argues that this number of parameters can be reduced by disentangling the actual geometry and the transformation (orientation, translation) of these local patches, modeling the latter separately.\n\n### Contributions\n- The authors thus introduce CoFie, a novel approach that utilizes an explicit coordinate field to transform local shapes, aligning them to reduce spatial complexity and improving MLP-based implicit local representation of 3D shapes.\n- Various theoretical and empirical evidences are provided w.r.t. the benefits of their formulation, as well as to justify other technical choices, e.g., the usage of quadratic layers to complement their MLPs.\n\n### Results\n- Experiments show that CoFie reduces the shape error by 48% to 56% compared to traditional generalizable methods (e.g., DeepSF [4], DeepLS [24]) and achieves comparable performance to prior works with 70% fewer parameters.\n\n### Relevance\n- This paper addresses the challenge of balancing accuracy and compactness in neural surface representations, a relevant and generic issue in 3D shape modeling.", "questions": "_see **Weaknesses** for key questions/suggestions._\n\n### Q1. Computational Cost\n- Besides the points made in **W2** above, I wonder what the computational cost of replacing linear layers with quadratic ones is (c.f. [L178-187])?\n\n### Q2. Grid Size and Surface Complexity\n- What is the impact of the voxel grid size on the surface complexity / ability of the model to learn the surface? Intuitively, the larger each voxel is, the more complex/discontinuous the surface that it intersects can become.\n\n### Q3. Purpose of $w$\n- What is the point of introducing $w$ [L168-171] if in practice it is set to the indicator function (simply selecting the voxel $v$ where $x$ belongs)? Is there a point to keeping Equation 5 this generic?\n\n### Q4. Hyper-parameter $k$\n- The authors mention that setting the number $k$ of quadratic layers (following $L-k$ linear ones) to 1 yields the best performance [L187]. Could the authors share the quantitative evaluation behind this conclusion?", "limitations": "- Limitations are discussed in the conclusion, with the authors sharing meaningful insight (e.g., the solution cannot perform shape completion or handle thin structures well).\n- Societal impact are however not explicitly discussed. Providing some potential positive or negative applications of this work would be meaningful.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "_(somewhat ordered from most to least important)_\n\n### S1. Extensive Theoretical Grounding and Methodological Explanations\n- The authors spent much effort formalizing and proving their contributions, providing the readers with extensive theoretical background.\n- The Methodology itself is clear and thoroughly explained. While the actual implementation has not been provided, an expert in the art should be able to re-implement this work.\n- The paper is also well illustrated and easy to follow (pipeline figures + qualitative results).\n\n### S2. Novelty.\n- To the best of my knowledge, the idea of disentangling geometry and transformation of local shapes is both new and interesting, and could benefit the 3DCV community.\n- The authors nicely tie together their original intuition and extensive formalization to justify their contribution.\n\n### S3. Convincing Evaluation\n- The proposed CoFie method outperforms other generalizable methods in terms of shape accuracy and weight compactness.\n- Qualitative results and the ablation study are also convincing.", "weaknesses": "### W1. Somewhat Limited Comparison to SOTA\n- The authors compare to 4 methods (DeepSDF [24], DeepLS [4], NGLOD [30], 3DS2VS [41]), 2 of which are fairly old for the domain (<2020 for [24] and [4]). The literature has many more recent approaches, e.g. [a,b,c], with code or results pre-available, which could have been considered.\n- Moreover, the authors compare to only one shape-specific method (i.e., performing test-time optimization), NGLOD [30]. While such methods tackle a slightly different task, comparing to a single solution makes it hard to draw solid conclusions.\n\n### W2. Partial Evaluation of \"Compactness\" Claims\n- The authors claim that \"CoFie achieves comparable results with prior work using 70% less parameters\" [L56-57]. However, there is a distinction to be made between parameter number (compactness of the model), convergence speed (number of iterations to convergence), and inference time / computational footprint. I find that claiming some improvement along one of those dimensions without studying the impact on the others not so meaningful. \n\n### W3. Code Not Available\n- While the authors claim that they are \"committed to releasing our code\" [L16], no code has been provided yet to reviewers.\n\n\n### W4. Minor Remarks\n- Citations are missing (empty brackets) in several places [L114, L128, L197].\n- Typos: \"practival: $\\rightarrow$ \"practical\" [L118] ; \"Perfomance\" $\\rightarrow$ \"Performance\" [L270] ; etc.\n\n\n------------\n### Additional References:\n\n[a] Ye, Jianglong, et al. \"Gifs: Neural implicit function for general shape representation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[b] Wang, Li, et al. \"Hsdf: Hybrid sign and distance field for modeling surfaces with arbitrary topologies.\" Advances in Neural Information Processing Systems 35 (2022): 32172-32185.\n\n[c] Lu, Yujie, et al. \"Unsigned Orthogonal Distance Fields: An Accurate Neural Implicit Representation for Diverse 3D Shapes.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and positive comments. Thank you for acknowledging the novelty, the theoretical foundation, and the good performance of our work. We will address your comments as follows.\n\n---\n\n**[W1, Baselines]** Although we understand your concern, we cannot find more directly comparable works. For example, GIFS, HSDF, and UODFs work on modeling non-watertight surfaces based on UDF. The input of GIFS and HSDF is the point cloud, which follows the setting of OccNet. However, the input of our method is SDF samples, where the points can be not on-surface. This follows the setting in DeepSDF, DeepLS and NGLOD. The mentioned UODFs work is a shape-specific method, which is directly comparable to NGLOD, but both NGLOD and UODFs are **different from our setting of generalizable shape representation as we noted in L244**. To further address your concerns, we report the performance of UODFs as follows on the Thingi mesh ID 252119. In general, UODFs demonstrate better performance than NGLOD. Our method, which takes much less time for inference and is not shape-specific, demonstrates comparable results.\n\n|     | NGLOD | UODFs |  Ours  |\n|:---:|:-----------:|:-------:|:------:|\n| CD  |    0.932    |  0.768  | 1.68 |\n| Inference Time | 105 min | 303 min | 10 min |\n\n---\n\n**[W2, Compactness]** Thanks for your suggestion. Except for the number of parameters, our method also demonstrates a faster convergence speed over the directly comparable baseline DeepLS. When using the latent code size of 125, our method achieves comparable performance with DeepLS using only 61% training iterations. During inference, we set the same number of iterations for our method and DeepLS, where the inference time is roughly the same.\n\n---\n\n**[W3, Code release]** We are firm supporters of open source. We commit to releasing the code right after the decision results.\n\n---\n\n**[W4, Minors]** Thanks for the detailed feedback. We will incorporate them in the revision.\n\n---\n\n**[Q1, Computational cost]** The quadratic layer makes the training/inference speed 14% slower. The used GPU memory increases by 3%.\n\n---\n\n**[Q2, Grid size]** We experiment with a grid size of 16x16x16. The model shows a 4.96 chamfer distance error (according to Table 1 in the paper). We agree that the grid size influences the performance. In the meantime, we want to clarify that using a large grid resolution doesn’t increase the complexity of our model significantly. Our method works on valid voxels that intersect with the shape surfaces, which represent only 6% of all voxels when using a resolution of 32x32x32. This property makes our method efficient. When extending to a resolution of 64x64x64, only 1.2% of voxels are valid, where the complexity is only 1.6 times of resolution 32 and is far less than a cubical factor of 8.\n\n---\n\n**[Q3, Aggregation function]** This function is a general representation of blending the local shapes into the global geometry. Using the indicator function is a special case of it, and is common in implementations, including DeepLS. Our method can also be extended to other blending methods, however, the blending method is not the focus and contribution of this paper.\n\n---\n\n**[Q4, Hyper-parameter]** In our experiment, we also tested using 2 quadratic layers and the experiment showed a 2.20 error, which is larger than using one layer (according to Table 1 in the paper).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "68aVC64Y5X", "reviewer": "Reviewer_58nR", "summary": "This paper proposes a new architecture for SDF auto-decoding task. It divides the whole SDF points into voxels and builds local coordinates for each surface patch included in valid voxels. Respective shape coding is learned separately and a generalizable MLP is used for SDF decoding. For MLP, a quadratic layer is proposed to better fit SDF values. Compared to the selected baselines, the proposed method shows great performance on both synthetic and real datasets.", "questions": "Besides the weaknesses, there is one more question: currently all points in one voxel are assumed to be related to the surface within the same voxel, what if the points near the boundary of the voxel are related to the surface included in the nearby voxel?", "limitations": "Refer to weaknesses and questions.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper theoretically proves that ReLU based MLP is not enough for modelling the quadratic surface and thoroughly explains the difficulty in training local coordinate transformation. The results on two datasets show that local shape coding and hybrid representations have good potential for shape decoding.", "weaknesses": "The evaluation on the main assertions of this paper is not thorough. The method should be evaluated in three levels – accuracy, scalability and generalizability. \n\n1. The first one has been partially demonstrated by comparing with the selected baselines. However, here a SOTA baseline - Neural Kernel Surface Reconstruction (CVPR’23) - is missing.\n \n2. Scalability is not studied by the paper. Only object level datasets are tried. Large scale datasets such as CARLA should also be evaluated to demonstrate the ability of the model. This is a reasonable experiment for a locally coded model, because it only needs to introduce more and more patches for larger cases. \n\n3. Generalizability is one of the selling points for this paper. However, there are some issues related to the baselines and experiment settings. \n\nFirst of all, the paper states that the baseline NGLOD is a per-scene model, not a generalizable model, so NGLOD naturally could have better performance. However, NGLOD also has generalization ability according to the original paper, only LOD features are trained per-scene while keeping the MLP fixed, which is actually a similar setting to this paper. \n\nSecondly, there are two kinds of generalizability: across domain and across density. In this paper only generalizability across domain, i.e. object level synthetic to real, is tried. Scene level, for example ShapeNet and Synthetic Room to ScanNet or Matterport3D, and generalization across density, that is trained and inferred on different sampling density, are missing.\n\n4. Besides the experiments, there is one problem in theory derivation: \n\nIn the proof of proposition 1, equation 14 seems unreasonable to be equal. It is suggested to explain to what extend these two expressions could be regarded as equal. If equation 14 is not equal, how the final equation could be held?\n\n5. Missing reference(s): Line 114, missing reference(s) for using quadratic surface patch", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed feedback, your acknowledgment of our theoretical proofs of fitting quadratic local patches, and the potential of our method. We will address your concerns as follows.\n\n---\n\n**[W1, Baseline]** Thank you for pointing us to NKSR. However, **NKSR is not a directly comparable method to ours**. As we mentioned in L189, L198, L247, and L249, our method works on the shape auto-decoding setting (the shape auto-decoder defined in DeepSDF), but NKSR works on shape auto-encoding. The differences are i) our method takes SDF samples as inputs, which is the same for DeepSDF and DeepLS, while the input of NKSR is the on-surface point cloud with normal as input; ii) After the shape decoder is trained, we fix it and perform back propagation to optimize the shape latent code, which is the same for DeepSDF, DeepLS and NGLOD, while NKSR uses feed-forward network to get the shape representation. Alternatively, as we mentioned in L248, even though the two settings are not directly comparable, we experimented with 3DS2VS, a SoTA shape auto-encoding method, as a reference method. To further address your concerns, we experiment with the mentioned NKSR method. We use 3K points as input without noise and use the full set of the 5 training ShapeNet categories for training; we test on the novel categories. We report the results on novel ShapeNet categories as follows. We note we also use the full set of 5 training ShapeNet categories for training 3DS2VS, while CoFie is trained on only 1000 shapes.\n\n|     | 3DS2VS | NKSR |  Ours  |\n|:---:|:-----------:|:-------:|:------:|\n| CD (1e-4)  |    9.32    |  4.24  | 3.18 |\n\n---\n\n**[W2, Scalability]** Thanks for the valuable suggestions. We agree that CoFie can be extended to scene-level shapes as it uses local shapes. However, most of the scene meshes are not watertight, where we cannot use the point sampling method of DeepSDF to get the data. DeepLS paper provided a method to fake the input by using on-surface points and surface normal. However, their scene-level code is not released. We are unable to re-implement the scene-level data sampling method during the short rebuttal period, and we are committed to adding the result in the next version.\n\n---\n\n**[W3.1, Generalization claim]** We thank you for the detailed feedback on generalization. However, the generalization in the NGLOD paper (reported in their Table 4) has a **different definition** from ours, even though we use the same terminology. As discussed at the beginning of Sec 4.3 of the NGLOD paper, their generalization works on the setting of i) training a model on **a single shape**, ii) applying the trained model to novel shapes, according to their description of “We now show that our surface extraction mechanism can generalize to multiple shapes, even from being trained on a single shape.” However, the generalization defined in our setting is i) training on **a set of shapes**, ii) applying the trained model to novel shapes, which is the setting in DeepSDF, DeepLS, and our work. We alternatively note that NGLOD doesn’t support training on multiple shapes. To further address your concern, we report the performance of NGLOD on Thingi by training the MLP on a ShapeNet Chair (denoted as NGLOD pre-trained). The performance is significantly worse than ours according to Table 3 in our paper. The reason is that the local shape distribution of the training shape and testing shape are very different. In contrast, our model can be trained on multiple shapes (1000 shapes with 3 million local surfaces), which enables us to learn general shape priors.\n\n|     | NGLOD (per-shape) | NGLOD (pre-trained) |  Ours  |\n|:---:|:-----------:|:-------:|:------:|\n| CD (1e-4)  |    1.04    | 3.68 | 1.87 |\n\n---\n\n\n**[W3.2, Generalization across density]** We report the performance of our work using only 10% SDF samples as input. We observe that the performance of CoFie is robust to the number of SDF samples.\n\n|     | 100% SDF samples | 10% SDF samples  |\n|:---:|:-----------:|:-------:|\n| CD (1e-4)  |    1.87 | 2.64 |\n\n---\n\n**[W4, Proof]**  Eq. 14 essentially applies Taylor expansion in x, y, z and ignores third-and-higher order terms in x, y, and z. Please see details in the rebuttal pdf.\n\n---\n\n**[W5, Reference]** Thanks for the detailed feedback. The missing references at the mentioned line are [22, 38, 8, 34], which are the same with the ones we discussed in the second paragraph of the introduction. We will add them in the revision.\n\n---\n\n**[W6, Shape at voxel boundary]** Good question! As mentioned in the paragraph of L211 of the paper, we deal with the problem by using an expanded “receptive field” for each voxel. In practice, we model a local shape using the SDF samples belonging to the neighbor voxel using a distance of 1.5V, where V is the size of the voxel. This improves the continuity of the shape at local voxel boundaries.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for their detailed feedback. We received positive comments from the reviewers who appreciated our clear ablation and convincing evaluation (LpYZ, Kq6m), the efficiency of our method (V5fn, 4hsy), good performance (V5fn, 4hsy), clear theoretical proof (4hsy, Kq6m), novelty (Kq6m). Our goal is to **introduce geometric insights when developing 3D neural representations**, which have proven to be effective in the experiments presented in the paper. Moreover, this paper follows a theoretical style, in which we provide rigorous insights on how to develop 3D representations.

The reviewers also had some additional questions and comments that we address independently in the answer for each reviewer. Moreover, we include a one-page pdf with some further explanation of equation 14 in our proof. Alternatively, **we noticed some of the reviewers misunderstood our setting**, where we are doing shape auto-decoding following DeepSDF and DeepLS, instead of shape auto-encoding, as mentioned in related works like 3DS2VS, NKSR, etc. Please see the details in our rebuttal.

Meta-review: The paper received a very large number of posts between authors and reviewers after the rebuttal, and also between reviewers during the post-rebuttal discussion. All reviewers agreed on the technical and intellectual novelty of the paper. The AC also read the paper and appreciated the geometric insight of attaching learnable local coordinate frames to each patch and approximating the surface through a composition of local, frame-aligned MLPs with quadratic layers. Two of the reviewers remained negative because of limitations in the evaluation i.e., the evaluation should be better done in the generalized auto-encoding setting and preferably on full ShapeNet. The AC agrees that the evaluation could have been more bulletproof, yet, the novelty and geometric insights deserve recognition. The AC recommends acceptance, yet urges the authors to incorporate their revised comparisons and additional experiments in larger datasets in the final version of the paper! The authors should also acknowledge the limitations of their evaluation, including the limitation of using shallow MLPs with ReLU activations in the comparisons (instead of more modern variants, such as GLUs/GEGLUs). The paper should also correct issues with citations, and also add citations to more recent works on 3D neural generative geometric representations e.g., CLAY, Make-A-Shape, GEM3D, Mosaic-SDF.
Decision: Accept (poster)