Title: CoFie: Learning Compact Neural Surface Representations with Coordinate Fields
Venue: NeurIPS 2024
Accepted: yes
Keywords: Neural Surface Representation

Shape Auto-Decoding

Coordinate Field.
Abstract: This paper introduces CoFie, a novel local geometry-aware neural surface representation. CoFie is motivated by the theoretical analysis of local SDFs with quadratic approximation. We find that local shapes are highly compressive in an aligned coordinate frame defined by the normal and tangent directions of local shapes. Accordingly, we introduce Coordinate Field, which is a composition of coordinate frames of all local shapes. The Coordinate Field is optimizable and is used to transform the local shapes from the world coordinate frame to the aligned shape coordinate frame. It largely reduces the complexity of local shapes and benefits the learning of MLP-based implicit representations. Moreover, we introduce quadratic layers into the MLP to enhance expressiveness concerning local shape geometry. CoFie is a generalizable surface representation. It is trained on a curated set of 3D shapes and works on novel shape instances during testing. When using the same amount of parameters with prior works, CoFie reduces the shape error by 48% and 56% on novel instances of both training and unseen shape categories. Moreover, CoFie demonstrates comparable performance to prior works when using even 70% fewer parameters. Code and model can be found here: https://hwjiang1510.github.io/CoFie/

CoFie : Learning Compact Neural Surface Representations with Coordinate Fields

Abstract

This paper introduces CoFie, a novel local geometry-aware neural surface representation. CoFie is motivated by the theoretical analysis of local SDFs with quadratic approximation. We find that local shapes are highly compressive in an aligned coordinate frame defined by the normal and tangent directions of local shapes. Accordingly, we introduce Coordinate Field, which is a composition of coordinate frames of all local shapes. The Coordinate Field is optimizable and is used to transform the local shapes from the world coordinate frame to the aligned shape coordinate frame. It largely reduces the complexity of local shapes and benefits the learning of MLP-based implicit representations. Moreover, we introduce quadratic layers into the MLP to enhance expressiveness concerning local shape geometry. CoFie is a generalizable surface representation. It is trained on a curated set of 3D shapes and works on novel shape instances during testing. When using the same amount of parameters with prior works, CoFie reduces the shape error by 48% and 56% on novel instances of both training and unseen shape categories. Moreover, CoFie demonstrates comparable performance to prior works when using even 70% fewer parameters. Code and model can be found here: https://hwjiang1510.github.io/CoFie/

1 Introduction

In the realm of geometry modeling, neural implicit shape representations have become a powerful tool [33, 7, 4, 13, 41, 39, 2]. These representations typically use latent codes to represent shapes and employ multilayer perceptions (MLPs) to decode their Signed Distance Functions (SDFs). Early works in this field use a single latent code to represent an entire shape [33]. Nevertheless, the decoded SDFs usually lack geometry details. To improve the shape modeling quality, recent approaches have introduced local-based designs [4, 26, 42]. By decomposing an entire shape into many local surfaces, the shape modeling task becomes effortless - local surfaces are in simpler geometry which are easier to represent. Despite the progress, the local-aware design significantly increases the number of parameters, as each local surface is represented by one or even multiple latent codes. Thus, proposing a neural surface representation that is both accurate and compact is necessary.

To achieve this goal, we argue it is important to understand the properties of local surfaces. Following prior works [31, 49, 10, 44], we approximate the local geometry with quadratic patches [9] and perform analysis. Results show the feasibility of fitting the geometry of a specific category of quadratic patches. In detail, the quadratic patches are aligned with the coordinate system defined by the normal, principal directions, and principal curvatures of quadratic patch [9, 31]. However, when the quadratic patches are not aligned - they are freely transformed with random rotations and translations in 3D, mimicking real local surfaces - the optimization will be easily trapped into local

(High Spatial Complexity)

Aligned

(Lower Spatial Complexity)

Transform

FH

minima. This analysis reveals the difficulty of jointly recovering transformation information and geometry of local patches.

Based on the analysis, we propose CoFie, a novel local geometry-aware neural surface representation. The key insight of CoFie is decomposing the transformation information of local shapes from its geometry . As shown in Fig. 1, we associate each local surface with a learnable coordinate frame, which forms a Coordinate Field. We use the Coordinate Field to transform all local surfaces into an aligned coordinate system, reducing their spatial complexity. Thus, the geometry space of local surfaces becomes more compact, where the MLP-based neural SDFs are easier to learn.

An important design aspect is how to represent the Coordinate Field. Departing from the implicitbased representations, we use an explicit representation. Specifically, the coordinate frame of each local surface is parameterized by a rotation and a translation, forming a 6 Degree-of-Freedom pose. Moreover, we initialize the rotation using the estimated normal, principal direction, and principal curvature of a local surface. This design makes CoFie local geometry-aware and facilitates the learning of Coordinate Fields.

To better represent local surfaces' geometry, we introduce quadratic layers to the MLP. Prior works typically employ ReLU-based MLP with shallow layers and limited hidden size [33, 4]. Thus, the MLP is piece-wise linear [25] and cannot represent the distribution of local surfaces well. We demonstrate a simple quadratic layer improves the geometry modeling capability.

CoFie is a generalizable shape representation. After training on a curated dataset, it can represent arbitrary shapes that belong to any novel category. We evaluate CoFie on novel shape instances from both seen (training) and unseen categories, encompassing both synthetic and real shapes. Results show that CoFie outperforms prior arts, reducing the chamfer distance by 50% on instances from both seen and unseen categories. Moreover, CoFie achieves comparable results with prior work using 70% less parameters. In addition, we demonstrate that CoFie, which uses a single shared MLP for all shapes, achieves comparable results with methods that overfit a specific model for each testing shape.

2 Related Work

Implicit Shape Representations. Implicit shape representations are state-of-the-art in encoding shape geometric details [33, 41, 39, 3, 37, 53, 48, 52, 45, 28, 15, 55, 16, 34]. To improve the shape modeling capability, researchers inject local-aware designs. For example, DeepLS [4] integrates voxel grids and local MLPs to decode geometric shapes. Another line of work explores hierarchical representations where the local surfaces are divided unevenly [30, 47, 26, 42, 46, 40], leveraging Octree. For example, Multilevel Partition of Pnity [30] (MPU) blends parametric implicit surface patches into a global implicit surface. DOGNet uses dual-octree designs for neural MPU. The contribution of CoFie is perpendicular to these methods. CoFie still works on evenly divided voxels. However, instead of resolving high-frequency details of local shapes by using higher local resolution, CoFie proposes the Coordinate Field to reduce the spatial complexity. This is motivated by the analysis result that local geometric shapes are highly compressive under suitable coordinate frames.

The idea of a coordinate field is related to several existing approaches. For example, MVP [27] introduced oriented boxes for 3D face synthesis. However, in our setting, the variations in geometry and topology are much more significant than those of 3D human faces. Another relevant work is LDIF [18], which transforms a 3D point in the local coordinate system of each primitive to decode the iso-value of each shape. However, LDIF uses a fixed coordinate frame for each primitive. In contrast, the coordinate field varies spatially in CoFie and can be optimized, allowing us to capture detailed variations of the parts flexibly. Moreover, CoFie is based on a rigorous analysis of the expressivity of SDF and SDF learning. A follow-up approach [56] uses a warping field to transform a 3D point into a canonical space of specific categories. In contrast, CoFie is category-agnostic, benefiting from the use of local shapes.

On the learning side, many approaches show that MLPs are expressive and that their performance depends on the loss of training. For example, SAL [1] and SALD [2] show the importance of integrating normal losses to capture geometric features. SIREN [39] introduced other regularization losses to improve the quality of implicit representations learned. Although these approaches focus on local shape details, CoFie focuses on network design using coordinate frames. The CoFie approach is orthogonal to the encoding schemes.

Hybrid 3D Representations. Each 3D representation has fundamental advantages and limitations from the machine learning and representation perspective. For example, implicit representations allow flexible topologies, whereas explicit representations are easier to edit. Therefore, hybrid 3D representations, which aim to add the strength of different 3D representations for representation learning, have received a lot of attention. The main stream in hybrid 3D representations sequentially applies hybrid 3D models [24, 50, 12, 11, 38, 29, 8, 51]. For example, GRASS [24] combines a part-based representation to capture geometric structures of 3D shapes and a volumetric representation per part to capture geometric details of the parts. DSG-Net [50] employs part-based deformations to capture geometric details of the part. Other examples [12, 11, 38, 4, 14, 5, 19, 20] combine explicit graph, mesh, voxel and triplane representations with implicit volumetric representations to encode geometry details. CoFie is relevant to this series of approaches, where it combines voxel grids to encode global shapes and an implicit representation to decode local geometric details. The novelty of CoFie is that the local module employs a coordinate frame representation and enforces the prior knowledge that the local shape is roughly a low-complexity polynomial surface in the coordinate system defined by normal and principal directions.

Coordinate Field Optimization. The task of computing the proposed cell-based coordinate field is related to the problem of vector-field and frame-field design on meshes, where we want to ensure that the coordinate field is smooth and consistent among adjacent cells, and where we want the normal and tangent directions of each coordinate frame to align with the local fitting results if the fitting results are highly confident. This problem was studied in [35], which introduced a global optimization framework to compute a global vector field on a triangular mesh. Several more recent approaches have developed improved formulations for vector field optimization [17, 22] and extensions to frame field optimization [32, 36]. We refer to [43] for surveys on this topic. Rather than solving a global optimization problem to compute the coordinate field, the learning of the coordinate field in CoFie is driven by learning a compressive MLP.

3 Analysis of Fitting SDFs of Local Patches

In this section, we provide an analysis of fitting local surfaces. Following prior works [31, 49, 10, 44], we simplify local surfaces as quadratic patches. Additionally, we note that some works approximate local surfaces with linear patches [23, 47]. However, to handle the geometry details, it usually requires extremely high [47] or infinite resolution [23] during local surface partition. Approximating local surfaces with quadratic patch is more practical.

3.1 Importance of Non-linearity

A quadratic surface patch can be represented by f ( u, v ) = ( u, v, 1 2 ( au 2 + cv 2 + 2 buv )) , where u 2 + v 2 ≤ r 2 for locality, and a, b, c are parameters for controlling the shape of the quadratic patch. The following proposition characterizes the SDF of a point p to f .

Proposition 1 For each point p = ( x, y, z ) T in the neighborhood of the origin o , the signed distance function from p to f ( u, v ) can be approximated as

where the approximation omits third-and-higher order terms in x , y , and z .

Prop. 1 suggests that the SDF is non-linear. However, a shallow MLP using ReLU activation is piecewise linear, where the ReLU activation functions essentially decompose the input space into subspaces and the function in each subspace is still linear. This motivates the use of quadratic layers instead of linear layers (Sec. 4.2).

To hold generality, in Appendix B, we also analyze the local surface that can not be simplified as a single quadratic patch, i.e. sharp edges as the intersection of two quadratic patches.

3.2 Difficulty of Fitting Transformation Information

We demonstrate the difficulty of recovering the transformation information of quadratic patches during geometry fitting.

Aligned Quadratic Patches. Same as the previous section, we define the SDF of a quadratic local patch as z -1 2 ( ax 2 + cy 2 + 2 bxy ) , where the quadratic patch is axis-aligned. Consider a set of samples { (( x i , y i , z i ) , d i ) , 1 ≤ i ≤ n } from this quadratic patch, where ( x i , y i , z i ) is the location of the point p i , d i is the SDF value, and n is the number of samples. To fit the surface from the samples, we solve the optimization problem as

which is a convex problem that has a unique global optimal.

Unaligned Quadratic Patches. Consider transforming the quadratic patch with a random rigid transformation ( R, t ) . This quadratic patch is not axis-aligned. In this case, the SDF function is given by z ′ -1 2 ( ax ′ 2 + cy ′ 2 +2 bx ′ y ′ ) where ( x ′ , y ′ , z ′ ) = R ( x, y, z ) + t . To fit the surface from the samples, we solve the optimization problem as

In this case, (3) becomes non-convex and has local minima. We defer a detailed characterization of the local minima of (3) to Appendix C.

In general, this non-convex problem makes geometry fitting non-trivial. It motivates the use of the Coordinate Field to explicitly model the transformation information and disentangle the transformation information of local patches from its geometry (Sec. 4.1).

4 CoFie

In this section, we introduce details of CoFie, including its representation (Sec. 4.1), MLP architecture (Sec. 4.2), and its learning scheme (Sec. 4.3).

4.1 CoFie Representation

As shown in Fig. 2, CoFie is based on a hierarchical representation, with coarse and fine-grained geometry. At the coarse level, it represents a shape with voxels. In detail, for a shape S , it divides the space that contains the shape into V × V × V non-overlapping voxel grids, where V is the resolution of the voxel grids. A subset of voxels that intersect with the shape surface will be valid and CoFie only consider the valid sparse voxels to ensure its efficiency.

Data Preparation

Training Shapes

(in Normalized Scale)

…

…

Local Shapes

(in Local Voxels)

Voxel-

based

Partition

…

Local Shapes

w. Coordinate Frames

Coordinate

Field

Initialization

𝒐𝑐

𝒏𝑐

𝒕 𝑐

𝒙

Point

𝒙

(World Coordinate)

Model

𝒙c

(Aligned Coordinate)

Re-orient

Cell

𝒄

(Zoom-in)

Train

𝑑

MLP

𝑧𝑐

Prediction

𝑑

sdf

loss

Optimized Parameters

MLP

𝑧𝑐

𝒐𝑐

𝒏𝑐

𝒕 𝑐

Test Points

Inference

Partition

&

Init.

𝒙𝑐

𝒙𝑐

Re-

orient

MLP

𝑧 𝑐

′

Prediction

sdf

𝒑 𝑐 ′

𝑖

Optimized

Parameters

update

𝒐 𝑐 ′

𝒏𝑐 ′

𝒕

𝑐 ′

𝑧 𝑐

′

Cell

𝒄 ′

…

outside

inside

At the fine-grained level, for each valid voxel v , we use an implicit representation to encode the geometry details for the local surface inside the voxel. Specifically, we use MLP-based neural SDFs. Each voxel v has a latent code z v representing the local geometry and we use the MLP g θ to decode the SDF values. For a point x , its SDF value contributed by the voxel v is

where ( o v , n v , t v ) parameterize the coordinate frame of voxel v . Ideally, o v , n v and t v are the origin, normal direction, and tangent direction of the local surface, respectively.

Intuitively, for decoding the SDF value, we transform the point from the world coordinate system to the shared coordinate system for all local surfaces. o v forms the translation between the two coordinate system, and ( n v , t v ) form the rotation.

The final SDF value at x is then given by

where V is the set of all valid voxels, w ( x , v ) is the weight assigned for the voxel v with regard to point x . In practice, we use w ( x , v ) = 1 if x ∈ v , and w ( x , v ) = 0 otherwise. Finally, the surface of a 3D shape is defined as the union set of local surfaces in its valid voxels V .

4.2 CoFie MLP Architecture

Following the common practice of MLP, we define

where g θ l l : R m l -1 → R m l is a layer with trainable parameters θ l , and where ϕ is an activation function. Denote z l as the output in layer l , i.e., z 0 = ( x ; z ) . A common strategy is to set each g θ l l as a linear function, i.e.,

where θ l = ( A l , b l ) . Furthermore, ϕ is chosen as the ReLU layer, i.e., ϕ ( z l ) = max( z l , 0 ) where the max operator is applied element-wise. This strategy is widely used in prior works [33, 7].

However, in Sec. 3.1, we demonstrate the SDF function has non-negligible quadratic components locally and its incompatibility with MLPs with linear layers and ReLU activation. Therefore, instead, we model the quadratic components with quadratic layers. We let the top k layers of g θ to be quadratic functions, where k ≥ 1 . The quadratic layer can be formulated as

l l -1 where T l ∈ R m l -1 × m l × m l -1 is a tensor, and θ l = ( T l , A l , b l ) .

We can understand the trade-offs between the use of linear layers (Eq. 6) and the quadratic layers (Eq. 7) as follows. With the same latent dimensions m l , the quadratic layers have many more parameters than the linear layers. Therefore, with the same network size, we have to use fewer layers or smaller latent dimensions for quadratic layers. This will limit the capability of the network instead. In practice, setting k = 1 leads to the best performance.

4.3 CoFie Learning Scheme

Problem Setup. Following DeepSDF-series, we perform shape auto-decoding [33, 4]. The task assesses the capability of models to fit/represent given shapes. During both training and inference, the input is points sampled freely in space with their ground-truth SDF values. The output is the neural SDF. Additionally, we notify that the task is different from shape reconstruction from point cloud inputs, or so-called shape auto-encoding , which is studied in [54, 8, 29].

Moreover, CoFie is a generalizable shape representation. It is trained on a curated dataset with multiple shapes. Once trained, the MLP can be used to represent or decode the SDF of any incoming shapes. We note the setting of generalizable shape representation is different from overfitting a shape , where an MLP is specialized for each shape.

Training and Inference. We follow the protocol of the shape auto-decoding task [33, 4]. We train CoFie with a set of shapes denoted as S = { S i , 1 ≤ i ≤ n } . For each shape, we perform voxelization (Sec. 4.1) and train CoFie with valid local shapes. We denote the set of valid local shapes of shape S i as V i . Following [33, 39, 4], we collect a set of point samples P v = ( p j , d j ) in the neighborhood of each voxel v ∈ V i , where p j and d j denote the position of the sample and the SDF value of p j . The point samples are sampled in free space and are not necessary to be on-surface points. For each local shape in voxel v , we associate it with a latent code z v and the coordinate frame ( o v , n v , t v ) . Then the training objective can be formulated as

where p j v = ( n v , t v , n v × t v ) T ( p j -o v ) . In this step, we jointly optimize the MLP, the latent codes, and the coordinate field for all training shapes. Intuitively, it trains the MLP to represent training shapes and optimize the compatibility between the MLP, latent codes and the coordinate fields.

During inference, we freeze the MLP g θ . We optimize the latent code and the coordinate field for a single target shape at one time. It is formulated as

Besides, we use the regularization term over the inferred latent codes following [33, 4].

Shape Consistency at Boundary of Voxels. If we sample the points P v within each voxel v , Eq. 8 and Eq. 9 optimize the local geometry within each voxel independently. This may lead the nonsmooth and inconsistency surface at the boundary of voxels. To solve this, we follow [4] to expand receptive field of each voxel by sampling points from their neighbouring voxels.

Coordinate Field Initialization. Eq. 8 has many unwanted local minima, especially for optimizing the coordinate field. Thus, a good initialization of the coordinate fields ensures the compactness of local shape at early stage of training, and facilitates the learning of MLP. Motivated by the analysis in Sec. 3.2, we use estimated normal and tangent directions to initialize the coordinate fields. In detail, we compute the derivatives of SDF values at these point samples and perform PCA to get them. Besides, o c is initialized as the center of the cell. We find that this initialization is important to reduce errors (Sec. 5.2).

5 Experiment

This section presents an experimental evaluation of CoFie. We begin with the experimental setup and then present the results and ablations.

Implementation Deatils. We use latent code of size 125 for all cells. The MLP is composed of 5 layers where the first 4 layers are linear layers and the last layer is quadratic. The hidden channel size is 128 . We use the voxel grid size of 32 × 32 × 32 . During training, we use 12 shapes for each batch. For each shape, we sample 3000 voxels that intersect with the surface of the shape (with return). We sample 24 points for each cell for training, and each point is sampled within 1.5 times the radius of the voxel to ensure boundary consistency between cells. We use the Adam optimizer [21] with learning rates 5 e -4 , 1 e -3 , and 1 e -3 for the MLP, coordinate fields, and latent codes. We train with 150000 iterations and reduce the learning rates by half for every 20000 iteration. During inference, we use a learning rate of 5 e -4 for 800 iterations. Reconstructed meshes are obtained by performing Marching Cubes with a 128 resolution by default. We use the quaternion representation for the rotation matrix of the coordinate frames. We train on 4 GPUs with 24 GB memory for 1 day.

Training and testing data. We train CoFie on 1000 shape instances sample from ShapeNet [6] of chairs, planes, tables, lamps, and sofas (200 instances for each category). We test CoFie with three test sets for comprehensive analysis of CoFie: i) 250 novel instances from the 5 training ShapeNet categories; ii) 250 novel instances from 10 unseen ShapeNet categories; iii) 24 meshes from the Thingi dataset [57], which captures real scenes. The test set i) checks how CoFie fits the training distribution. Test sets ii) and iii) are used to test the generalization capability of CoFie on novel shapes that observe different structures with training shapes.

Baseline Approaches We compare our CoFie with three types of methods: generalizable methods , which use a single MLP to represent multiple shapes; shape-specific methods , which train an MLP for each testing shape. Generally, the latter genre demonstrates a better performance as the MLP model can be trained to overfit a single testing shape. Both the two types of methods performs shape auto-decoding. Besides, we also report results for a state-of-the-art shape auto-encoding method. We note that it is a reference method while the result is not directly comparable.

Note that CoFie is a generalizable method for shape auto-decoding. We include more details for baselines as follows.

DeepSDF [33] is a generalizable shape auto-decoding method using a global latent code to represent one shape.

DeepLS [4] is a generalizable shape auto-decoding method using local-based representations. DeepLS is a direct comparable baseline.

NGLOD [40] is a shape-specific method for shape auto-decoding, achieving state-of-the-art performance. For a fair comparison with CoFie, we use the level of detail as 3, keeping the number of parameters of the latent codes in the same magnitude as our CoFie.

3DS2VS [54] is a generalizable shape auto-encoding method. It employs transformers to predict the shape latent code, rather than getting it by optimization (shape auto-decoding). The input is on-surface point clouds.

6

DeepLS

5

Error

Shape

3

2

20

40

60

80

100

120

140

DeepSDF

DeepLS

Ours

GT

NGLOD

Ours

(Zoom-in)

GT

Besides, we also compare with state-of-the-art shape auto-encoding (point cloud reconstruction) methods. We note these methods are used as reference for understanding the model performance. They are not directly comparable.

We train DeepSDF, DeepLS, and CoFie using the same dataset for fair comparisons. NGLOD is trained on each test shape. All methods receive the same inputs during inference.

Evaluation Metrics We report the mesh reconstruction error as the chamfer -L 2 distance between the reconstructed and ground-truth meshes. We sample 30000 points to compute the chamfer distances. The meshes are normalized into a unit scale.

5.1 Experimental Results

Qualitative Results. As shown in Fig. 3, CoFie demonstrates strong surface representation capability. The details of geometry are maintained well. The results on out-of-distribution shapes from unseen categories are comparable to the training categories.

Performance on Training Categories. As shown in Table 1, CoFie outperforms baselines by a large margin. In detail, the average chamfer distance of CoFie is 1 . 86 ( 48% relatively) smaller than the best baseline DeepLS. Moreover, we provide a more detailed comparison with DeepLS, as shown in Fig. 4. We observe that CoFie is consistently better than DeepLS with different latent code and MLP size. Specifically, CoFie with latent code size 48 achieves slightly better performance compared with DeepLS with latent code size 128. Note that the number of MLP parameters for the former is about 15% for the latter.

Performance on Unseen Categories. We compare CoFie with previous generalizable methods on ShapeNet unseen categories and the state-of-the-art per-shape-based method on the challenging real scans. We provide visualization results in Fig. 5.

ShapeNet Unseen Categories. As shown in Table 2, CoFie achieves better generalization on 9 out of 10 novel shape categories. We also observe that the performance gap between CoFie and prior works is larger in the unseen categories, showing the strong generalization capability of CoFie.

Thingi Real Shapes. As shown in Table 3, CoFie achieves comparable results with NGLOD. We note that NGLOD is a per-shape-based method, which trains a model for each shape and performs better naturally. In contrast, CoFie is trained on ShapeNet shapes.

5.2 Ablation Study

As shown in Table 4, we experiment with CoFie variants to validate the effectiveness of our coordinate field and MLP designs.

Coordinate Field and Initialization. As shown in Table 4 (1), using coordinate fields with different initialization strategies can both reduce the shape error. In detail, when using axis-aligned coordinate field initialization, where all coordinate frames are initialized as the world frame, the shape error reduced slightly from 3 . 91 to 3 . 45 . The result demonstrates the difficulty of optimizing coordinate frames. In contrast, when using geometry-aware initialization, i.e., initializing local frames with estimated normal and tangent directions of local shapes, the shape error is reduced to 2 . 33 , observing a 40% improvement.

MLPDesign. As shown in Table 4 (2), using a quadratic layer as the last layer of the MLP observes a 0 . 9 ( 23% relatively) reduction of shape error. As the use of the quadratic layer introduces additional parameters, we compare it with a variant for a fair comparison. In detail, we compare it with a linear MLP with an additional layer (6 layers in total), where the two MLPs have the same amount of parameters because the output channel size of the last layer is 1. The result shows that increasing the number of linear layers can only reduce the shape error slightly.

Moreover, Table 4 (3) demonstrates the combination of the two introduced techniques can jointly reduce the shape error.

6 Conclusions and Future Work

This paper has introduced CoFie, a novel neural surface representation. It is based on the theoretical results of using a ReLU-based MLP to encode geometric shapes. The results strongly motivate

the use of local coordinate frames, which encompass the coordinate fields, to transform a point before decoding its SDF value using an MLP. This leads to a hybrid representation combined with coordinate frames associated with local voxels. The experimental results show a strong generalization behavior of CoFie in new instances for shape reconstruction, which significantly outperforms previous generalizable methods and achieves comparable results to shape-specific methods.

Limitations. One limitation of CoFie is that it is based on local shapes and cannot be used for the shape completion task. Different from DeepSDF, which learns global shape priors and can fill the large missing components in the input, CoFie is restricted to observable parts. We plan to incorporate more global priors into CoFie. Besides, with a fixed cell resolution, the local shape analysis is broken when a local cell intersects with thin structures. We plan to extend it with adaptive local cell resolutions.

Broader Impact. CoFie is a neural surface representation, which have the potential to be used for 3D reconstruction and generation.

Acknowledgment. Q. H. would like to acknowledge NSF IIS 2047677 and NSF IIS 2413161

References

Matan Atzmon and Yaron Lipman. SAL: sign agnostic learning of shapes from raw data. In CVPR , 2020.

Matan Atzmon and Yaron Lipman. SALD: sign agnostic learning with derivatives. In ICLR , 2021.

Tristan Aumentado-Armstrong, Stavros Tsogkas, Sven Dickinson, and Allan D Jepson. Representing 3d shapes with probabilistic directed distance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19343-19354, 2022.

Rohan Chabra, Jan Eric Lenssen, Eddy Ilg, Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard A. Newcombe. Deep local shapes: Learning local SDF priors for detailed 3d reconstruction. In ECCV , 2020.

Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16123-16133, 2022.

Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository, 2015.

Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In CVPR , 2019.

Yen-Chi Cheng, Hsin-Ying Lee, Sergey Tulyakov, Alexander G Schwing, and Liang-Yan Gui. Sdfusion: Multimodal 3d shape completion, reconstruction, and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4456-4465, 2023.

Manfredo P. do Carmo. Differential geometry of curves and surfaces. 1976.

Roderik GF Erens, Astrid ML Kappers, and Jan J Koenderink. Perception of local shape from shading. Perception & psychophysics , 54:145-156, 1993.

Lin Gao, Jie Yang, Tong Wu, Yu-Jie Yuan, Hongbo Fu, Yu-Kun Lai, and Hao Zhang. Sdm-net: Deep generative network for structured deformable mesh. ACM Trans. Graph. , 2019.

Georgia Gkioxari, Justin Johnson, and Jitendra Malik. Mesh R-CNN. In ICCV , 2019.

Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization for learning shapes. In ICML , 2020.

Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.04400 , 2023.

Jiahui Huang, Zan Gojcic, Matan Atzmon, Or Litany, Sanja Fidler, and Francis Williams. Neural kernel surface reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4369-4379, 2023.

Ka-Hei Hui, Aditya Sanghi, Arianna Rampini, Kamal Rahimi Malekshan, Zhengzhe Liu, Hooman Shayani, and Chi-Wing Fu. Make-a-shape: a ten-million-scale 3d shape model. In Forty-first International Conference on Machine Learning , 2024.

Wenzel Jakob, Marco Tarini, Daniele Panozzo, and Olga Sorkine-Hornung. Instant field-aligned meshes. ACM Trans. Graph. , 2015.

Chiyu "Max" Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, and Thomas A. Funkhouser. Local implicit grid representations for 3d scenes. In CVPR , 2020.

Hanwen Jiang, Zhenyu Jiang, Kristen Grauman, and Yuke Zhu. Few-view object reconstruction with unknown categories and camera poses. arXiv preprint arXiv:2212.04492 , 2022.

Hanwen Jiang, Zhenyu Jiang, Yue Zhao, and Qixing Huang. Leap: Liberate sparse-view 3d modeling from camera poses. arXiv preprint arXiv:2310.01410 , 2023.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR , 2015.

Felix Knöppel, Keenan Crane, Ulrich Pinkall, and Peter Schröder. Globally optimal direction fields. ACM Trans. Graph. , 2013.

Ravikrishna Kolluri. Provably good moving least squares. ACM Transactions on Algorithms (TALG) , 4(2):1-25, 2008.

Jun Li, Kai Xu, Siddhartha Chaudhuri, Ersin Yumer, Hao Zhang, and Leonidas Guibas. Grass: Generative recursive autoencoders for shape structures. ACM Trans. Graph. , 2017.

Ruiyuan Lin, Suya You, Raghuveer Rao, and C-C Jay Kuo. Constructing multilayer perceptrons as piecewise low-order polynomial approximators: a signal processing approach. arXiv preprint arXiv:2010.07871 , 2020.

Qinqing Liu, Peng-Shuai Wang, Chunjiang Zhu, Blake Blumenfeld Gaines, Tan Zhu, Jinbo Bi, and Minghu Song. Octsurf: Efficient hierarchical voxel-based molecular surface representation for protein-ligand affinity prediction. Journal of Molecular Graphics and Modelling , 105:107865, 2021.

Stephen Lombardi, Tomas Simon, Gabriel Schwartz, Michael Zollhoefer, Yaser Sheikh, and Jason Saragih. Mixture of volumetric primitives for efficient neural rendering. ACM Trans. Graph. , 2021.

Yujie Lu, Long Wan, Nayu Ding, Yulong Wang, Shuhan Shen, Shen Cai, and Lin Gao. Unsigned orthogonal distance fields: An accurate neural implicit representation for diverse 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20551-20560, 2024.

Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, and Shubham Tulsiani. Autosdf: Shape priors for 3d completion, reconstruction and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 306-315, 2022.

Yutaka Ohtake, Alexander Belyaev, Marc Alexa, Greg Turk, and Hans-Peter Seidel. Multi-level partition of unity implicits. ACM Trans. Graph. , 2003.

Yutaka Ohtake, Alexander G. Belyaev, Marc Alexa, Greg Turk, and Hans-Peter Seidel. Multi-level partition of unity implicits. ACM SIGGRAPH 2003 Papers , 2003.

Daniele Panozzo, Enrico Puppo, Marco Tarini, and Olga Sorkine-Hornung. Frame fields: Anisotropic and non-orthogonal cross fields. ACM Trans. Graph. , 2014.

Jeong Joon Park, Peter R. Florence, Julian Straub, Richard A. Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR , 2019.

Dmitry Petrov, Pradyumn Goyal, Vikas Thamizharasan, Vladimir Kim, Matheus Gadelha, Melinos Averkiou, Siddhartha Chaudhuri, and Evangelos Kalogerakis. Gem3d: Generative medial abstractions for 3d shape synthesis. In ACM SIGGRAPH 2024 Conference Papers , pages 1-11, 2024.

Nicolas Ray, Wan Chiu Li, Bruno Lévy, Alla Sheffer, and Pierre Alliez. Periodic global parameterization. ACM Trans. Graph. , 2006.

Nicolas Ray, Dmitry Sokolov, and Bruno Lévy. Practical 3d frame field generation. ACM Trans. Graph. , 2016.

Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, and Sanja Fidler. Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. Advances in Neural Information Processing Systems , 34:6087-6101, 2021.

Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, and Sanja Fidler. Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. In NeurIPS , 2021.

Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In NeurIPS , 2020.

Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11358-11367, Washington, DC, USA, 2021. IEEE.

Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In NeurIPS , 2020.

Jia-Heng Tang, Weikai Chen, Jie Yang, Bo Wang, Songrun Liu, Bo Yang, and Lin Gao. Octfield: Hierarchical implicit functions for 3d modeling. arXiv preprint arXiv:2111.01067 , 2021.

Amir Vaxman, Marcel Campen, Olga Diamanti, David Bommes, Klaus Hildebrandt, Mirela Ben-Chen Technion, and Daniele Panozzo. Directional field synthesis, design, and processing. In ACM SIGGRAPH 2017 Courses , 2017.

Timothy P Wallace, Owen Robert Mitchell, and Keinosuke Fukunaga. Three-dimensional shape analysis using local shape descriptors. IEEE Transactions on Pattern Analysis and Machine Intelligence , (3):310323, 1981.

Li Wang, Weikai Chen, Xiaoxu Meng, Bo Yang, Jintao Li, Lin Gao, et al. Hsdf: Hybrid sign and distance field for modeling surfaces with arbitrary topologies. Advances in Neural Information Processing Systems , 35:32172-32185, 2022.

Peng-Shuai Wang, Yang Liu, and Xin Tong. Dual octree graph networks for learning adaptive volumetric shape representations. 2022.

Peng-Shuai Wang, Chun-Yu Sun, Yang Liu, and Xin Tong. Adaptive o-cnn: A patch-based deep representation of 3d shapes. ACM Transactions on Graphics (TOG) , 37(6):1-11, 2018.

Zixiong Wang, Yunxiao Zhang, Rui Xu, Fan Zhang, Peng-Shuai Wang, Shuangmin Chen, Shiqing Xin, Wenping Wang, and Changhe Tu. Neural-singular-hessian: Implicit neural representation of unoriented point clouds by enforcing singular hessian. ACM Transactions on Graphics (TOG) , 42(6):1-14, 2023.

Ying Xiong, Ayan Chakrabarti, Ronen Basri, Steven J Gortler, David W Jacobs, and Todd Zickler. From shading to local shape. IEEE transactions on pattern analysis and machine intelligence , 37(1):67-79, 2014.

Jie Yang, Kaichun Mo, Yu-Kun Lai, Leonidas J. Guibas, and Lin Gao. Dsg-net: Learning disentangled structure and geometry for 3d shape generation. ACM Trans. Graph. , 2022.

Lior Yariv, Omri Puny, Oran Gafni, and Yaron Lipman. Mosaic-sdf for 3d generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4630-4639, 2024.

Jianglong Ye, Yuntao Chen, Naiyan Wang, and Xiaolong Wang. Gifs: Neural implicit function for general shape representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12829-12839, 2022.

Tarun Yenamandra, Ayush Tewari, Nan Yang, Florian Bernard, Christian Theobalt, and Daniel Cremers. Fire: Fast inverse rendering using directional and signed distance functions. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 3077-3087, 2024.

Biao Zhang, Jiapeng Tang, Matthias Niessner, and Peter Wonka. 3dshape2vecset: A 3d shape representation for neural fields and generative diffusion models. ACM Transactions on Graphics (TOG) , 42(4):1-16, 2023.

Longwen Zhang, Ziyu Wang, Qixuan Zhang, Qiwei Qiu, Anqi Pang, Haoran Jiang, Wei Yang, Lan Xu, and Jingyi Yu. Clay: A controllable large-scale generative model for creating high-quality 3d assets. ACM Transactions on Graphics (TOG) , 43(4):1-20, 2024.

Zerong Zheng, Tao Yu, Qionghai Dai, and Yebin Liu. Deep implicit templates for 3d shape representation. In CVPR , 2021.

Qingnan Zhou and Alec Jacobson. Thingi10k: A dataset of 10,000 3d-printing models, 2016.

Appendices

A Proof of Prop. 1

Let ( u, v ) be the parameters of the closest point of p = ( x, y, z ) T on f ( u, v ) . We have the following constraints on ( u, v ) :

Ignoring quadratic-and-higher order terms in u , v , x , y , and z in (10) and (11), we have

The normal direction at ( u, v ) is

The signed-distance function of p to f ( u, v ) is given by

Substituting (15), (14) into (16) and ignoring third-and-higher terms in u, v, x, y, z , we have

Note that

This leads to

□

B Representing Sharp Edges as Quadratic Patches

We consider the intersection of two quadratic patches where the intersection is along the y -axis. In this case, we can define the surface patch as f ( u, v ) = ( u, v, f ( u, v )) T where

In (17), we do not have any linear term in v , so that the normals to these two patches at (0 , 0 , 0) T are in the xz plane. In addition, the coefficients in front of v 2 are identical, so these two patches stitch along u = 0 .

The following proposition provides an approximation to the SDF function of f ( u, v ) .

Proposition 2 For each point p = ( x, y, z ) T in the neighborhood of the origin o , the signed distance function from p to f ( u, v ) can be approximated as

The proof is very similar to that of Prop. 1. When x ≥ 0 , the parameters ( u, v ) of the closest point satisfy u ≥ 0 , and vice versa. Therefore, the proof applies the description in Section A. □

C Local Minima of (3)

We will show that there are nontrivial local minima due to symmetries induced by the rotation group. However, those local minima do not recover the underlying ground-truth shape. As a result, they force the network to learn the wrong patterns from the data. For simplicity, we focus on the 2D setting. The extension to 3D is straightforward.

In 2D, we assume that the underlying curve is ( x, k 0 x 2 ) . SDF samples are given by ( x, k 0 x 2 + y, y ) where x ∼ p, y ∼ q . Consider the 2D rigid pose parameters θ, t x , t y . Let k be the curve parameter. Our goal is to optimize parameters θ, t x , t y , k to minimize the following L 2 reconstruction loss:

Clearly, ( k 0 , 0 , 0 , 0) is a global minimum of r . The following proposition shows that there is another local minimum of r .

Proposition 3 Suppose p and q are independent, and

Then ( -k 0 , 0 , 2 c, π ) is a critical point of r , where c = E y ∼ q y . In addition, it is a local minimum of r if we assume

We defer the proof of Prop. 3 to Appendix C.1. Prop. 3 shows that there is a non-trivial critical point whose parameters depend on the sampling pattern. As neural network training mostly uses firstorder methods that can be trapped into critical points, this means that without careful initialization, the network will memorize non-shape-related patterns from data, and significantly impairs the generalization ability of the resulting network.

C.1 Proof of Prop. 3

Denote

It is easy to check that

The first-order gradients of l with respect to k, t x , t y , θ are given by

Therefore, we have

and and

and

This means that ( -k 0 , 0 , 2 c, π ) is a critical point of r . To show that it is indeed a local minimum, we study the second-order derivatives of r . We begin with the second-order derivatives of l . They are

and and

Note that ∀ α, β ∈ { k, t x , t y , θ } ,

Denote

We have and

and

It remains to show that

and

The difference between the left and right-hand sides of (24)

As y ≪ x , the above quantity is above zero if

which can be derived from Cauchy inequality. (25) is equivalent to

which can be derived from the Cauchy inequality.

D More Results

Visualization . We include more visualization comparisons. We show the comparison with generalizable methods and scene-specific methods in Fig. 6 and Fig. 7, respectively. We also include a failure case of CoFie in Fig. 8.

Quantitative Results. We include a more comprehensive comparison with generalizable shape auto-encoding (GAE) and shape-specific auto-decoding (SSAD) methods for understanding the performance of our model. Again, we note CoFie performs generalizable shape auto-decoding (GAD) and is not directly comparable to these models.

□

DeepSDF

DeepLS

Ours

GT

D

口

P

P

D

Ours

NGLOD

GT

S

Ours (2 Views)

GT (2 Views)

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We include the experiments to validate every claim.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please see the last page of the paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have included the theoretical analysis in the supplementary.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have included all experiment details.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We committed to releasing code upon acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have included all experiment details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Our method outperforms the baselines with significant margins. Visualization also verifies the contributions.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have included the details in the experiment details section.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our paper does not contain any harmful results. It is neutral research.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed the broader impacts on the last page of the paper.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper on shape representation does not have the risk of being misused.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification:

ShapeNet: Custom (non-commercial)

Thingi: CC BY-NC 2.0 license

DeepSDF: MIT License

3DLatent2VecSet: Custom (non-commercial)

DeepLS: MIT License

NGLOD: MIT License

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We will release the code with documentation.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new architecture for SDF auto-decoding task. It divides the whole SDF points into voxels and builds local coordinates for each surface patch included in valid voxels. Respective shape coding is learned separately and a generalizable MLP is used for SDF decoding. For MLP, a quadratic layer is proposed to better fit SDF values. Compared to the selected baselines, the proposed method shows great performance on both synthetic and real datasets.\n\nSTRENGTHS:\nThe paper theoretically proves that ReLU based MLP is not enough for modelling the quadratic surface and thoroughly explains the difficulty in training local coordinate transformation. The results on two datasets show that local shape coding and hybrid representations have good potential for shape decoding.\n\nWEAKNESSES:\nThe evaluation on the main assertions of this paper is not thorough. The method should be evaluated in three levels – accuracy, scalability and generalizability. \n\n1. The first one has been partially demonstrated by comparing with the selected baselines. However, here a SOTA baseline - Neural Kernel Surface Reconstruction (CVPR’23) - is missing.\n \n2. Scalability is not studied by the paper. Only object level datasets are tried. Large scale datasets such as CARLA should also be evaluated to demonstrate the ability of the model. This is a reasonable experiment for a locally coded model, because it only needs to introduce more and more patches for larger cases. \n\n3. Generalizability is one of the selling points for this paper. However, there are some issues related to the baselines and experiment settings. \n\nFirst of all, the paper states that the baseline NGLOD is a per-scene model, not a generalizable model, so NGLOD naturally could have better performance. However, NGLOD also has generalization ability according to the original paper, only LOD features are trained per-scene while keeping the MLP fixed, which is actually a similar setting to this paper. \n\nSecondly, there are two kinds of generalizability: across domain and across density. In this paper only generalizability across domain, i.e. object level synthetic to real, is tried. Scene level, for example ShapeNet and Synthetic Room to ScanNet or Matterport3D, and generalization across density, that is trained and inferred on different sampling density, are missing.\n\n4. Besides the experiments, there is one problem in theory derivation: \n\nIn the proof of proposition 1, equation 14 seems unreasonable to be equal. It is suggested to explain to what extend these two expressions could be regarded as equal. If equation 14 is not equal, how the final equation could be held?\n\n5. Missing reference(s): Line 114, missing reference(s) for using quadratic surface patch\n\nQUESTIONS:\nBesides the weaknesses, there is one more question: currently all points in one voxel are assumed to be related to the surface within the same voxel, what if the points near the boundary of the voxel are related to the surface included in the nearby voxel?\n\nLIMITATIONS:\nRefer to weaknesses and questions.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n### Motivation\n- In the realm of implicit 3D shape representations, local-based solutions [4, 19, 32] (which decomposes the target shape into a set of local surfaces to model) provide higher accuracy but at the cost of a higher number of parameters to optimize. \n- The authors argues that this number of parameters can be reduced by disentangling the actual geometry and the transformation (orientation, translation) of these local patches, modeling the latter separately.\n\n### Contributions\n- The authors thus introduce CoFie, a novel approach that utilizes an explicit coordinate field to transform local shapes, aligning them to reduce spatial complexity and improving MLP-based implicit local representation of 3D shapes.\n- Various theoretical and empirical evidences are provided w.r.t. the benefits of their formulation, as well as to justify other technical choices, e.g., the usage of quadratic layers to complement their MLPs.\n\n### Results\n- Experiments show that CoFie reduces the shape error by 48% to 56% compared to traditional generalizable methods (e.g., DeepSF [4], DeepLS [24]) and achieves comparable performance to prior works with 70% fewer parameters.\n\n### Relevance\n- This paper addresses the challenge of balancing accuracy and compactness in neural surface representations, a relevant and generic issue in 3D shape modeling.\n\nSTRENGTHS:\n_(somewhat ordered from most to least important)_\n\n### S1. Extensive Theoretical Grounding and Methodological Explanations\n- The authors spent much effort formalizing and proving their contributions, providing the readers with extensive theoretical background.\n- The Methodology itself is clear and thoroughly explained. While the actual implementation has not been provided, an expert in the art should be able to re-implement this work.\n- The paper is also well illustrated and easy to follow (pipeline figures + qualitative results).\n\n### S2. Novelty.\n- To the best of my knowledge, the idea of disentangling geometry and transformation of local shapes is both new and interesting, and could benefit the 3DCV community.\n- The authors nicely tie together their original intuition and extensive formalization to justify their contribution.\n\n### S3. Convincing Evaluation\n- The proposed CoFie method outperforms other generalizable methods in terms of shape accuracy and weight compactness.\n- Qualitative results and the ablation study are also convincing.\n\nWEAKNESSES:\n### W1. Somewhat Limited Comparison to SOTA\n- The authors compare to 4 methods (DeepSDF [24], DeepLS [4], NGLOD [30], 3DS2VS [41]), 2 of which are fairly old for the domain (<2020 for [24] and [4]). The literature has many more recent approaches, e.g. [a,b,c], with code or results pre-available, which could have been considered.\n- Moreover, the authors compare to only one shape-specific method (i.e., performing test-time optimization), NGLOD [30]. While such methods tackle a slightly different task, comparing to a single solution makes it hard to draw solid conclusions.\n\n### W2. Partial Evaluation of \"Compactness\" Claims\n- The authors claim that \"CoFie achieves comparable results with prior work using 70% less parameters\" [L56-57]. However, there is a distinction to be made between parameter number (compactness of the model), convergence speed (number of iterations to convergence), and inference time / computational footprint. I find that claiming some improvement along one of those dimensions without studying the impact on the others not so meaningful. \n\n### W3. Code Not Available\n- While the authors claim that they are \"committed to releasing our code\" [L16], no code has been provided yet to reviewers.\n\n\n### W4. Minor Remarks\n- Citations are missing (empty brackets) in several places [L114, L128, L197].\n- Typos: \"practival: $\\rightarrow$ \"practical\" [L118] ; \"Perfomance\" $\\rightarrow$ \"Performance\" [L270] ; etc.\n\n\n------------\n### Additional References:\n\n[a] Ye, Jianglong, et al. \"Gifs: Neural implicit function for general shape representation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[b] Wang, Li, et al. \"Hsdf: Hybrid sign and distance field for modeling surfaces with arbitrary topologies.\" Advances in Neural Information Processing Systems 35 (2022): 32172-32185.\n\n[c] Lu, Yujie, et al. \"Unsigned Orthogonal Distance Fields: An Accurate Neural Implicit Representation for Diverse 3D Shapes.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\nQUESTIONS:\n_see **Weaknesses** for key questions/suggestions._\n\n### Q1. Computational Cost\n- Besides the points made in **W2** above, I wonder what the computational cost of replacing linear layers with quadratic ones is (c.f. [L178-187])?\n\n### Q2. Grid Size and Surface Complexity\n- What is the impact of the voxel grid size on the surface complexity / ability of the model to learn the surface? Intuitively, the larger each voxel is, the more complex/discontinuous the surface that it intersects can become.\n\n### Q3. Purpose of $w$\n- What is the point of introducing $w$ [L168-171] if in practice it is set to the indicator function (simply selecting the voxel $v$ where $x$ belongs)? Is there a point to keeping Equation 5 this generic?\n\n### Q4. Hyper-parameter $k$\n- The authors mention that setting the number $k$ of quadratic layers (following $L-k$ linear ones) to 1 yields the best performance [L187]. Could the authors share the quantitative evaluation behind this conclusion?\n\nLIMITATIONS:\n- Limitations are discussed in the conclusion, with the authors sharing meaningful insight (e.g., the solution cannot perform shape completion or handle thin structures well).\n- Societal impact are however not explicitly discussed. Providing some potential positive or negative applications of this work would be meaningful.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CoFie, a novel neural surface representation method designed to efficiently learn and represent complex 3D shapes. CoFie addresses the challenges of existing methods by introducing a coordinate field that optimizes the representation of local shapes, significantly reducing shape errors and improving the efficiency of parameter usage. The method is based on a hierarchical representation with coarse and fine-grained geometry, utilizing MLP with quadratic layers to enhance the expressiveness of the model. Experimental results demonstrate CoFie's strong generalization capabilities in shape reconstruction tasks, outperforming previous methods in terms of accuracy and efficiency.\n\nSTRENGTHS:\nCoFie uses a coordinate field to transform local shapes into an aligned coordinate system, reducing spatial complexity and making the learning of MLP-based implicit representations more efficient. By incorporating quadratic layers into the MLP, CoFie enhances the model's ability to capture local shape geometry, improving the quality of shape modeling. CoFie is trained on a curated dataset and can represent arbitrary shapes from novel categories, demonstrating strong generalization capabilities.\nThe theoretical proofs in the paper are sufficient and the articulation is clear.\n\nWEAKNESSES:\n1.CoFie's performance in detailing is strongly related to the cell size, which hinders its ability to represent fine-grained details. \n2.Representations based on nonoverlapping local patches may not favor expressing rich colors, a capability that implicit representations possess. \n3.It excels in displaying regular shapes, such as furniture, but is weak in showcasing shapes with rich curves.\n4. Line 197, citation is missing\n\nQUESTIONS:\nI noticed that the article mentions, \"We train CoFie on 1000 shape instances sampled from ShapeNet [5] consisting of chairs, planes, tables, lamps, and sofas (200 instances for each category).\" Why not train on the entire ShapeNet?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPaper proposes CoFie — 3D shape representation as a set of latents arranged on a regular voxel grid. Each latent encodes an oriented local quadratic patch. This local oriented patch defines local SDF in a local coordinate frame which is decoded via conditional MLP for which the last layer is quadratic: it defines a bilnear form on input vector instead of linear mapping). SDF value for query point x is decided based on the SDF value of local implicit function belonging to the nearest voxel and transformed from local coordinate frame to global. \n\nAuthors train proposed methods and baselines on a single random subsample of 1000 ShapeNet shapes (5 categories, 200 shapes each) and evaluate on holdout ShapeNet shapes and out-of-distribution shapes from Thingi10K. Proposed method outperforms baselines trained in the same limited data regime. The same ShapeNet model is also evaluated with the Neural Geometric Level of Detail method and DeepSDF fitted in the shape overfitting scenario.\n\nSTRENGTHS:\n— Proposed representation is more efficient compared to regular latent grids and is comparable to surface latent grids (3DILG) in terms of latents needed to represent shape; \n— Method figure is clear and well done and quantitative results are well formatted;\n— Qualitative results look comparable to NGLOD which is a strong baseline for shape overfitting setting;\n— Design choices are clearly ablated (Table 4);\n\nWEAKNESSES:\n— Proposed method seems to be very similar to AutoSDF and SDFusion: both methods utilize a regular latent grid that encodes local SDF that is used to infer the global SDF. Main difference seems to be in a local SDF decoder that is quadratic (last layer) for proposed method and also encodes SDF in the local coordinate frame (SDFusion and AutoSDF use global coordinates). Overall, it is fine as long as proposed method compares to these similar methods but evaluation of the paper is limited (see below). \n\n— Similarly to other regular latent grids methods (e.g. SDFusion, AutoSDF), this model scales cubically with respect to grid resolution and thus might not be well suited for representation of topologically challenging shapes with thin parts. \n\n— Evaluation of the paper is extremely limited. For point cloud reconstruction (auto-encoding), proposed methods and baselines are trained on 5 ShapeNet categories with 200 shapes per category (overall 1000 shapes). This evaluation is not enough to support the claim that CoFie “can represent arbitrary shapes that belong to any novel category” (LL52-53). These results might only indicate that CoFie might be better representation in low-data regime but in this case evaluation should be done similar to few-shot learning setting: all models should be trained on several different subsets of ShapeNet (e.g. 5) and average/std of test errors should be reported. I also want to note that 3D2VS trained on full ShapeNet is an extremely strong baseline that achieves almost perfect reconstruction quality for some categories (like airplanes). Also, the paper only uses Chamfer distance for evaluation and ignores other common measures like IoU and surface F-Score. \n\n\n— The choice of baselines is also limited. DeepSDF is a 2019 paper that uses a very simple Pointnet (not even PointNet++) encoder. DeepLS is a 2020 paper. 3D2VS seems to be the only recent baseline for 3D point cloud reconstruction. It is a very strong baseline but authors have not used pretrained models and trained their version on limited data instead. Given the fact that 3D2VS uses a very high capacity attention-based decoder, this training regime might not be a fair comparison because attention-based models often struggle in low data regimes. For the same reason, 3D2VS might not be a good baseline if the goal of the paper is to show that their representation is compact. In this case, 3DILG might be a more suitable and strong baseline (see below) because it uses a simpler local patch encoder (Pointnet), so it might work better in a limited data regime. Since the model utilizes local surface patches similarly to AtlasNet (see links below), it also can be a strong baseline, especially in low data setting. \n\n— Evaluation in an overfitting setting  does not seem methodologically correct. : NGLOD and DeepSDF overfit to single shapes while CoFie was pretrained trained on 1000 shapes. These are completely different settings: overfitting regime measures capability of the model to efficiently fit one shape with preservation of geometric detail, and fitting and evaluation on collection of shapes evaluates the ability of the model to generalize to unseen shapes. This experiment should either be done if fully overfitting setting (similar to NGLOD paper) or in a reconstruction setting (see concerns above). \n\n— Quadratic MLP contribution seems a little bit weak to me. It looks like it is basically equivalent to linear MLP being run on quadratic feature expansion and in this case commonly fourier embeddings of points can be a very strong alternative (this is not tested). \n\n— Paper writing can be significantly improved in clarity. For example, paper does not specify input to auto-encoder for baselines. Is it point cloud or mesh? If it is a point cloud, what is the density of sampling and what sampling was used (FPS, random, poisson disk sampling, etc). How many codes were used to train 3D2VS? 512? What were hyperparameters for baseline training? On the other hand, paper spends a lot of space describing relatively common knowledge like MLP with ReLU activations (LL172-177). Another example is equation (5). On first glance it appears as weighted average, but since authors use voxel indicator function as weight, it means that for each query point x, global implicit function is based only on implicit function of nearest voxel grid latent. \n\n— Some statements in the paper are not correct. For example, the paper states that 3D2VS “employs transformers to predict the shape latent code” (LL258-260) . This is not factually correct: 3D2VS does not represent shape as one vector, it uses latent code clouds (usually 512 codes per shape). \n\n— Representation seems to be computationally expensive. Training on 1000 shapes takes one day on 4 GPUs with 24 GB VRAM. \n\n— Some relevant work is missing\n\nAutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation https://arxiv.org/abs/2203.09516 \n\nSDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation\nhttps://arxiv.org/abs/2212.04493 \n\n4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks https://arxiv.org/abs/1904.08755 \n\n3DILG: Irregular Latent Grids for 3D Generative Modeling https://arxiv.org/abs/2205.13914 \n\n\nAtlasNet: A Papier-Mâché Approach to Learning 3D Surface Generation https://arxiv.org/abs/1802.05384\n\nQUESTIONS:\n— What is the input for the model for auto-encoding settings? Is it a point cloud? Or model back propagates into learnable latent code like DeepSDF? If it is a point cloud, what encoder model is used? If model back propagates into latent code, how come it is generalizable?\n\n— How were baselines trained? Were they trained on the same inputs as the proposed method? Did all methods use the same SDF sampling? As far as I know, 3D2VS uses occupancy as supervision. Was it trained with SDF or occupancy supervision for the paper? \n\n— What motivated size 1000 for a training dataset? Majority of modern shape auto-encoder train either on full ShapeNet of full ShapeNet categories (e.g. ~6000 ShapeNet chairs). If 5 different subsets of ShapeNet of size 1000 would be selected (same categories), how high would be the variance in evaluation across these subsets? I have strong suspicion that results shown int Tables 1 and Tables 2 might not generalize across training subsamples. \n\n— Have you tried using Fourier embeddings of input coordinates instead of quadratic layers? This might be a very strong alternative since it helps coordinate based models to encode high-frequency details very efficiently.\n\nLIMITATIONS:\n— Proposed method represents the shape as a set of latents arranged on a regular latent grid and thus scales cubically with latent grid resolution. This limits method’s ability to efficiently encode topologically challenging shapes. \n\n— Method requires a lot of compute to fit 1000 shapes (1 day on 4GPUs with 24GB VRAM). This might be due to fact that authors use 32^3 latent grid (SDFusion for example uses 8^3). This severely limits scalability of the proposed method even for training on full ShapeNet, not to mention larger datasets like Objaverse.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a local prior based method where the model is trained on a dataset and learns a prior over local patches of the shapes, and then at test time reconstructs patches by optimising to minimise the reconstruction error w.r.t. the trained model's prior. While this has already been done in the literature, they propose to learn a coordinate transformation for each local region and initialise the coordinate transformation based on the geometry in that region. They also use a quadratic layer at the end of the network, inspired by their analysis of representing local patches by a quadratic surface.\n\nSTRENGTHS:\n- Good ablation study showing the benefits of each component. Interestingly the geometric initialization is very important for performance improvement, suggesting that the latent based model has trouble training efficiently.\n\nWEAKNESSES:\n- A lot of things are not made clear (asked in the questions) which makes it hard to understand the method\n- As mentioned in the strengths, it seems that the benefit of this method is that the coordinate field (and especially its init) it enables the latent based model to train better, rather than being a better representation. This could be explored more deeply. Also it makes the question below (\"It seems the proposed method has no regularization, is there a reason for this?\")\n\nQUESTIONS:\n- The method is called coordinate field, but it is not actually a true coordinate field: the coordinate transformation is learned discretely as parameters for each voxel, not continuously throughout the domain?\n- Usually shape autodecoding uses regularization on the latent parameters so then when doing test time latent optimisation the latent parameters can be initialized to zero and be \"close\" to many possible shapes. It seems the proposed method has no regularization, is there a reason for this?\n- Line 219: \"compute the derivatives of the SDF\" however at test time you don't have access to the full SDF (or do you?), so how to you get well oriented normals for initialization?\n- Would like more information about what SDF supervision exactly is given at test time. Is it just the surface points, or is assumed the SDF is known at test time as well?\n- Would like timing performance of test time optimization (general ballpark and if this compares to other methods)\n- Line 243: two types of methods or three types of methods?\n- Missing ref on line 114, line 128 and line 197\n\nLIMITATIONS:\n- Yes, limitations have been discussed adequately.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-15"}