CoFie: Learning Compact Neural Surface Representations with Coordinate Fields

Authors: hanwen  jiang
haitao  yang
georgios  pavlakos
qixing  huang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 00:14:16

Abstract: This paper introduces CoFie, a novel local geometry-aware neural surface representation. CoFie is motivated by the theoretical analysis of local SDFs with quadratic approximation. We find that local shapes are highly compressive in an aligned coordinate frame defined by the normal and tangent directions of local shapes. Accordingly, we introduce Coordinate Field, which is a composition of coordinate frames of all local shapes. The Coordinate Field is optimizable and is used to transform the local shapes from the world coordinate frame to the aligned shape coordinate frame. It largely reduces the complexity of local shapes and benefits the learning of MLP-based implicit representations. Moreover, we introduce quadratic layers into the MLP to enhance expressiveness concerning local shape geometry. CoFie is a generalizable surface representation. It is trained on a curated set of 3D shapes and works on novel shape instances during testing. When using the same amount of parameters with prior works, CoFie reduces the shape error by 48% and 56% on novel instances of both training and unseen shape categories. Moreover, CoFie demonstrates comparable performance to prior works when using even 70% fewer parameters. Code and model can be found here: https://hwjiang1510.github.io/CoFie/

Main paper: 
 introduction: 
 In the realm of geometry modeling, neural implicit shape representations have become a powerful tool [32, 7, 4, 13, 39, 37, 2] . These representations typically use latent codes to represent shapes and employ multilayer perceptions (MLPs) to decode their Signed Distance Functions (SDFs). Early works in this field use a single latent code to represent an entire shape [32] . Nevertheless, the decoded SDFs usually lack geometry details. To improve the shape modeling quality, recent approaches have introduced local-based designs [4, 25, 40] . By decomposing an entire shape into many local surfaces, the shape modeling task becomes effortless -local surfaces are in simpler geometry which are easier to represent. Despite the progress, the local-aware design significantly increases the number of parameters, as each local surface is represented by one or even multiple latent codes. Thus, proposing a neural surface representation that is both accurate and compact is necessary.To achieve this goal, we argue it is important to understand the properties of local surfaces. Following prior works [30, 47, 10, 42] , we approximate the local geometry with quadratic patches [9] and perform analysis. Results show the feasibility of fitting the geometry of a specific category of quadratic patches. In detail, the quadratic patches are aligned with the coordinate system defined by the normal, principal directions, and principal curvatures of quadratic patch [9, 30] . However, when the quadratic patches are not aligned -they are freely transformed with random rotations and translations in 3D, mimicking real local surfaces -the optimization will be easily trapped into local minima. This analysis reveals the difficulty of jointly recovering transformation information and geometry of local patches. Based on the analysis, we propose CoFie, a novel local geometry-aware neural surface representation.The key insight of CoFie is decomposing the transformation information of local shapes from its geometry. As shown in Fig. 1 , we associate each local surface with a learnable coordinate frame, which forms a Coordinate Field. We use the Coordinate Field to transform all local surfaces into an aligned coordinate system, reducing their spatial complexity. Thus, the geometry space of local surfaces becomes more compact, where the MLP-based neural SDFs are easier to learn.An important design aspect is how to represent the Coordinate Field. Departing from the implicitbased representations, we use an explicit representation. Specifically, the coordinate frame of each local surface is parameterized by a rotation and a translation, forming a 6 Degree-of-Freedom pose. Moreover, we initialize the rotation using the estimated normal, principal direction, and principal curvature of a local surface. This design makes CoFie local geometry-aware and facilitates the learning of Coordinate Fields.To better represent local surfaces' geometry, we introduce quadratic layers to the MLP. Prior works typically employ ReLU-based MLP with shallow layers and limited hidden size [32, 4] . Thus, the MLP is piece-wise linear [24] and cannot represent the distribution of local surfaces well. We demonstrate a simple quadratic layer improves the geometry modeling capability.CoFie is a generalizable shape representation. After training on a curated dataset, it can represent arbitrary shapes that belong to any novel category. We evaluate CoFie on novel shape instances from both seen (training) and unseen categories, encompassing both synthetic and real shapes. Results show that CoFie outperforms prior arts, reducing the chamfer distance by 50% on instances from both seen and unseen categories. Moreover, CoFie achieves comparable results with prior work using 70% less parameters. In addition, we demonstrate that CoFie, which uses a single shared MLP for all shapes, achieves comparable results with methods that overfit a specific model for each testing shape. 
related work: 
 Implicit Shape Representations. Implicit shape representations are state-of-the-art in encoding shape geometric details [32, 39, 37, 3, 35, 50, 46, 49, 43, 27, 15] . To improve the shape modeling capability, researchers inject local-aware designs. For example, DeepLS [4] integrates voxel grids and local MLPs to decode geometric shapes. Another line of work explores hierarchical representations where the local surfaces are divided unevenly [29, 45, 25, 40, 44, 38] , leveraging Octree. For example, Multilevel Partition of Pnity [29] (MPU) blends parametric implicit surface patches into a global implicit surface. DOGNet uses dual-octree designs for neural MPU. The contribution of CoFie is perpendicular to these methods. CoFie still works on evenly divided voxels. However, instead of resolving high-frequency details of local shapes by using higher local resolution, CoFie proposes the Coordinate Field to reduce the spatial complexity. This is motivated by the analysis result that local geometric shapes are highly compressive under suitable coordinate frames.The idea of a coordinate field is related to several existing approaches. For example, MVP [26] introduced oriented boxes for 3D face synthesis. However, in our setting, the variations in geometry and topology are much more significant than those of 3D human faces. Another relevant work is LDIF [17] , which transforms a 3D point in the local coordinate system of each primitive to decode the iso-value of each shape. However, LDIF uses a fixed coordinate frame for each primitive. In contrast, the coordinate field varies spatially in CoFie and can be optimized, allowing us to capture detailed variations of the parts flexibly. Moreover, CoFie is based on a rigorous analysis of the expressivity of SDF and SDF learning. A follow-up approach [52] uses a warping field to transform a 3D point into a canonical space of specific categories. In contrast, CoFie is category-agnostic, benefiting from the use of local shapes.On the learning side, many approaches show that MLPs are expressive and that their performance depends on the loss of training. For example, SAL [1] and SALD [2] show the importance of integrating normal losses to capture geometric features. SIREN [37] introduced other regularization losses to improve the quality of implicit representations learned. Although these approaches focus on local shape details, CoFie focuses on network design using coordinate frames. The CoFie approach is orthogonal to the encoding schemes.Hybrid 3D Representations. Each 3D representation has fundamental advantages and limitations from the machine learning and representation perspective. For example, implicit representations allow flexible topologies, whereas explicit representations are easier to edit. Therefore, hybrid 3D representations, which aim to add the strength of different 3D representations for representation learning, have received a lot of attention. The main stream in hybrid 3D representations sequentially applies hybrid 3D models [23, 48, 12, 11, 36, 28, 8] . For example, GRASS [23] combines a partbased representation to capture geometric structures of 3D shapes and a volumetric representation per part to capture geometric details of the parts. DSG-Net [48] employs part-based deformations to capture geometric details of the part. Other examples [12, 11, 36, 4, 14, 5, 18, 19] combine explicit graph, mesh, voxel and triplane representations with implicit volumetric representations to encode geometry details. CoFie is relevant to this series of approaches, where it combines voxel grids to encode global shapes and an implicit representation to decode local geometric details. The novelty of CoFie is that the local module employs a coordinate frame representation and enforces the prior knowledge that the local shape is roughly a low-complexity polynomial surface in the coordinate system defined by normal and principal directions.Coordinate Field Optimization. The task of computing the proposed cell-based coordinate field is related to the problem of vector-field and frame-field design on meshes, where we want to ensure that the coordinate field is smooth and consistent among adjacent cells, and where we want the normal and tangent directions of each coordinate frame to align with the local fitting results if the fitting results are highly confident. This problem was studied in [33] , which introduced a global optimization framework to compute a global vector field on a triangular mesh. Several more recent approaches have developed improved formulations for vector field optimization [16, 21] and extensions to frame field optimization [31, 34] . We refer to [41] for surveys on this topic. Rather than solving a global optimization problem to compute the coordinate field, the learning of the coordinate field in CoFie is driven by learning a compressive MLP.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Please see the last page of the paper.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
analysis of fitting sdfs of local patches: 
 In this section, we provide an analysis of fitting local surfaces. Following prior works [30, 47, 10, 42] , we simplify local surfaces as quadratic patches. Additionally, we note that some works approximate local surfaces with linear patches [22, 45] . However, to handle the geometry details, it usually requires extremely high [45] or infinite resolution [22] during local surface partition. Approximating local surfaces with quadratic patch is more practical.A quadratic surface patch can be represented by f (u, v) = (u, v, 1 2 (au 2 + cv 2 + 2buv)), where u 2 + v 2 ≤ r 2 for locality, and a, b, c are parameters for controlling the shape of the quadratic patch. The following proposition characterizes the SDF of a point p to f . Proposition 1 For each point p = (x, y, z) T in the neighborhood of the origin o, the signed distance function from p to f (u, v) can be approximated asEQUATIONwhere the approximation omits third-and-higher order terms in x, y, and z.Proof: See Appendix A.Prop. 1 suggests that the SDF is non-linear. However, a shallow MLP using ReLU activation is piecewise linear, where the ReLU activation functions essentially decompose the input space into subspaces and the function in each subspace is still linear. This motivates the use of quadratic layers instead of linear layers (Sec. 4.2).To hold generality, in Appendix B, we also analyze the local surface that can not be simplified as a single quadratic patch, i.e. sharp edges as the intersection of two quadratic patches.We demonstrate the difficulty of recovering the transformation information of quadratic patches during geometry fitting.Aligned Quadratic Patches. Same as the previous section, we define the SDF of a quadratic local patch as z -1 2 (ax 2 + cy 2 + 2bxy), where the quadratic patch is axis-aligned. Consider a set of samples {((x i , y i , z i ), d i ), 1 ≤ i ≤ n} from this quadratic patch, where (x i , y i , z i ) is the location of the point p i , d i is the SDF value, and n is the number of samples. To fit the surface from the samples, we solve the optimization problem asEQUATIONwhich is a convex problem that has a unique global optimal.Unaligned Quadratic Patches. Consider transforming the quadratic patch with a random rigid transformation (R, t). This quadratic patch is not axis-aligned. In this case, the SDF function is given by z ′ -1 2 (ax ′ 2 + cy ′ 2 + 2bx ′ y ′ ) where (x ′ , y ′ , z ′ ) = R(x, y, z) + t. To fit the surface from the samples, we solve the optimization problem asEQUATIONIn this case, (3) becomes non-convex and has local minima. We defer a detailed characterization of the local minima of (3) to Appendix C.In general, this non-convex problem makes geometry fitting non-trivial. It motivates the use of the Coordinate Field to explicitly model the transformation information and disentangle the transformation information of local patches from its geometry (Sec. 4.1).Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
cofie: 
 In this section, we introduce details of CoFie, including its representation (Sec. 4.1), MLP architecture (Sec. 4.2), and its learning scheme (Sec. 4.3).As shown in Fig. 2 , CoFie is based on a hierarchical representation, with coarse and fine-grained geometry. At the coarse level, it represents a shape with voxels. In detail, for a shape S, it divides the space that contains the shape into V × V × V non-overlapping voxel grids, where V is the resolution of the voxel grids. A subset of voxels that intersect with the shape surface will be valid and CoFie only consider the valid sparse voxels to ensure its efficiency. At the fine-grained level, for each valid voxel v, we use an implicit representation to encode the geometry details for the local surface inside the voxel. Specifically, we use MLP-based neural SDFs. Each voxel v has a latent code z v representing the local geometry and we use the MLP g θ to decode the SDF values. For a point x, its SDF value contributed by the voxel v isEQUATIONwhere (o v , n v , t v ) parameterize the coordinate frame of voxel v. Ideally, o v , n v and t v are the origin, normal direction, and tangent direction of the local surface, respectively.Intuitively, for decoding the SDF value, we transform the point from the world coordinate system to the shared coordinate system for all local surfaces. o v forms the translation between the two coordinate system, and (n v , t v ) form the rotation.The final SDF value at x is then given byEQUATION)where V is the set of all valid voxels, w(x, v) is the weight assigned for the voxel v with regard to point x. In practice, we use w(x, v) = 1 if x ∈ v, and w(x, v) = 0 otherwise. Finally, the surface of a 3D shape is defined as the union set of local surfaces in its valid voxels V.Following the common practice of MLP, we defineg θ (x, z) = g θ L L • ϕ • g θ L-1 L-1 • ϕ • • • • ϕ • g θ1 1 (x, z)where g θ l l : R m l-1 → R m l is a layer with trainable parameters θ l , and where ϕ is an activation function. Denote z l as the output in layer l, i.e., z 0 = (x; z). A common strategy is to set each g θ l l as a linear function, i.e.,EQUATIONwhere θ l = (A l , b l ). Furthermore, ϕ is chosen as the ReLU layer, i.e., ϕ(z l ) = max(z l , 0) where the max operator is applied element-wise. This strategy is widely used in prior works [32, 7] .However, in Sec. 3.1, we demonstrate the SDF function has non-negligible quadratic components locally and its incompatibility with MLPs with linear layers and ReLU activation. Therefore, instead, we model the quadratic components with quadratic layers. We let the top k layers of g θ to be quadratic functions, where k ≥ 1. The quadratic layer can be formulated asEQUATIONwhere T l ∈ R m l-1 ×m l ×m l-1 is a tensor, andθ l = (T l , A l , b l ).We can understand the trade-offs between the use of linear layers (Eq. 6) and the quadratic layers (Eq. 7) as follows. With the same latent dimensions m l , the quadratic layers have many more parameters than the linear layers. Therefore, with the same network size, we have to use fewer layers or smaller latent dimensions for quadratic layers. This will limit the capability of the network instead.In practice, setting k = 1 leads to the best performance.Problem Setup. Following DeepSDF-series, we perform shape auto-decoding [32, 4] . The task assesses the capability of models to fit/represent given shapes. During both training and inference, the input is points sampled freely in space with their ground-truth SDF values. The output is the neural SDF. Additionally, we notify that the task is different from shape reconstruction from point cloud inputs, or so-called shape auto-encoding, which is studied in [51, 8, 28] .Moreover, CoFie is a generalizable shape representation. It is trained on a curated dataset with multiple shapes. Once trained, the MLP can be used to represent or decode the SDF of any incoming shapes. We note the setting of generalizable shape representation is different from overfitting a shape, where an MLP is specialized for each shape.Training and Inference. We follow the protocol of the shape auto-decoding task [32, 4] . We train CoFie with a set of shapes denoted as S = {S i , 1 ≤ i ≤ n}. For each shape, we perform voxelization (Sec. 4.1) and train CoFie with valid local shapes. We denote the set of valid local shapes of shape S i as V i . Following [32, 37, 4] , we collect a set of point samples P v = (p j , d j ) in the neighborhood of each voxel v ∈ V i , where p j and d j denote the position of the sample and the SDF value of p j . The point samples are sampled in free space and are not necessary to be on-surface points. For each local shape in voxel v, we associate it with a latent code z v and the coordinate frame(o v , n v , t v ).Then the training objective can be formulated asEQUATIONwherep j v = (n v , t v , n v × t v ) T (p j -o v ).In this step, we jointly optimize the MLP, the latent codes, and the coordinate field for all training shapes. Intuitively, it trains the MLP to represent training shapes and optimize the compatibility between the MLP, latent codes and the coordinate fields.During inference, we freeze the MLP g θ . We optimize the latent code and the coordinate field for a single target shape at one time. It is formulated asEQUATIONBesides, we use the regularization term over the inferred latent codes following [32, 4] .Shape Consistency at Boundary of Voxels. If we sample the points P v within each voxel v, Eq. 8 and Eq. 9 optimize the local geometry within each voxel independently. This may lead the nonsmooth and inconsistency surface at the boundary of voxels. To solve this, we follow [4] to expand receptive field of each voxel by sampling points from their neighbouring voxels.Coordinate Field Initialization. Eq. 8 has many unwanted local minima, especially for optimizing the coordinate field. Thus, a good initialization of the coordinate fields ensures the compactness of local shape at early stage of training, and facilitates the learning of MLP. Motivated by the analysis in Sec. 3.2, we use estimated normal and tangent directions to initialize the coordinate fields. In detail, we compute the derivatives of SDF values at these point samples and perform PCA to get them. Besides, o c is initialized as the center of the cell. We find that this initialization is important to reduce errors (Sec. 5.2). 
experiment: 
 This section presents an experimental evaluation of CoFie. We begin with the experimental setup and then present the results and ablations. For each shape, we sample 3000 voxels that intersect with the surface of the shape (with return). We sample 24 points for each cell for training, and each point is sampled within 1.5 times the radius of the voxel to ensure boundary consistency between cells. We use the Adam optimizer [20] with learning rates 5e -4, 1e -3, and 1e -3 for the MLP, coordinate fields, and latent codes. We train with 150000 iterations and reduce the learning rates by half for every 20000 iteration. During inference, we use a learning rate of 5e -4 for 800 iterations. Reconstructed meshes are obtained by performing Marching Cubes with a 128 resolution by default. We use the quaternion representation for the rotation matrix of the coordinate frames. We train on 4 GPUs with 24GB memory for 1 day.Training and testing data. We train CoFie on 1000 shape instances sample from ShapeNet [6] of chairs, planes, tables, lamps, and sofas (200 instances for each category). We test CoFie with three test sets for comprehensive analysis of CoFie: i) 250 novel instances from the 5 training ShapeNet categories; ii) 250 novel instances from 10 unseen ShapeNet categories; iii) 24 meshes from the Thingi dataset [53] , which captures real scenes. The test set i) checks how CoFie fits the training distribution. Test sets ii) and iii) are used to test the generalization capability of CoFie on novel shapes that observe different structures with training shapes.Baseline Approaches We compare our CoFie with three types of methods: generalizable methods, which use a single MLP to represent multiple shapes; shape-specific methods, which train an MLP for each testing shape. Generally, the latter genre demonstrates a better performance as the MLP model can be trained to overfit a single testing shape. Both the two types of methods performs shape auto-decoding. Besides, we also report results for a state-of-the-art shape auto-encoding method. We note that it is a reference method while the result is not directly comparable.Note that CoFie is a generalizable method for shape auto-decoding. We include more details for baselines as follows.• DeepSDF [32] is a generalizable shape auto-decoding method using a global latent code to represent one shape.• DeepLS [4] is a generalizable shape auto-decoding method using local-based representations. DeepLS is a direct comparable baseline.• NGLOD [38] is a shape-specific method for shape auto-decoding, achieving state-of-the-art performance. For a fair comparison with CoFie, we use the level of detail as 3, keeping the number of parameters of the latent codes in the same magnitude as our CoFie.• 3DS2VS [51] is a generalizable shape auto-encoding method. It employs transformers to predict the shape latent code, rather than getting it by optimization (shape auto-decoding).The input is on-surface point clouds. We note that NGLOD is a shape-specific method that overfits one MLP on one testing shape.Besides, we also compare with state-of-the-art shape auto-encoding (point cloud reconstruction) methods. We note these methods are used as reference for understanding the model performance.They are not directly comparable.We train DeepSDF, DeepLS, and CoFie using the same dataset for fair comparisons. NGLOD is trained on each test shape. All methods receive the same inputs during inference.Evaluation Metrics We report the mesh reconstruction error as the chamfer-L 2 distance between the reconstructed and ground-truth meshes. We sample 30000 points to compute the chamfer distances.The meshes are normalized into a unit scale.Qualitative Results. As shown in Fig. 3 Performance on Unseen Categories. We compare CoFie with previous generalizable methods on ShapeNet unseen categories and the state-of-the-art per-shape-based method on the challenging real scans. We provide visualization results in Fig. 5 .• ShapeNet Unseen Categories. As shown in Table 2 , CoFie achieves better generalization on 9 out of 10 novel shape categories. We also observe that the performance gap between CoFie and prior works is larger in the unseen categories, showing the strong generalization capability of CoFie. • Thingi Real Shapes. As shown in Table 3 , CoFie achieves comparable results with NGLOD.We note that NGLOD is a per-shape-based method, which trains a model for each shape and performs better naturally. In contrast, CoFie is trained on ShapeNet shapes.As shown in Table 4 , we experiment with CoFie variants to validate the effectiveness of our coordinate field and MLP designs.Coordinate Field and Initialization. As shown in Table 4 (1), using coordinate fields with different initialization strategies can both reduce the shape error. In detail, when using axis-aligned coordinate field initialization, where all coordinate frames are initialized as the world frame, the shape error reduced slightly from 3.91 to 3.45. The result demonstrates the difficulty of optimizing coordinate frames. In contrast, when using geometry-aware initialization, i.e., initializing local frames with estimated normal and tangent directions of local shapes, the shape error is reduced to 2.33, observing a 40% improvement.MLP Design. As shown in Table 4 (2), using a quadratic layer as the last layer of the MLP observes a 0.9 (23% relatively) reduction of shape error. As the use of the quadratic layer introduces additional parameters, we compare it with a variant for a fair comparison. In detail, we compare it with a linear MLP with an additional layer (6 layers in total), where the two MLPs have the same amount of parameters because the output channel size of the last layer is 1. The result shows that increasing the number of linear layers can only reduce the shape error slightly.Moreover, Table 4 (3) demonstrates the combination of the two introduced techniques can jointly reduce the shape error. 
conclusions and future work: 
 This paper has introduced CoFie, a novel neural surface representation. It is based on the theoretical results of using a ReLU-based MLP to encode geometric shapes. The results strongly motivate the use of local coordinate frames, which encompass the coordinate fields, to transform a point before decoding its SDF value using an MLP. This leads to a hybrid representation combined with coordinate frames associated with local voxels. The experimental results show a strong generalization behavior of CoFie in new instances for shape reconstruction, which significantly outperforms previous generalizable methods and achieves comparable results to shape-specific methods.Limitations. One limitation of CoFie is that it is based on local shapes and cannot be used for the shape completion task. Different from DeepSDF, which learns global shape priors and can fill the large missing components in the input, CoFie is restricted to observable parts. We plan to incorporate more global priors into CoFie. Besides, with a fixed cell resolution, the local shape analysis is broken when a local cell intersects with thin structures. We plan to extend it with adaptive local cell resolutions.Broader Impact. CoFie is a neural surface representation, which have the potential to be used for 3D reconstruction and generation. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We have discussed the broader impacts on the last page of the paper.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper on shape representation does not have the risk of being misused.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? • The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: We will release the code with documentation.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A Proof of Prop. 1Let (u, v) be the parameters of the closest point of p = (x, y, z) T on f (u, v). We have the following constraints on (u, v):EQUATIONEQUATIONNote thatf u (u, v) = (1, 0, au + bv) T f v (u, v) = (0, 1, bu + cv) TIgnoring quadratic-and-higher order terms in u,v,x, y, and z in ( 10) and ( 11), we haveEQUATION)This leads toEQUATIONThe normal direction at (u, v) isEQUATION)The signed-distance function of p to f (u, v) is given byEQUATIONSubstituting ( 15), ( 14) into ( 16) and ignoring third-and-higher terms in u, v, x, y, z, we haved(p, f (u, v)) ≈ -(x -u)(au + bv) -(y -v)(bu + cv) + z - 1 2 (au 2 + 2buv + cv 2 ) ≈ z - 1 2 (au 2 + 2buv + cv 2 ) □We consider the intersection of two quadratic patches where the intersection is along the y-axis. In this case, we can define the surface patch asEQUATIONIn (17) , we do not have any linear term in v, so that the normals to these two patches at (0, 0, 0) T are in the xz plane. In addition, the coefficients in front of v 2 are identical, so these two patches stitch along u = 0.The following proposition provides an approximation to the SDF function of f (u, v).Proposition 2 For each point p = (x, y, z) T in the neighborhood of the origin o, the signed distance function from p to f (u, v) can be approximated asEQUATION)The proof is very similar to that of Prop. 1. When x ≥ 0, the parameters (u, v) of the closest point satisfy u ≥ 0, and vice versa. Therefore, the proof applies the description in Section A. □We will show that there are nontrivial local minima due to symmetries induced by the rotation group. However, those local minima do not recover the underlying ground-truth shape. As a result, they force the network to learn the wrong patterns from the data. For simplicity, we focus on the 2D setting. The extension to 3D is straightforward.In 2D, we assume that the underlying curve is (x, k 0 x 2 ). SDF samples are given by (x, k 0 x 2 + y, y) where x ∼ p, y ∼ q. Consider the 2D rigid pose parametersθ, t x , t y . Let k be the curve parameter.Our goal is to optimize parameters θ, t x , t y , k to minimize the following L 2 reconstruction loss:EQUATIONClearly, (k 0 , 0, 0, 0) is a global minimum of r. The following proposition shows that there is another local minimum of r.Proposition 3 Suppose p and q are independent, andE x∼p x = 0.Then (-k 0 , 0, 2c, π) is a critical point of r, where c = E y∼q y. In addition, it is a local minimum of r if we assumeE x∼p x 3 = E x∼p x 5 = 0, |y| ≪ |x|.We defer the proof of Prop. 3 to Appendix C.1. Prop. 3 shows that there is a non-trivial critical point whose parameters depend on the sampling pattern. As neural network training mostly uses firstorder methods that can be trapped into critical points, this means that without careful initialization, the network will memorize non-shape-related patterns from data, and significantly impairs the generalization ability of the resulting network.C.1 Proof of Prop. 3 Denote l(x, y, k, t x , t y , θ) = sin(θ)x + cos(θ)(k 0 x 2 + y) + t y - k cos(θ)x -sin(θ)(k 0 x 2 + y) + t x 2 -y.It is easy to check that l(x, y, -k 0 , 0, 2c, π) = 2c -2y.The first-order gradients of l with respect to k, t x , t y , θ are given byEQUATIONEQUATIONEQUATIONEQUATIONTherefore, we have ∂r ∂k(-k 0 , 0, 2c, π) = E x∼p E y∼q ∂l ∂k (x, y, -k 0 , 0, 2c, π)l(x, y, -k 0 , 0, 2c, π) = -E x∼p E y∼q (2c -2y)x 2 = 0,and ∂r ∂t x (-k 0 , 0, 2c, π) = E x∼p E y∼q ∂l ∂t x (x, y, -k 0 , 0, 2c, π)l(x, y, -k 0 , 0, 2c, π) = E x∼p E y∼q (2c -2y)kx = 0,and ∂r ∂t y (-k 0 , 0, 2c, π) = E x∼p E y∼q ∂l ∂t y (x, y, -k 0 , 0, 2c, π)l(x, y, -k 0 , 0, 2c, π) = E x∼p E y∼q (2c -2y) = 0,and ∂r ∂t x (-k 0 , 0, 2c, π) = E x∼p E y∼q ∂l ∂t x (x, y, -k 0 , 0, 2c, π)l(x, y, -k 0 , 0, 2c, π) = -E x∼p E y∼q (2c -2y)(x + k 2 0 x 3 + k 0 xy) = 0.This means that (-k 0 , 0, 2c, π) is a critical point of r. To show that it is indeed a local minimum, we study the second-order derivatives of r. We begin with the second-order derivatives of l. They areEQUATIONEQUATION∂ 2 l ∂ 2 t y (x, y, -k 0 , 0, 2c, π) = 0, ∂ 2 l ∂t y ∂k (x, y, -k 0 , 0, 2c, π) = 0 ∂ 2 l ∂t y ∂t x (x, y, -k 0 , 0, 2c, π) = 0, ∂ 2 l ∂t y ∂θ (x, y, -k 0 , 0, 2c, π) = 0 Note that ∀α, β ∈ {k, t x , t y , θ}, ∂ 2 r ∂α∂β (-k 0 , 0, 2c, π) = E x∼p E y∼q l(x, y, -k 0 , 0, 2c, π) ∂l 2 ∂α∂β (x, y, -k 0 , 0, 2c, π) + ∂l ∂α (x, y, -k 0 , 0, 2c, π) ∂l ∂β (x, y, -k 0 , 0, 2c, π) Denote V i x = E x∼p x i , V i y = E y∼q y i .We have∂ 2 r ∂ 2 θ (-k 0 , 0, 2c, π) = V 2 x + 2k 0 c(V 2 x + 2V 2 y ) + (2 + 4V 2 x k 2 0 )(c 2 -V 2 y ) -4k 0 V 3 y + k 2 0 (2V 4 x + V 2 x V 2 y ) + 2k 3 0 cV 4 x + k 4 0 V 6 x ∂ 2 r ∂θ∂t x (-k 0 , 0, 2c, π) = k 0 V 2 x + k 2 0 V 4 x + k 0 cV 2 x + 4(c 2 -V 2 y ) ∂ 2 r ∂ 2 t x (-k 0 , 0, 2c, π) = k 2 0 V 2 x ,andEQUATION∂ 2 r ∂ 2 t y (-k 0 , 0, 2c, π) = 1, ∂ 2 r ∂t y ∂k (-k 0 , 0, 2c, π) = -V 2 x ∂ 2 r ∂t y ∂t x (-k 0 , 0, 2c, π) = 0, ∂ 2 r ∂t y ∂θ (-k 0 , 0, 2c, π) = 0It remains to show thatEQUATIONandEQUATIONThe difference between the left and right-hand sides of ( 24)k 2 0 V 2 x V 2 x + 2k 0 c(V 2 x + 2V 2 y ) + (2 + 4V 2 x k 2 0 )(c 2 -V 2 y ) -4k 0 V 3 y + k 2 0 (2V 4 x + V 2 x V 2 y ) + 2k 3 0 cV 4 x + k 4 0 V 6 x -V 2 x + k 2 0 V 4 x 2 -k 0 cV 2 x + 4(c 2 -V 2 y ) 2 -2 V 2 x + k 2 0 V 4 x k 0 cV 2 x + 4(c 2 -V 2 y ) =k 2 0 k 4 0 V 2 x V 6 x -V 4 x 2 + V 2 x 4k 0 c(3V 2 y -2c 2 ) + (6 + 5V 2 x k 2 0 )(V 2 y -c 2 ) -4k 0 V 3 y + 2k 2 0 V 4 x -16(c 2 -V 2 y ) 2As y ≪ x, the above quantity is above zero ifV 2 x V 6 x > V 4x which can be derived from Cauchy inequality. ( 25) is equivalent toV 4 x > V x 2 .which can be derived from the Cauchy inequality.Visualization . We include more visualization comparisons. We show the comparison with generalizable methods and scene-specific methods in Fig. 6 and Fig. 7 , respectively. We also include a failure case of CoFie in Fig. 8 .Quantitative Results. We include a more comprehensive comparison with generalizable shape auto-encoding (GAE) and shape-specific auto-decoding (SSAD) methods for understanding the performance of our model. Again, we note CoFie performs generalizable shape auto-decoding (GAD) and is not directly comparable to these models.Table 5 : Performance on ShapeNet 10 novel categories. Specifically, the reported 3DS2VS [51] and NKSR [15] are trained on the full set of the training categories. In contrast, the reported numebrs in the main paper use a subset of 1000 instances for training. Figure 7 : Compare with the shape-specific method NGLOD on Thingi shapes. We show two images for each method, one for the overall shape quality, and a zoom-in detail check. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The paper received a very large number of posts between authors and reviewers after the rebuttal, and also between reviewers during the post-rebuttal discussion. All reviewers agreed on the technical and intellectual novelty of the paper. The AC also read the paper and appreciated the geometric insight of attaching learnable local coordinate frames to each patch and approximating the surface through a composition of local, frame-aligned MLPs with quadratic layers. Two of the reviewers remained negative because of limitations in the evaluation i.e., the evaluation should be better done in the generalized auto-encoding setting and preferably on full ShapeNet. The AC agrees that the evaluation could have been more bulletproof, yet, the novelty and geometric insights deserve recognition. The AC recommends acceptance, yet urges the authors to incorporate their revised comparisons and additional experiments in larger datasets in the final version of the paper! The authors should also acknowledge the limitations of their evaluation, including the limitation of using shallow MLPs with ReLU activations in the comparisons (instead of more modern variants, such as GLUs/GEGLUs). The paper should also correct issues with citations, and also add citations to more recent works on 3D neural generative geometric representations e.g., CLAY, Make-A-Shape, GEM3D, Mosaic-SDF.

Reviews: review: paper_summary: paper_summary: The paper proposes a local prior based method where the model is trained on a dataset and learns a prior over local patches of the shapes, and then at test time reconstructs patches by optimising to minimise the reconstruction error w.r.t. the trained model's prior. While this has already been done in the literature, they propose to learn a coordinate transformation for each local region and initialise the coordinate transformation based on the geometry in that region. They also use a quadratic layer at the end of the network, inspired by their analysis of representing local patches by a quadratic surface.
strength_weakness: strength_weakness: - A lot of things are not made clear (asked in the questions) which makes it hard to understand the method
- As mentioned in the strengths, it seems that the benefit of this method is that the coordinate field (and especially its init) it enables the latent based model to train better, rather than being a better representation. This could be explored more deeply. Also it makes the question below ("It seems the proposed method has no regularization, is there a reason for this?")
questions: questions: - The method is called coordinate field, but it is not actually a true coordinate field: the coordinate transformation is learned discretely as parameters for each voxel, not continuously throughout the domain?
- Usually shape autodecoding uses regularization on the latent parameters so then when doing test time latent optimisation the latent parameters can be initialized to zero and be "close" to many possible shapes. It seems the proposed method has no regularization, is there a reason for this?
- Line 219: "compute the derivatives of the SDF" however at test time you don't have access to the full SDF (or do you?), so how to you get well oriented normals for initialization?
- Would like more information about what SDF supervision exactly is given at test time. Is it just the surface points, or is assumed the SDF is known at test time as well?
- Would like timing performance of test time optimization (general ballpark and if this compares to other methods)
- Line 243: two types of methods or three types of methods?
- Missing ref on line 114, line 128 and line 197
limitations: limitations: - Yes, limitations have been discussed adequately.
paper_summary: paper_summary: Paper proposes CoFie — 3D shape representation as a set of latents arranged on a regular voxel grid. Each latent encodes an oriented local quadratic patch. This local oriented patch defines local SDF in a local coordinate frame which is decoded via conditional MLP for which the last layer is quadratic: it defines a bilnear form on input vector instead of linear mapping). SDF value for query point x is decided based on the SDF value of local implicit function belonging to the nearest voxel and transformed from local coordinate frame to global. 

Authors train proposed methods and baselines on a single random subsample of 1000 ShapeNet shapes (5 categories, 200 shapes each) and evaluate on holdout ShapeNet shapes and out-of-distribution shapes from Thingi10K. Proposed method outperforms baselines trained in the same limited data regime. The same ShapeNet model is also evaluated with the Neural Geometric Level of Detail method and DeepSDF fitted in the shape overfitting scenario.
strength_weakness: strength_weakness: — Proposed method seems to be very similar to AutoSDF and SDFusion: both methods utilize a regular latent grid that encodes local SDF that is used to infer the global SDF. Main difference seems to be in a local SDF decoder that is quadratic (last layer) for proposed method and also encodes SDF in the local coordinate frame (SDFusion and AutoSDF use global coordinates). Overall, it is fine as long as proposed method compares to these similar methods but evaluation of the paper is limited (see below). 

— Similarly to other regular latent grids methods (e.g. SDFusion, AutoSDF), this model scales cubically with respect to grid resolution and thus might not be well suited for representation of topologically challenging shapes with thin parts. 

— Evaluation of the paper is extremely limited. For point cloud reconstruction (auto-encoding), proposed methods and baselines are trained on 5 ShapeNet categories with 200 shapes per category (overall 1000 shapes). This evaluation is not enough to support the claim that CoFie “can represent arbitrary shapes that belong to any novel category” (LL52-53). These results might only indicate that CoFie might be better representation in low-data regime but in this case evaluation should be done similar to few-shot learning setting: all models should be trained on several different subsets of ShapeNet (e.g. 5) and average/std of test errors should be reported. I also want to note that 3D2VS trained on full ShapeNet is an extremely strong baseline that achieves almost perfect reconstruction quality for some categories (like airplanes). Also, the paper only uses Chamfer distance for evaluation and ignores other common measures like IoU and surface F-Score. 


— The choice of baselines is also limited. DeepSDF is a 2019 paper that uses a very simple Pointnet (not even PointNet++) encoder. DeepLS is a 2020 paper. 3D2VS seems to be the only recent baseline for 3D point cloud reconstruction. It is a very strong baseline but authors have not used pretrained models and trained their version on limited data instead. Given the fact that 3D2VS uses a very high capacity attention-based decoder, this training regime might not be a fair comparison because attention-based models often struggle in low data regimes. For the same reason, 3D2VS might not be a good baseline if the goal of the paper is to show that their representation is compact. In this case, 3DILG might be a more suitable and strong baseline (see below) because it uses a simpler local patch encoder (Pointnet), so it might work better in a limited data regime. Since the model utilizes local surface patches similarly to AtlasNet (see links below), it also can be a strong baseline, especially in low data setting. 

— Evaluation in an overfitting setting  does not seem methodologically correct. : NGLOD and DeepSDF overfit to single shapes while CoFie was pretrained trained on 1000 shapes. These are completely different settings: overfitting regime measures capability of the model to efficiently fit one shape with preservation of geometric detail, and fitting and evaluation on collection of shapes evaluates the ability of the model to generalize to unseen shapes. This experiment should either be done if fully overfitting setting (similar to NGLOD paper) or in a reconstruction setting (see concerns above). 

— Quadratic MLP contribution seems a little bit weak to me. It looks like it is basically equivalent to linear MLP being run on quadratic feature expansion and in this case commonly fourier embeddings of points can be a very strong alternative (this is not tested). 

— Paper writing can be significantly improved in clarity. For example, paper does not specify input to auto-encoder for baselines. Is it point cloud or mesh? If it is a point cloud, what is the density of sampling and what sampling was used (FPS, random, poisson disk sampling, etc). How many codes were used to train 3D2VS? 512? What were hyperparameters for baseline training? On the other hand, paper spends a lot of space describing relatively common knowledge like MLP with ReLU activations (LL172-177). Another example is equation (5). On first glance it appears as weighted average, but since authors use voxel indicator function as weight, it means that for each query point x, global implicit function is based only on implicit function of nearest voxel grid latent. 

— Some statements in the paper are not correct. For example, the paper states that 3D2VS “employs transformers to predict the shape latent code” (LL258-260) . This is not factually correct: 3D2VS does not represent shape as one vector, it uses latent code clouds (usually 512 codes per shape). 

— Representation seems to be computationally expensive. Training on 1000 shapes takes one day on 4 GPUs with 24 GB VRAM. 

— Some relevant work is missing

AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation https://arxiv.org/abs/2203.09516 

SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation
https://arxiv.org/abs/2212.04493 

4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks https://arxiv.org/abs/1904.08755 

3DILG: Irregular Latent Grids for 3D Generative Modeling https://arxiv.org/abs/2205.13914 


AtlasNet: A Papier-Mâché Approach to Learning 3D Surface Generation https://arxiv.org/abs/1802.05384
questions: questions: — What is the input for the model for auto-encoding settings? Is it a point cloud? Or model back propagates into learnable latent code like DeepSDF? If it is a point cloud, what encoder model is used? If model back propagates into latent code, how come it is generalizable?

— How were baselines trained? Were they trained on the same inputs as the proposed method? Did all methods use the same SDF sampling? As far as I know, 3D2VS uses occupancy as supervision. Was it trained with SDF or occupancy supervision for the paper? 

— What motivated size 1000 for a training dataset? Majority of modern shape auto-encoder train either on full ShapeNet of full ShapeNet categories (e.g. ~6000 ShapeNet chairs). If 5 different subsets of ShapeNet of size 1000 would be selected (same categories), how high would be the variance in evaluation across these subsets? I have strong suspicion that results shown int Tables 1 and Tables 2 might not generalize across training subsamples. 

— Have you tried using Fourier embeddings of input coordinates instead of quadratic layers? This might be a very strong alternative since it helps coordinate based models to encode high-frequency details very efficiently.
limitations: limitations: — Proposed method represents the shape as a set of latents arranged on a regular latent grid and thus scales cubically with latent grid resolution. This limits method’s ability to efficiently encode topologically challenging shapes. 

— Method requires a lot of compute to fit 1000 shapes (1 day on 4GPUs with 24GB VRAM). This might be due to fact that authors use 32^3 latent grid (SDFusion for example uses 8^3). This severely limits scalability of the proposed method even for training on full ShapeNet, not to mention larger datasets like Objaverse.
paper_summary: paper_summary: This paper introduces CoFie, a novel neural surface representation method designed to efficiently learn and represent complex 3D shapes. CoFie addresses the challenges of existing methods by introducing a coordinate field that optimizes the representation of local shapes, significantly reducing shape errors and improving the efficiency of parameter usage. The method is based on a hierarchical representation with coarse and fine-grained geometry, utilizing MLP with quadratic layers to enhance the expressiveness of the model. Experimental results demonstrate CoFie's strong generalization capabilities in shape reconstruction tasks, outperforming previous methods in terms of accuracy and efficiency.
strength_weakness: strength_weakness: 1.CoFie's performance in detailing is strongly related to the cell size, which hinders its ability to represent fine-grained details. 
2.Representations based on nonoverlapping local patches may not favor expressing rich colors, a capability that implicit representations possess. 
3.It excels in displaying regular shapes, such as furniture, but is weak in showcasing shapes with rich curves.
4. Line 197, citation is missing
questions: questions: I noticed that the article mentions, "We train CoFie on 1000 shape instances sampled from ShapeNet [5] consisting of chairs, planes, tables, lamps, and sofas (200 instances for each category)." Why not train on the entire ShapeNet?
limitations: limitations: Yes
paper_summary: paper_summary: ### Motivation
- In the realm of implicit 3D shape representations, local-based solutions [4, 19, 32] (which decomposes the target shape into a set of local surfaces to model) provide higher accuracy but at the cost of a higher number of parameters to optimize. 
- The authors argues that this number of parameters can be reduced by disentangling the actual geometry and the transformation (orientation, translation) of these local patches, modeling the latter separately.

### Contributions
- The authors thus introduce CoFie, a novel approach that utilizes an explicit coordinate field to transform local shapes, aligning them to reduce spatial complexity and improving MLP-based implicit local representation of 3D shapes.
- Various theoretical and empirical evidences are provided w.r.t. the benefits of their formulation, as well as to justify other technical choices, e.g., the usage of quadratic layers to complement their MLPs.

### Results
- Experiments show that CoFie reduces the shape error by 48% to 56% compared to traditional generalizable methods (e.g., DeepSF [4], DeepLS [24]) and achieves comparable performance to prior works with 70% fewer parameters.

### Relevance
- This paper addresses the challenge of balancing accuracy and compactness in neural surface representations, a relevant and generic issue in 3D shape modeling.
strength_weakness: strength_weakness: ### W1. Somewhat Limited Comparison to SOTA
- The authors compare to 4 methods (DeepSDF [24], DeepLS [4], NGLOD [30], 3DS2VS [41]), 2 of which are fairly old for the domain (<2020 for [24] and [4]). The literature has many more recent approaches, e.g. [a,b,c], with code or results pre-available, which could have been considered.
- Moreover, the authors compare to only one shape-specific method (i.e., performing test-time optimization), NGLOD [30]. While such methods tackle a slightly different task, comparing to a single solution makes it hard to draw solid conclusions.

### W2. Partial Evaluation of "Compactness" Claims
- The authors claim that "CoFie achieves comparable results with prior work using 70% less parameters" [L56-57]. However, there is a distinction to be made between parameter number (compactness of the model), convergence speed (number of iterations to convergence), and inference time / computational footprint. I find that claiming some improvement along one of those dimensions without studying the impact on the others not so meaningful. 

### W3. Code Not Available
- While the authors claim that they are "committed to releasing our code" [L16], no code has been provided yet to reviewers.


### W4. Minor Remarks
- Citations are missing (empty brackets) in several places [L114, L128, L197].
- Typos: "practival: $\rightarrow$ "practical" [L118] ; "Perfomance" $\rightarrow$ "Performance" [L270] ; etc.


------------
### Additional References:

[a] Ye, Jianglong, et al. "Gifs: Neural implicit function for general shape representation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[b] Wang, Li, et al. "Hsdf: Hybrid sign and distance field for modeling surfaces with arbitrary topologies." Advances in Neural Information Processing Systems 35 (2022): 32172-32185.

[c] Lu, Yujie, et al. "Unsigned Orthogonal Distance Fields: An Accurate Neural Implicit Representation for Diverse 3D Shapes." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
questions: questions: _see **Weaknesses** for key questions/suggestions._

### Q1. Computational Cost
- Besides the points made in **W2** above, I wonder what the computational cost of replacing linear layers with quadratic ones is (c.f. [L178-187])?

### Q2. Grid Size and Surface Complexity
- What is the impact of the voxel grid size on the surface complexity / ability of the model to learn the surface? Intuitively, the larger each voxel is, the more complex/discontinuous the surface that it intersects can become.

### Q3. Purpose of $w$
- What is the point of introducing $w$ [L168-171] if in practice it is set to the indicator function (simply selecting the voxel $v$ where $x$ belongs)? Is there a point to keeping Equation 5 this generic?

### Q4. Hyper-parameter $k$
- The authors mention that setting the number $k$ of quadratic layers (following $L-k$ linear ones) to 1 yields the best performance [L187]. Could the authors share the quantitative evaluation behind this conclusion?
limitations: limitations: - Limitations are discussed in the conclusion, with the authors sharing meaningful insight (e.g., the solution cannot perform shape completion or handle thin structures well).
- Societal impact are however not explicitly discussed. Providing some potential positive or negative applications of this work would be meaningful.
paper_summary: paper_summary: This paper proposes a new architecture for SDF auto-decoding task. It divides the whole SDF points into voxels and builds local coordinates for each surface patch included in valid voxels. Respective shape coding is learned separately and a generalizable MLP is used for SDF decoding. For MLP, a quadratic layer is proposed to better fit SDF values. Compared to the selected baselines, the proposed method shows great performance on both synthetic and real datasets.
strength_weakness: strength_weakness: The evaluation on the main assertions of this paper is not thorough. The method should be evaluated in three levels – accuracy, scalability and generalizability. 

1. The first one has been partially demonstrated by comparing with the selected baselines. However, here a SOTA baseline - Neural Kernel Surface Reconstruction (CVPR’23) - is missing.
 
2. Scalability is not studied by the paper. Only object level datasets are tried. Large scale datasets such as CARLA should also be evaluated to demonstrate the ability of the model. This is a reasonable experiment for a locally coded model, because it only needs to introduce more and more patches for larger cases. 

3. Generalizability is one of the selling points for this paper. However, there are some issues related to the baselines and experiment settings. 

First of all, the paper states that the baseline NGLOD is a per-scene model, not a generalizable model, so NGLOD naturally could have better performance. However, NGLOD also has generalization ability according to the original paper, only LOD features are trained per-scene while keeping the MLP fixed, which is actually a similar setting to this paper. 

Secondly, there are two kinds of generalizability: across domain and across density. In this paper only generalizability across domain, i.e. object level synthetic to real, is tried. Scene level, for example ShapeNet and Synthetic Room to ScanNet or Matterport3D, and generalization across density, that is trained and inferred on different sampling density, are missing.

4. Besides the experiments, there is one problem in theory derivation: 

In the proof of proposition 1, equation 14 seems unreasonable to be equal. It is suggested to explain to what extend these two expressions could be regarded as equal. If equation 14 is not equal, how the final equation could be held?

5. Missing reference(s): Line 114, missing reference(s) for using quadratic surface patch
questions: questions: Besides the weaknesses, there is one more question: currently all points in one voxel are assumed to be related to the surface within the same voxel, what if the points near the boundary of the voxel are related to the surface included in the nearby voxel?
limitations: limitations: Refer to weaknesses and questions.

Comments: title: Additional Comments
final reply to authors
Response to comments #2
Response to main rebuttal
comment: Dear All, 

             We think the reviewers are split into two groups. We do understand the reasons. On one hand, 3D neural representations and 3D generation are quite crowded areas now. To publish a paper, one has to have amazing results and compare to many baselines. This was the reason of two negative reviews. We do admit that we did not go to the level of typical papers in this area now in terms of baseline comparisons.

            However, we believe that our paper opens a new direction in 3D representations which seek to develop geometry-aware representations. All existing work in this space follow 2D strategies under some 3D representations. These approaches, in our view, are black box. We have little understanding on what are learned. The only way to convince the reviewers is to show amazing figures and extensive comparisons, many of those have overfitting behaviors. From a scientific point of view, we do not think this is necessarily good thing. A paper should not be evaluated just on experimental results. Our approach is based on  simple and effective geometric insights, which are based on rigorous analysis. We hope it will simulate future research in this direction. Most of questions raised by the reviewers can be done as follow-up work either by ourself or others.

           If this paper was done 2-3 years ago, then we would not have these evaluation issues. However, we should not be penalized because of this, as it opens a new direction in 3D representations.
Thanks for the feedback. We would like to conclude the discuss as follows.

> Tables 1-2 provide comparison with either very old models (DeepSDF, 2019; DeepLS, 2020) or with models that were not designed for GAD setting(3DS2VS).

- We agree that DeepSDF and DeepLS are old methods. In the meantime, we kindly remind that **they are the only two directly comparable methods for GAD** as far as we know. We are happy to compare with any baseline for GAD if you could provide any references. Whether DeepSDF and DeepLS are old is orthogonal to our contributions in this work, as our contributions are ablated.
- **We note we have mentioned many times GAE and SSAD methods are used as reference and they are not directly comparable methods.** See Line 243 for categorizing generalizable and shape-specific models. In Line 247 we clarify the setting of generalizable shape auto-encoding and noted it is not directly comparable. We also note classify each method, as in Line 251, 253, 255, 258. 
- **The reviewer got an wrong impression by him/herself and urged us comparing with GAE methods** because him/herself works on GAE (mentioned as "as a person who trained reconstruction and generative models on full ShapeNet", where "reconstruction" is GAE)  and this leads to the biased reviews towards GAE. **Again, we never claim we should compare GAD methods to GAE and SSAD methods. Instead, this is the understanding of the reviewer.** Please see the initial review of V5fn for the details of requesting GAE method experiments.

> Even if we consider GAD setting, models are evaluated based on training on 1000 shapes.
- We acknowledge that we trained the model with 1000 shapes.
- **Using 1000 shapes is standard in prior works.** See DeepLS.
- At the same time, we would like to point the review to the results provided during the rebuttal period of training CoFie on 20K shapes.
- **The new experiment demonstrated that CoFie shows good performance in the context for both low-data and larger-data regime.**

> Choice of baselines in Table 3 has the same issue as choice of baselines in Tables 1-2.
- **This is your own fault for comparing these methods together.** Again, we note **we have mentioned many times** GAE and SSAD methods are used as reference and they are not directly comparable methods. See ine 243 for categorizing generalizable and shape-specific models. In Line 247 we clarify the setting of generalizable shape auto-encoding and noted it is not directly comparable. We also note classify each method, as in Line 251, 253, 255, 258. 
- The reviewer didn't realize the differences between GAE, GAD, SSAD until the last reply. Even after he/she realized this, he/she still attacked us using his/her own wrong understanding.
- Again, at the very beginning, we note we have mentioned many times GAE and SSAD methods are used as reference and they are not directly comparable methods. See ine 243 for categorizing generalizable and shape-specific models. In Line 247 we clarify the setting of generalizable shape auto-encoding and noted it is not directly comparable. We also note classify each method, as in Line 251, 253, 255, 258.
I think we have reached limits of constructive discussion and further engagement will be waste of everyone's time. 

I am not going to change my rating. I strongly believe that evaluation of the paper is flawed and if we accept it in the current state, we are going to normalize this evaluation and thus will create a bad precedent for follow-up work. Authors have failed to convince me that it is not the case. Below is summary of my concerns based on my initial review and discussion. 

**1. Tables 1-2 provide comparison with either very old models (DeepSDF, 2019; DeepLS, 2020) or with models that were not designed for GAD setting(3DS2VS)**. As I pointed out in my initial review and many times during the discussion, 3DS2VS was designed for GAE setting and authors acknowledged this in the discussion. I think that choice of baselines in this experiment is weak and favors proposed method. I think that evaluation should be done in GAE setting (preferably on full ShapeNet) because GAD setting is rarely used these days and majority of novel shape representation methods evaluate in GAE setting. 

**2. Even if we consider GAD setting, models are evaluated based on training on 1000 shapes.** This is a low data regime with potentially high variance in evaluation results if training and evaluation is done using different samples of 1000 shapes. I don't think that this evaluation is enough to support the claim that proposed method generalizes better than (weak) baselines. Even if paper uses low-data regime setting for evaluation, results should be presented for at least 5 different runs on different subsamples of the ShapeNet. 

**3. Choice of baselines in Table 3 has the same issue as choice of baselines in Tables 1-2**. NGLOD is SSAD model, DeepLS and Cofie are GAD models. Again, this is not a correct comparison and thus is not enough to support claims in the paper. I think that either proposed method should be evaluated in SSAD setting, or this evaluation should be removed. 

I am not going to reply to further comments by authors. I think that this thread has provided enough feedback to both authors to revise their submission and other reviewers/ACs to help reach final decisions.
Dear Reviewer 58nR,

We have updated the discussion with reviewer V5fn, where we added GAE baselines trained with larger data (3DS2VS/NKSR with 20K shapes), and also added SoTA SSAD method UODFs. Wo hope these results could address your concerns on evaluation.

---

For your comment
> I believe the current version clearly needs significant revisions to establish a decent baseline for future works, and therefore I would keep my negative score.

We believe our content doesn't need significant revisions. The truly important changes are just i) reporting results for 3DS2VS trained with 20K shapes, ii) reporting results for NKSR and UODFs, which are newly added baselines, iii) discussing these methods a bit more. All other content in the rebuttal and discussion are trying to resolving the questions from the reviewers.

**Besides, we believe this is not a solid reason to keep the negative score just because we need to update our content. Otherwise, why we need this discussion and rebuttal period?** We hope the ACs could take this into consideration.

---

Finally, we tried to resolve your concerns on accuracy (we provide NKSR results as you requested, and we also have other new baselines in the reply to other reviewers), scalability (we committed experiments on scene-level data in the future version), and generalizability (we clarified the different definition os generalization in NGLOD and our paper, which leads confusion to the reviewer). We hope these could be considered. The only one of your concerns on scalability is not directly addressed by experiments. At the same time, scalability to unbounded scenes is not a core property we focus in this paper, and we hope our contributions of i) theoretical analysis of local shape patches and ii) our coordinate field and its initialization could be considered.
Thanks for the reply. We would like to conclude the latest discussion as follows.

- We are happy the reviewer finally agree with us that GAE, GAD, SSAD methods are not directly comparable.
- For the GAE method 3DS2VS, training it on 1000 shapes in the original paper is not convincing enough. The reviewer think the training set is too small and suggested us to train with larger data. We agreed with this point and we provided new results.

With the updated content, what we achieved is
- Best GAD performance, no matter in low-data (1000 training shape) or larger-data (20K, added during rebuttal) regime.
- Similar performance with SoTA GAE and SSAD methods (3DS2VS/NKSR trained with 20K shapes, NGLOD/UODFs). I would say the performance is "similar" rather than "comparable", as they are not directly comparable.

---

Regarding your question why we put NGLOD, DeepLS, CoFie together in tables, we agree this might lead to confusion. However, we have noted in the paper Line 243 for categorizing generalizable and shape-specific models. In Line 247 we clarify the setting of generalizable shape auto-encoding and noted it is not directly comparable. We also note classify each method, as in Line 251, 253, 255, 258. But we appreciate the feedback from the reviewer, and we will also group the methods in the tables for better clarity.

---

Finally, the reviewer suggested 
> I really like the proposed method and I think it might be very powerful if trained on sufficient data in auto-encoding setting.

We would like to clarify that this method is designed for GAD. Thus, it is not applicable to GAE, as the coordinate field need to be updated by optimization (equation 8 and 9). We are not able to supervise the prediction of coordinate field if we use a feed-forward model to do GAE, as we don't have the ground truth for the coordinate field. GAE, GAD and SSAD are different tasks, **requiring the model designed for one task can be applied to other tasks is not rational**. Research on how to build an unified framework for GAE/GAD/SSAD is promising, but this is out of our current scope.

---

For your comment
> Personally, I am not even sure that GAD setting is the proper way to evaluate shape representations (for me, GAE setting offers more practical advantages) 

We agree GAE has the advantage of i) can learn general priors with large model and large data, ii) the learned latent space can be directly used for downstream tasks, e.g. generation. But we would like to clarify that every task has its own value. The GAD task benefits from its compactness, lower requirement on training data and model size.

We appreciate the detailed feedbacks and the insights on GAE method from the reviewer.
Again, I appreciate the effort that went into additional results.

> Similarly, we would like to clarify the two kinds of methods (GAD v.s. SSAD) cannot be directly compared. 

I agree and I never said that the methods should be directly compared. Honestly speaking, I think majority of the discussion we are having right now stems from the fact that initial submission mixes different models in the first place. For example, Tables 1 and 3 has 3DS2VS (GAE model); DeepSDF, DeepLS and Cofie (GAD models). In my opinion, it is not correct comparison on the first place because 3DS2VS was designed as GAE model and authors themselves highlighted many times that GAE and GAD models are not directly comparable. 

Similarly, Table 3 has NGLOD (SSAD model); DeepSDF (or DeepLS judging by update) and Cofie -- both GAD models. Again, authors themselves stress out that these models are not directly comparable (and I agree with that!), yet this comparison is in the current version of the paper. 

In conclusion, I really like the proposed method and I think it might be very powerful if trained on sufficient data in auto-encoding setting. But evaluation in the current iteration of the method feels all over the place: GAE and GAD models are mixed for evaluation; GAD an SSAD models are mixed as well. Personally, I am not even sure that GAD setting is the proper way to evaluate shape representations (for me, GAE setting offers more practical advantages) but this does not hold a weight in my decision. 

I also want to stress out again that I really appreciate the effort the authors put in this discussion.
Again, I really appreciate the commitment of the authors to the rebuttal and for providing additional details. 

> Number of parameters for compared models. 

I appreciate this clarification. This again confirms my concerns about evaluation and the choice of baselines. 3DS2VS has a very hight capacity decoder -- 200x larger than Cofie or DeepLS! To me, it means that might really struggle in low data regimes due to overfitting and thus training and evaluation on larger dataset is needed. 

> In our submission, we train 3DS2VS with the 1000 shape for a fair comparison with GAD methods, and using 1000 shapes for training GAD methods are adopted in prior works, e.g. DeepLS.

See above. With such high capacity of the decoder, 3DS2VS can memorize all separate local geometries for 1000 training shapes hurting its ability to generalize. This model was never designed  for auto-decoding task or for low-data regime. Yet, it achieves very strong performance in auto-encoding task. I think that 3DS2VS is either not a good baseline for auto-decoding or all models should be evaluated in auto-encoding setting. 

> Additional results on 20K ShapeNet shapes in GAE and GAD settings.

I appreciate the effort that went into these experiments. Unfortunately, I still think they are not convincing. As authors pointed out many times in previous replies GAE setting cannot be directly compared to GAD setting and I agree with this notion.

> We additionally note that CoFie (generalizable shape auto-decoding) and 3DS2VS/NKSR (generalizable shape auto-encoding) are not directly comparable. The comparison above is in the context that the reviewer requested this comparison. 

I have not requested a comparison of GAE and GAD setting. My point was that 3DS2VS was designed as GAE model and its evaluation in GAD setting might not be fair comparison for 3DS2VS. My another point was that GAE models can very powerful if trained on sufficient data (GAE setting) and that maybe evaluation of the proposed method should be done in this setting instead of GAD. 

> The results implies that shape auto-encoding methods, e.g. 3DS2VS, suffer from limited generalization capability on novel categories, as the training distribution doesn't offer prior knowledge of them.

In my experience, this statement might need additional empirical evidence. First, 3DS2VS architecture is based in on latent grid/array/cloud that learns local geometries which fosters generalization capabilities due to learning of local, simpler, geometric priors instead of global ones (Cofie utilizes the same powerful idea). Second, I have tested 3DS2VS and other auto-encoding methods based on latent grids in challenging out-of-distribution setting: models were trained on full ShapeNet and tested on large dataset of high-quality out-of-distribution meshes. 3DS2VS showed very strong results in this setting.
Thanks for the very detailed rebuttal materials. I'm convinced that the proposed method is novel and could be potential to better represent 3D shapes. Nevertheless, my concerns regarding the three levels (accuracy, scalability and generalizability) of evaluation are just partially addressed. In the meantime, I also generally agree with reviewer V5fn that the paper needs more solid and fairer evaluations. Given the whole set of comments and newly added materials, I believe the current version clearly needs significant revisions to establish a decent baseline for future works, and therefore I would keep my negative score.
> Performance of CoFie on Thingi, comparison with NGLOD/UODFs

We would like to clarify NGLOD/UODFs are shape-specific auto-decoding methods. Different from generalizable shape auto-decoding where only the latent shape representation is optimized, NGLOD/UODFs also optimize the model for each specific shape. Thus, shape-specific auto-decoding methods naturally have better performance. We include the comparison of the two settings as follows.

|                             | Latent representation         | Decoder                | Methods                        | Inference speed |
|-----------------------------|-------------------------------|------------------------|--------------------------------|-----------------|
| **Generalizable Auto-Decoding** | Optimized during inference  | Fixed weight after training  | DeepSDF, DeepLS, CoFie         | Minutes         |
| **Shape-specific Auto-decoding** | Optimized during inference | Optimized for each shape | NGLOD, UODFs                   | Hours           |

The close performance between CoFie and NGLOD/UODFs in Table 2R demonstrates the effectiveness of our work. We also report the performance of CoFie trained by 20K shapes as follows based on Table R2, where CoFie is further improved with more training data.

|                            | CoFie (our paper) | CoFie (new) | NGLOD                 | UODFs                 |
|----------------------------|-------------------|-------------|-----------------------|-----------------------|
| **Task**                   | GAD               | GAD         | SSAD                  | SSAD                  |
| **#Training Shapes**       | 1K                | 20K         | Each testing shape    | Each testing shape    |
| **CD**                     | 1.87              | 1.22        | 1.04                  | 0.932                 |
| **gloU**                   | 99.0              | 99.2        | 99.3                  | 99.4                  |
| **Inference Time/Shape**   | 10 min            | 10 min      | 105 min               | 300 min               |

Similarly, we would like to clarify the two kinds of methods (GAD v.s. SSAD) cannot be directly compared. This analysis is performed as it was required by the reviewer for understanding CoFie’s performance.

---

We thank the reviewer for the suggestions on understanding the performance of CoFie in the context of recent generalizable shape auto-encoding and shape-specific auto-decoding works. Although we don't agree the methods of different settings are directly comparable, we hope the analysis could provide better understanding of our model. We also hope the added results and analysis could clarify the effectiveness of CoFie in the context of using more data for training. We appreciate the time and effort spent in the discussion period, and we will incorporate the discussion content and all your suggestions in the revision.
We thank you for the detailed feedback. We will include more clarifications and experiment results to address your concerns.

---

> Model capacity and training data size

We would like to provide more context for the task of generalizable shape auto-decoding. In the table below, we compare the number of parameters for the representative generalizable shape auto-encoding (GAE) and shape auto-decoding (GAD) works.

| Task | Model  | # Params in Encoder | # Params in Decoder |
|------|--------|---------------------|---------------------|
| GAE  | 3DS2VS | 4.2 M               | 101.9 M             |
| GAD  | DeepLS | No Encoder          | 0.05 M             |
| GAD  | CoFie  | No Encoder          | 0.06 M             |

In the context of GAE, we agree that the encoder of 3DS2VS is not large. However, the situation becomes different compared with GAD methods.


---

> NKSR and 3DS2VS training and evaluation

We thank the reviewer for bringing the comment that using small data for training might not be sufficient for GAE methods, e.g. 3DS2VS. In general, we agree with the comment as these GAE methods learn priors of latent shape space under large data (acknowledged in our first rebuttal reply as “We agree that shape auto-encoding requires a large amount of training data as it learns priors to map the shape/point cloud into latent code. ”). In our submission, we train 3DS2VS with the 1000 shape for a fair comparison with GAD methods, and using 1000 shapes for training GAD methods are adopted in prior works, e.g. DeepLS.

In our last rebuttal reply, we have included results of NKSR and 3DS2VS on the full set of 5 training categories in Table R1, where we use 20K shapes for training, which is comparable to original 3DS2VS paper. For a more comprehensive understanding, we include the training setting and performance on ShapeNet 10 **novel categories** as follows. Besides, the NKSR method is also trained with 20K shapes (see Table R2).

|                 | 3DS2VS (our paper) | 3DS2VS (Table R1) | 3DS2VS (original paper) | CoFie (our paper) | CoFie (new) |
|-----------------|--------------------|-------------------|-------------------------|-------------------|-------------|
| **Task**        | GAE                | GAE               | GAE                     | GAD               | GAD         |
| **#Training Shapes** | 1K            | 20K               | 55K                     | 1K                | 20K         |
| **CD**          | 16.0               | 9.30              | /                       | 3.18              | 2.04        |
| **gloU**        | 81.2               | 94.8              | /                       | 98.3              | 98.7        |

The results implies that shape auto-encoding methods, e.g. 3DS2VS, suffer from limited generalization capability on novel categories, as the training distribution doesn't offer prior knowledge of them.

Additionally, we note that we didn’t report performance of 3DS2VS (original paper) as its training categories overlap with our novel testing categories, which cannot be used for evaluating the generalization capability.

---

> Performance of CoFie on ShapeNet, comparison with 3DS2VS/NKSR.

We would like to point the reviewer to Table R1 in our last reply, where our method which trained with 1000 shapes has already demonstrated better performance than 3DS2VS/NKSR trained on 20K shapes. With 20K training shapes, CoFie demonstrates an even better performance.

We additionally note that CoFie (generalizable shape auto-decoding) and 3DS2VS/NKSR (generalizable shape auto-encoding) are not directly comparable. The comparison above is in the context that the reviewer requested this comparison. We include the comparison between the two settings as follows for clarification.

|                         | Input during inference                            | Latent Code                                 | Decoder            | Methods                     | Inference speed |
|-------------------------|---------------------------------------------------|---------------------------------------------|--------------------|-----------------------------|-----------------|
| **Generalizable Auto-Encoding** | On-surface points (w./w.o. additional point normals) | Predicted by the encoder in feed-forward | Large-size decoder | 3DS2VS, NKSR                | Seconds         |
| **Generalizable Auto-Decoding** | SDF samples (not on-surface)                 | Optimized by back-propagation              | Small-size MLP     | DeepSDF, DeepLS, CoFie       | Minutes         |
Again, I appreciate the effort authors are putting in the rebuttal. 

I want to reiterate that I have no doubts about novelty of the proposed method. My main concern is **weak evaluation** of the proposed method. The proposed method either compares to very old (by field's standards) methods or compares to very strong recent methods in a setting that might be unfavorable to them like training on 1000 shapes. When the model compares to strong baselines in fair setting (see Thingi10K results) the proposed method does not outperform them (again, see Thingi10K results). 

Minor comments below:

> 3DS2VS terminology. 

I appreciate authors acknowledging that usage of 'latent code' with respect to 3DS2VS representation can lead to ambiguity, especially for readers who are less familiar with the recent advances in shape representation via latent grids/arrays/clouds. 

> We agree that this is studied in the context of shallow MLP with ReLU and we kindly point the reviewer to DeepSDF, DeepLS, NGLOD where the context is adopted and we directly compare CoFie with them.

This again brings us to the question whether the choice of the baselines of the paper is strong. As I mentioned, a lot of recent methods don't use ReLU and don't even use MLPs that much (for example 3DS2VS uses GEGLU in attention modules that are utilized both in encoder and decoder modules). My point is that 'strong theoretical foundation' for the paper is justified in very limited setting that does not hold in vast majority of practical cases. 

>  We kindly refer the reviewer to Figure 8 in the appendix, where CoFie fails on super challenging thin structures, and we believe this is a common problem for any shape representation.

3DS2VS results (Fig. 8 in the paper) suggest that it is able to handle proposed shapes well if trained on sufficient amount of data. 

> Computational efficiency.

I think number of local patches seen during the training is irrelevant in this discussion. My main concern that this model might be expensive to train on full ShapeNet or Objaverse. Rebuttal does not address it. That being said, this issue is way less concerning to me than weak evaluation so in the current state it does not affect my decision at all.
I really appreciate the effort that authors are putting in the rebuttal. However, I still don't find proposed arguments convincing. 

> We note that the data size is closely related to the task and model capacity. The recent generalizable shape auto-encoding methods use large encoder networks, where larger data is necessary to train those large models. 

First, I want to point out that 3DS2VS (recent method) does not have 'large encoder network'. As encoder, it uses cross-attention with 1 layer (!) and 1 head (!) ([see source code, LL 204-207](https://github.com/1zb/3DShape2VecSet/blob/master/models_ae.py)). The majority of capacity of this model actually concentrated in the decoder.  That being said, I agree that these methods usually perform better if trained on large amount of data. This brings us to my second point: strong performance on small data is not indicative about strong performance on large scale data. Large capacity model can perform worse on small scale data due to overfitting but strongly outperform low capacity model on large scale data. To me, strong results on 1000 shapes are not indicative of potentially strong performance on large scale data.  

Also, paper does not include direct comparison to capacity of the proposed models (in terms of mln parameters), so I am not even sure that Cofie has less parameters than baselines. 

> We perform experiments on ShapeNet-15 categories (5 categories for training, and 15 categories for testing); comparison to NKSR

Again, as I mentioned above, my concern is that strong performance of the proposed model on small scale training might not be indicative of performance on large scale data. In the end, one of the common application of the proposed method would be either large scale reconstruction or using this representation as input to generative models. Both of these applications require large-scale training and proposed results don't convince me as a person who trained reconstruction and generative models on full ShapeNet. 

> Additional evalution and metrics on ShapeNet.

First, I appreciate authors adding additional metrics. I strongly recommend including them in the revised version of the paper. Second, I want to point out that these metrics are important: as your results suggest, good CD performance does not mean god IoU performance and vice versa.  Finally, I want to point out that these results are still not looking convincing to me because training was done on 1000 shapes. What will happen if we will retrain proposed method and other baselines on 1000 other shapes? Will results be the same? I think that even if we talk about evaluation on 1000 shapes, the training should be done across 5 different subsamples of the data, results averaged and variance reported along with mean (similar to what people do with few-shot learning models). Since model takes 1 day to train, this evaluation is absolutely feasible given authors computational resources and can serve a strong argument in favor of the proposed method. 

> Additional metrics on Thingi10K;

Again, I appreciate the effort that went into these results. Unfortunately, they validate my concerns about weak evaluation of the paper. This additional evaluation shows that Cofie might not be as strong as was initially presented: both NGLOD and UODFs have better performance both in teroms of gIOU and CD after additional evaluation. My concern is that similar pattern will hold for ShapeNet results if trained with stronger baselines and larger data. 

SIDENOTE. After discussion, I think that maybe this representation can be useful for text-to-3D or image-to-3D distillation but this is not evaluated as well. This does not affect my rating -- just a note that I thought might be useful to authors.
Thank you for the response and thanks for championing our paper. We will keep discussing with other reviewers.
I thank the authors for their thorough response, as well as my fellow reviewers for their insightful comments. I appreciate the authors' effort to address my concerns and questions. I am, therefore, still leaning towards acceptance, though I believe that Reviewer `V5fn` raised a few relevant points (e.g., SOTA selection, dataset size). I am following the ongoing discussion with attention.
Thank you for your acknowledgment of the effectiveness of our rebuttal. We will incorporate the discussion in revision!
We will continue to reply to the comments.

---

- **3DS2VS terminology.** We realized that “latent code” can lead to misunderstanding from the clarification of the reviewer. We will revise it to “latent code set” as it titled. In the experiment, we use a set of 512 codes so no worries about it. Besides, it seems that the material you provided also doesn’t contain “latent code clouds” you mentioned, but we realize this may not be a meaningful battle. 
- **Quadratic MLP**. We agree that this is studied in the context of shallow MLP with ReLU and we kindly point the reviewer to DeepSDF, DeepLS, NGLOD where the context is adopted and we directly compare CoFie with them. We agree that using better activations or deeper MLP can improve the performance, but this is perpendicular to our contribution.
- **Patch-level property**. Please see our reply to your concern 2 on why using small-scale data for training. CoFie sees 36K local patches in one iteration. Our original reply was explaining why 1000 shapes can be sufficient and we never claim this is unique to CoFie. DeepLS has the same property.
- **Challenging shapes**. Thanks for the clarification. We will incorporate more visualization in these cases. We kindly refer the reviewer to Figure 8 in the appendix, where CoFie fails on super challenging thin structures, and we believe this is a common problem for any shape representation.
- **Computational efficiency**. Please see our reply to your concern 2 for more details.

 ---

Finally, we would like to **reiterate our contributions**. Generally, we intend to **inject geometry-aware inductive bias for shape representation**. Thus, we i) provide theoretical analysis on the quadratic shape patch properties, and propose to ii) disentangle the geometry and transformation information of local patches by using geometry-aware coordinate fields and initialization method, and iii) leverage quadratic operations to improve the representation capability on quadratic local patches. These contributions make our shape representation easier to learn and compact (achieving same performance using a much smaller code size), and the contributions are ablated. **We would appreciate it if these contexts of larger goals/contributions could be considered.**

Again, we would thank the reviewer for the detailed feedback. We will discuss the mentioned works in the revision and incoporate the discussions. We sincerely hope this reply can be considered.
We thank the reviewer for the detailed reply and we appreciate the efforts! We are happy that we have resolved the factual errors in the original rebuttal. We will address the new comments as follows.

---

**[C1, Baselines]** We understand your concern regarding the baseline. However, as we noted in our rebuttal, **we didn’t miss any directly comparable baselines** (generalizable shape auto-decoding) as far as we know. **The suggested baselines work on different tasks** of generalizable shape auto-encoding or shape-specific auto-decoding, where the train-test paradigm and inputs are different from our setting. **We have already included the representative works for the two tasks**, NGLOD (2021) and 3DS2VS (2023), **as reference in our paper to understand the performance of CoFie**.

To further address your concerns, we would like to point to the **two new baselines we added during the rebuttal period** (in the reply to other reviewers). We include the results of **NKSR** (2023, generalizable shape auto-encoding with additional point normal input) and **UODFs** (2024, shape-specific auto-decoding) in Tables R1 and Table R2. We report the results of NKSR on the 10 novel ShapeNet categories, and report the results of UODFs on the Thingi shapes (due to its slow inference speed of 300 minutes per shape).

We note that directly comparing the numbers across different settings is not meaningful. Instead, we should compare CoFie with methods in the same setting (DeepSDF, DeepLS). **We agree that DeepSDF and DeeepLS are relatively old works, but this is perpendicular to our contributions in this work, as every contribution is ablated**. 

---

**[C2, Training data size]** In general, we agree that recent works are using larger data for training. At the same time, we would like to provide more explanation why we didn’t use super large-scale data for training. We note that **the data size is closely related to the task and model capacity**. *The recent generalizable shape auto-encoding methods use large encoder networks, where larger data is necessary to train those large models. In contrast, shape auto-decoding methods typically use shallow MLPs (DeepSDF, DeepLS, NGLOD, etc), where the benefit of using larger data is smaller*. Again, we note that **we strictly follow the training data scale of prior directly comparable works**, i.e., DeepSDF and DeepLS.


Moreover, the reviewer questioned our training efficiency, e.g. 1000 shapes takes 1 day for training so using larger data takes longer training time linearly. We would like to clarify that we set a total iteration number for training, e.g. 150K iterations (in each iteration we train with 36K local patches), which is sufficient. When using larger data, we won’t linearly scale the training iterations.

Besides, the authors mentioned recent shape auto-encoding methods using **ShapeNet-13** for training/testing and requested our method to perform experiments on the same amount of data/categories. In response, we would like to remind you that **we perform experiments on ShapeNet-15 categories** (5 categories for training, and 15 categories for testing). The performance on the 10 novel testing categories strongly suggests that CoFie can generalize to novel categories.

Finally, we are happy to provide experimental results using more data (full set of ShapeNet-5 categories) for training. Additionally, we note that **this experiment is not explicitly required in the first review comment**. We will try our best to finish the experiment within the discussion period.

---

**[C3, More Evaluation Metric]** We thank the reviewer for the suggestion! We report the gIoU metric following NGLOD as follows.

**Table R1. Performance on ShapeNet 10 novel categories.** GAD denotes Generalizable Shape Auto-decoding, and GAE denotes Generalizable Shape Auto-encoding.

| Setting | GAD        | GAD        | GAD        | GAE               | GAE               |
|---------|------------|------------|------------|-------------------|-------------------|
| Method  | DeepSDF    | DeepLS     | CoFie      | 3DS2VS (full set) | NKSR (full set)   |
| CD (1e-4)| 10.4      | 7.27       | 3.18       | 9.30              | 4.24              |
| gIoU     | 83.1       | 96.2       | 98.3       | 94.8              | 96.9              |

---

**Table R2. Performance on Thingi shapes of arbitrary categories.** SSAD denotes shape-specific auto-decoding.

| Setting | GAD        | GAD        | GAD        | SSAD              | SSAD              |
|---------|------------|------------|------------|-------------------|-------------------|
| Method  | DeepSDF    | DeepLS     | CoFie      | NGLOD             | UODFs             |
| CD (1e-4)| 9.79      | 3.68       | 1.87       | 1.04              | 0.932             |
| gIoU     | 87.1       | 97.4       | 99.0       | 99.3              | 99.4              |

Additionally, we noticed an error in our Table 3 in the paper, where results for DeepSDF should be for DeepLS.
>  We searched in the 3DS2VS paper, and there is not a single matching regarding the term “latent code clouds” you mentioned.
> Shape latent code is the official terminology in 3DS2VS. 

I refer authors to the 3DShape2VecSet paper introduction and contribution 1 in particular: "We propose a new representation for 3D shapes. Any shape can be represented by a fixed-length array of latents".  I also refer authors to section 5.1 to 3DS2VS paper that explicitly states: "In our final model, the number of latents 𝑀 is set as 512, and the number of channels 𝐶 is 512". Finally, I refer authors to the [source code of 3DS2VS (LL182-283)](https://github.com/1zb/3DShape2VecSet/blob/master/models_ae.py). 

>  “Latent code” doesn’t mean there is only one code

I don't believe authors are making this argument in good faith. DeepSDF paper explicitly states that shape is represented by 1xD vector. 3DS2VS paper explicitly states that shape is represented by 512xD tensor (see above). In scientific research, precision of language matters. For example, nobody call 'matrix' (2D tensor) a 'vector' (1D or 0D tensor).

> The use of quadratic MLP is based on our analysis with a strong theoretical foundation (Sec 3)

The argument in Sec. 3 is done in comparison to 'shallow MLP using ReLU'. This assumption does not hold in majority of practical cases: majority of methods either use non-shallow MLPs, or non-locally linear activations like (ELU, GEGLU, GELU, SiLU, etc), or both.  

> We experiment with using Fourier embeddings and show the results below <...>

Thank you! This is solid empirical evidence in favor of proposed method and I recommend including it in updated version of the paper. 

>  Our method works on a patch level, and the 1000 shapes introduce 3 million local patches, which are sufficient for training.

I respectfully disagree. The fact that models sees 3 mln local patches of 1000 ShapeNet shapes during training does not mean that it will be able to generalize to 50000 unseen ShapeNet shapes. I also want to note that 3DILG also utilizes local surface point patches (512 per shape) that are randomly sampled for each iteration. It means that for each iteration of 1000 shapes 3DILG sees ~0.5 mln of unique local patches, so this property is not unique to the proposed method.  

> Ability to deal with topologically challenging shapes

I respectfully disagree. Maybe the issue here is that we that we consider different types of shapes topologically challenging. I refer authors to Fig. 8 of 3DS2VS paper for examples of topologically challenging shapes that I find topologically challenging (esp. rows 2,3,5,8,9). My concern is that proposed method won't be able to handle such shapes (and 3DS2VS is capable of it). 

>Our method is computationally efficient <...>. 

I respectfully disagree. While proposed method might be more computationally efficient than DeepLS (2020 paper), it still requires 4 24GB GPUs and 24 hours to train on 1000 shapes. To me, it means that it will take at least a month to train this model on full ShapeNet. But given more pressing issues with evaluation and baselines, this does not hold high weight in my evaluation. 

> You mentioned that DeepSDF overfits single shapes, but in fact, it trains the MLP decoder on a set of training shapes, and optimizes the latent code for each shape during testing.

My confusion is understandable since proposed method compares both to NGLOD that was trained in overfitting setting and 3DS2VS that is trained in auto-encoding setting. That being said, I apologize for the confusion but want to point out that this confusion does not invalidate my concerns. 

> The clarity of our writing is acknowledged by other reviewers <...>

The goal of review process is to gather unbiased feedback from fellow researchers, so it is only natural that opinions differ.
I appreciate very detailed feedback by the authors: it clarified a lot of details in the original submission. Below I will address main rebuttal points (and I will address other rebuttal points in separate comments). 

After the rebuttal, I have no doubts that Cofie proposes a novel method that can be viewed as a sparse irregular latent cloud/grid/array of 1/32 sized voxel cells that intersect with shape surface. These cells are decoded via novel quadratic MLP decoder (and this choice is ablated in the rebuttal).  This makes it different from 3DILG, Dynamic Code Clouds, 3DS2VS, AutoSDF and SDFusion. 

Unfortunately, my three major concerns about evaluation are not addressed in the rebuttal. 

**Concern 1. Choice of baselines for auto-decoding setting (Tab. 1) is very weak.**

DeepSDF is the 2019 paper. DeepLS is a 2020 paper. Since then, a lot of strong shape representations were proposed: Local Implicit Grids (2020), AutoSDF (2022), SDFusion (2023), Dynamic Code Clouds (2023), 3DILG (2022). I don’t expect authors to compare to all of them but in addition to 3DS2VS proposed method needs comparison with 2-3 recent methods that utilize latent grids/clouds/arrays. 

**Concern 2. Proposed method is evaluated via training on 1000 ShapeNet shapes (Table 1).**

Authors claim that “using a thousand instances for training is typical in previous works [24,4]”. Again, DeepSDF [24] is the 2019 paper; DeepLS [4] is the 2020 paper. For comparison, 3DS2VS trains on 48597 ShapeNet shapes (auto-encoding task) and evaluates on 2592 shapes. In fact, recent works use at least ShapeNet-13 (see Dynamic Code Clouds papers). In my opinion, the proposed method must either be trained and evaluated at least on ShapeNet-13 (as Dynamic Code Clouds does) or results for 1000 shapes must be averaged across at least 5 different runs on different 1000-shape subsamples of ShapeNet. 

**Concern 3. The paper only uses Chamfer Distance as an evaluation metric.** 

Majority of baselines use at least 2 evaluation metrics. DeepSDF [4] uses CD and EMD; 3DS2VS uses CD, IOU and F-score. Only DeepLS uses just a CD. Given the strong claims in the paper and limited training/test size in the submission, I recommend having at least one additional evaluation metric.  

*If the paper with current evaluation pipeline is accepted, my concern that it might create bad precedent for follow-up work that is going to use the same evaluation pipeline. Thus, my rating remains unchanged.*

======================================

Below are additional main rebuttal points I want do address: 

> By searching “PointNet” in the DeepSDF paper, we don’t find any matches in the method section

Authors are right and I acknowledge my mistake and apologize for the confusion. But I want to point out that this mistake does not invalidate my concerns outlined above and the fact that these concerns were not sufficiently addressed in the rebuttal. 

> The effectiveness of our evaluation is acknowledged by other reviewers

The goal of the review process is to gather unbiased and diverse feedback from fellow researchers, so it is only natural that opinions differ.
I have read the authors' rebuttal and appreciate their efforts during the rebuttal process. Therefore, I maintain my initial rating.
We are glad to see that the reviewer's concerns were resolved, and we will update our paper based on our discussion with the reviewer. Thank you!
Thank you for your replies, they have addressed my concerns (and I hope they are clarified in the revised paper). I have also read the concerns of other reviewers and your replies, and am happy with them. I maintain my previous position of weak accept.
**[W5, Evaluation]**  **You mentioned that DeepSDF overfits single shapes, but in fact, it trains the MLP decoder on a set of training shapes, and optimizes the latent code for each shape during testing**. The MLP decoder is shared for all shapes, thus, it is not overfitting. Please see details in the DeepSDF paper and the training split of the DeepSDF official GitHub repo. We strictly follow this setting. We agree that NGLOD overfits each single shape (shape-specific) for both the MLP decoder and shape representation. As we discussed in L245, shape-specific methods generally have better accuracy as it is specialized in a single shape. We use it as an “oracle” comparison, and the results in Table 3 show the strong performance of our method.

---

**[W6, Quadratic MLP]** The use of quadratic MLP is based on our analysis with a strong theoretical foundation (Sec 3).

---

**[W7, Paper details]** The clarity of our writing is acknowledged by other reviewers, including “proofs are sufficient and articulation is clear” (4hsy) and “well illustrated and easy to follow” (Kq6M). As we mentioned in L201, the inputs are SDF samples, not point cloud or mesh. The density and method of SDF sampling strictly follow DeepSDF, which is also the same in our experiment of DeepLS and NGLOD. For 3DS2VS, we use 10K points as input for a fair comparison, as DeepSDF sampling has the number of SDF samples with this quantity.

---

**[W8, Statements]** Shape latent code is the official terminology in 3DS2VS. See the second line of the caption of Figure 2 in 3DS2VS. “Latent code” doesn’t mean there is only one code. We searched in the 3DS2VS paper, and there is not a single matching regarding the term “latent code clouds” you mentioned.

---

**[W9, Computation]** Our method is computationally efficient. The 1000 training shapes contain 3 million local patches, and we train on the patch level. When using the latent code size of 125, our method achieves comparable performance with DeepLS using only 61% training iterations. This is strong evidence that our method is computationally efficient.

---

**[W10, Reference]** We are happy to cite and discuss the mentioned papers in the revised version of our paper.

---

**[Q1, Input]** As we mentioned above, we are performing shape auto-decoding (updating latent code via back propagation and optimization during testing), where the standard input is SDF samples. The input of DeepSDF, DeepLS, and CoFie are strictly the same. We don’t have an encoder as we perform shape auto-decoding. Regarding generalization, there is no correlation between generalization and whether it uses back propagation. See DeepLS, which is a generalizable method using back propagation, for more details.

---

**[Q2, Baselines]** As we mentioned above, DeepSDF, DeepLS, NGLOD, and our method use strictly the same inputs. We follow the 3DS2VS official code for training and testing.

---

**[Q3, Training data size]** Again, we are doing shape auto-decoding, which is a different task from auto-encoding. We agree that shape auto-encoding requires a large amount of training data as it learns priors to map the shape/point cloud into latent code. However, this is not true for shape auto-decoding as these methods optimize the latent code during inference. For example, DeepLS also uses 1000 shapes for training (as discussed in the experiment setup paragraph in the DeepLS paper) and we follow their setting strictly. Our method works on a patch level, and the 1000 shapes introduce 3 million local patches, which are sufficient for training.

---

**[Q4, Using Fourier embedding]** Good question! We experiment with using Fourier embeddings and show the results below. To make the comparison fair, we use a code size of 64 and Fourier embedding with a dimension of 64, which makes the input dimension the same as using point coordinates (3 dimensions) with a 125-dimensional latent code. We provide the results as follows, and they show the effectiveness of our method. Alternatively, we note that our idea of coordinate field is orthogonal to using positional encoding, as we can also apply positional encoding on the transformed coordinate system.

|     | DeepLS| DeepLS + Fourier Embedding | Ours |
|:---:|:-----------:|:-------:|:----:|
| CD (1e-4)  |   7.27 | 6.22 | 3.18 |

---

**[L1, Representing complex shapes]** We have already shown the capability of CoFie in Figure 3, Figure 6, and Figure 7, which include topologically challenging shapes like thin structures and unsmooth local surfaces, as well as edges and corners.

---

**[L2, Computation]** Please see W2 and W9. Alternatively, we report that NGLOD takes 105 minutes for a single shape, which is much slower than ours.