NeuForm: Adaptive Overfitting for Neural Shape Editing
Authors: Connor Zhizhen Lin
Niloy Mitra
Gordon Wetzstein
Leonidas Guibas
Paul Guerrero
Venue: NeurIPS 2022 Accept (2022)
Keywords: shape editing
localized overfitting
neural shape representations

Abstract: Neural representations are popular for representing shapes as they can be used for data cleanup, model completion, shape editing, and shape synthesis. Current neural representations can be categorized as either overfitting to a single object instance, or representing a collection of objects. However, neither allows accurate editing of neural scene representations: on the one hand, methods that overfit objects achieve highly accurate reconstructions but do not support editing, as they do not generalize to unseen object configurations; on the other hand, methods that represent a family of objects with variations do generalize but produce approximate reconstructions. We propose NeuForm to combine the advantages of both overfitted and generalizable representations by adaptively overfitting a generalizable representation to regions where reliable data is available, while using the generalizable representation everywhere else. We achieve this with a carefully designed architecture and an approach that blends the network weights of the two representations. We demonstrate edits that successfully reconfigure parts of human-made shapes, such as chairs, tables, and lamps, while preserving the accuracy of an overfitted shape representation. We compare with two state-of-the-art competitors and demonstrate clear improvements in terms of plausibility and fidelity of the resultant edits.

NEUFORM: Adaptive Overfitting for Neural Shape Editing
Connor Z. Lin Stanford University
Niloy J. Mitra Adobe UCL
Gordon Wetzstein
Stanford University
Leonidas Guibas
Paul Guerrero
Stanford University
Adobe
Abstract
Neural representations are popular for representing shapes, as they can be learned form sensor data and used for data cleanup, model completion, shape editing, and shape synthesis. Current neural representations can be categorized as either overfitting to single object instance, representing a collection of objects. However, neither allows accurate editing of neural scene representations: on the one hand, methods that overfit objects achieve highly accurate reconstructions, but do not generalize to unseen object configurations and thus cannot support editing; on the other hand, methods that represent family of objects with variations do generalize but produce only approximate reconstructions. We propose NEUFORM combine the advantages both overfitted and generalizable representations by adaptively using the one most appropriate for each shape region: the overfitted representation where reliable data available, and the generalizable representation everywhere else. We achieve this with carefully designed architecture and an approach that blends the network weights of the two representations, avoiding seams and other artifacts. We demonstrate edits that successfully reconfigure parts human-designed shapes, such as chairs, tables, and lamps, while preserving semantic integrity and the accuracy of an overfitted shape representation. We compare with two state-of-the-ar competitors and demonstrate clear improvements in terms of plausibility and fidelity of the resultant edits.
Introduction
Neural formulations have emerged as efficient and scalable representation of complex spatial signals, such as radiance fields, 3D occupancy fields, or signed distance functions. These repre- sentations popular as they allow uniform formulation that can support range of applications including denoising, data completion, and editing. In the context of shapes, two main types of neural representations have emerged. Starting from input description (e.g., point clouds, meshes, or distance/occupancy fields), current representations either overfit to a single shape or learn a model that generalizes a collection of varying shapes. However, neither of the representations alone allows effective shape editing.
Overfitted models [9, 35, 32, 22, 26, 20, 26] reproduce single shape with high fidelity. While this allows for operations like efficient rendering, surface-based optimization, and data compression, such representation does not support shape editing or synthesis, since it does not generalize to novel shape configurations.
In contrast, generalizable representations [28, 15, 21, 6] are trained on large collection of shapes and learn shape priors allowing the representation adapt to previously unseen shape configurations. Thus, they can be used for shape editing and novel shape synthesis [16, 15, 34, 24, 19, 23]. However, this comes the cost of lower-fidelity representation, as the network needs to represent a full dataset and its variations, instead of single shape. Specifically, these models typically require 'projecting'
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

shape into the learned latent space before editing it, where the idiosyncrasies of the starting model, in the form of local geometrie details, are often lost (see Figure 1).
We propose a novel blended architecture, called NEUFORM, combine the advantages of the two representations described above. Specifi- cally, we retain distinctive properties of the in- put shape by relying on an overfitted model and switch to the generalizable model to complete parts where information is missing (e.g., near new joint locations or regions with holes). The main challenge is to train an adaptive mixing network that blends the information between the overfitted and generalizable models, without in- troducing artifacts such as undesirable seams or gaps. The NEUFORM architecture allows this seamless sharing of information between the in- dividual networks. Our main technical insights are that (i) it is possible to smoothly interpolate between two neural shape representations by blending between the weights of two networks sharing an architecture and a training history, and (ii) it is possible to do this blending between a generalizable network that works on a global view of the shape and an overfitted network that only has access to part of the shape by carefully pruning the information flow during overfitting.
original
edit
original
generalizable
overfitted NEUFORM
NEUFORM Part Mixing
Purt Editing
Figure Adaptive overfitting. NEUFORM en- ables detail preserving shape edits that generalize to new part configurations by combining advan- tages of a generalizable representation (c.g.. gen- eration of plausible joint geometry) and an over- fitted representation (e.g.. detail preservation on the backrest), and also allows mixing parts from different shapes.
We evaluate NEUFORM on multiple applications: (i) reconstruction (i.e., projecting given input to an adaptive overfitted latent space); (ii) part based shape editing; and (iii) shape mixing (i.e., converting an arrangement of parts taken from different models into a coherent shape model). We compare with two state-of-the-art approaches [16, 39] and demonstrate advantages, both quantitatively and qualitatively. Figure 1 shows example of shape edit where we can see clear advantage for NEUFORM over both purely generalizable and purely overfitted representations.
2 Related Work
Single-Scene Neural Shape Representations Overfitting networks represent one specific shape via a single network by optimizing network weights. Such overfitted networks are useful for several applications including compression [9, 35], adaptive network parameter allocation [20], multiview reconstruction [32, 22), shape optimization [27], or multi-resolution shape representation [26, 38, 35]. While such networks, by construction, accurately capture the original shapes, faithfully encoding their finer details, they can neither used for editing shapes nor for creating new shapes by combining parts from multiple (source) shapes.
Multi-Scene Neural Shape Representations Neural networks have been used to approximate implicit models, as example of complex spatial functions, to represent shapes volumetric signed distance fields [28, 6] or occupancy values [21]. Such network learning has been further regularized by geometric constraints like the Eikonal equation [13, 3, 2] using intermediate meta- network for faster convergence [18]. Other approaches model shapes using their 2D parameterizations [14, 37]. Improved versions of such methods optimize for low-distortion atlases [4], learn task-specific geometry of 2D domain [10, 31], or force the surface to agree with an implicit function [29]. Most of these methods encode shape collections in lower-dimensional latent space, proxy for the underlying shape space, and support shape editing and generative modeling. For example, sampling
from and optimizing in the (restricted) latent spaces can produce voxel grids [19, 12, 5, 8]. point
clouds [1, 33], meshes [7], or collections of deformable primitives [11]. Others [15, 24, 34] use two level representations with a primitive-based coarse structure capturing the part arrangement, and a detailing network that adds high-resolution part level geometric details. While these methods do generalize across shapes, and can be used for editing [15, 34, 16, 39], the source models often lose their finer details during the projection to the underlying latent space and subsequent editing process. In Section 4, we compare against two of the most relevant methods: COALESCE [39], which focuses on part-based modeling and synthesizing part connections focuses on inter-part relations towards shape editing and mixing. Our method, NEUFORM, M, generates higher quality joints than the former while preserving more (original) surface detail than the latter.
and SPAGHETTI [16]. which

---

Method
Given a manifold and watertight 3D shape S with known part annotations, our goal is to edit the parts of without introducing objectionable artifacts or losing geometric detail. The shape can be given as a mesh, signed distance function, occupancy func- tion, and the part annotations are specified set of oriented cuboid bounding boxes (CI...., C. where n is the number of parts of S. During editing, parts may be
x
NRUFORM generalizable rep.
f
P
Pr
E
a(())
P-
P
θ
W
a
P
P1
0
x(x))
arranged via scaling and translation, and/or mixed across multiple shapes. To avoid ar- tifacts in the edited shape, some regions of the shape geometry, such the joints be-
P
m
P
overfitte rep.
W (x)
θ
P1
tween individual shape parts, need to be ad- justed to adapt to the new part configuration. To enable part-based editing without losing geometric detail, we construct representations of shape S: generalizable shape representation and an overfitted shape representation.
x((z)
P
neural
Figure 2: Architecture overview. NEUFORM blends between a generalizable neural shape representation (green) and an overfitted neural shape representation (red) by interpolating their network weights and some feature layers. This combines the benefits of detail preservation from the overfitted representation and ed- itability from the generalizable representation.
The generalizable shape representation is a part-aware neural shape representation trained to represent large shape space. This parameterization can generalize to previ- ously unseen part configurations, including the edited configuration of shape S, but can only provide a low-fidelity reconstruction of S.
The overfitted shape representation is neural shape representation overfitted to single shape S. It represents the input shape geometry great detail, but does not generalize to unseen part configurations, such edited configurations of S.
combine these representations by blending between them, as explained Section 3. regions where reliable data is available for overfitting, such as regions unaffected by edits, we use the overfitted shape representation. In regions where geometry should be adjusted, e.g. joint regions between parts, leverage the generalizable representation. Both representations share the same architecture and we blend between them by directly interpolating their network parameters, which requires careful design of both the architecture and overfitting setup. We call this approach adaptive overfitting
Generalizable Shape Representation
Shape parameters. In the generalizable representation, shape S is represented as set of part parameters == (PI...., P. The parameters of part P. := (C,, gi) consist of a cuboid bounding box C := (w: cs, o,), where v; e,, O; are the centroid position, size, and orientation of the cuboid, respectively, and latent vector y defining the part's geometry in the local coordinate frame of the cuboid. We obtain y from S by encoding part P. with a PointNet [30] encoder as 9i == (ri.ri...... r ). although other options obtain 9i such as an auto-decoder setup with inference time optimization are also possible.
surface and volume points rī, ri...., r sampled from
Generalizable occupancy function. Given the part parameters P, a neural network fo models the occupancy field as of shape S at any query location as,
(2( R ap(x)= = fe(z)P).
(I)
The architecture of f is illustrated in Figure/ 2. This is similar to the formulation proposed in SPAGHETTI [16), but with changes that are necessary for adaptive overfitring. The network composed of three parts: A part mixing network f to exchange information between per-part latent

---

vectors; part query network fø to query each part at the query point 2; and a global occupancy network fa. aggregating the results of the per-part queries and output the occupancy at æ.
(i) Part mixing network. The mixing network f first converts parameters P; into per-part latent vectors, and then exchanges information between parts using self-attention layer:
p = fm (P.IP).
(2)
(ii) Part query network. The part query network f" queries each part p. at the local query point locations using cross-attention from each local query point to all per-part latent vectors pf:
(x) qf xx)x)= = fö_ (Tc. (x) pf + bo...... P + bi......P. P. + bo), fo
(3)
where Tc denotes the transformation to the local coordinate frame of C.. Like f is run once per part. For a given part i, we augment the input latent vectors p." by adding a learned indicator feature that equals by for the current part and by for all other parts, giving the network knowledge of which parts it is currently processing. The resulting latent vector of encodes the local geometry region of part that is relevant the query point æ.
(iii) Global occupancy network. Finally, we aggregate the per-part latent vectors of into global latent vector using weighted sum and the global occupancy network fo computes the occupancy the query location I:
op(x) = 18. (E w.
(x) of (x)).
where the weights w:, = x< max dz, C:))) are based on the signed distance de from query combines finite support with a smooth falloff: k(a,) = (1 - d ورا² where p is the radius of the kernel and a; = min (max(d", 0), p) is the bounded distance to cuboid C. Essentially, P defines the extent of joint regions and k provides a smooth fall-off to a, approaches p. We set p 0.35 in all our experiments.
point to cuboid C₁. We choose the triweight kernel for
p
Training setup. We jointly train the part encoder hy and the occupancy network fo on large dataset of shapes using a binary cross-entropy loss between the predicted occupancy px(r) and the ground truth occupancy (I)). More details are given in the supplementary material.
Shape editing. Due to training on large dataset, the generalizable shape representation captures a large space of part configurations. Shape edits can be performed by modifying the parameters of one or multiple cuboids, such as the position Uy or scale e,, to obtain the modified part set PE and infer a modified occupancy p (x) ;= fo(x
Overfitted Shape Representation
Overlitted occupancy function. The goal of the overfitted representation is to accurately capture the geometric detail of individual parts of a single shape. We use an overfitted occupancy function with the same architecture as in the generalizable representation to facilitate blending between the two, as described in the next section. Naively overfitting this occupancy function to a shape S would result in artifacts when reconstructing an edited shape Sg, since the overfitted occupancy function does not generalize to unseen part configurations. Instead, we carefully sever the information flow between parts during overfitting such that querying the overfitted occupancy funccion does not use information about the full edited part configuration. We employ two-part strategy: (i) We freeze the part latent vectors pf before overfitting and only update the query network f and the occupancy network f
@p(x)= (1) 18. Σ
(c) P. (z))
(5)
with /.
P() (z) f (T (a) .. bb.... ......... + b . P. bo).
where + is the occupancy predicted by the overfitted network, 6,, 8. are the overfitted parameters of the query and occupancy networks, and T" denotes part latent vectors that were frozen to the part

---

set P. (ii) We change the weights ui to only select the single part latent vector q" that is closest to the query point x: v (x) 1 (1) (arg min; d2 (2, C.),, where is the indicator function. These two changes effectively make the occupancy (x) at each query point dependent on only the single closest part, preventing the overfitted occupancy function from being exposed to unseen part configuration.
Training setup. We start with trained generalizable network fo and part set P we would like to overfit to. We freeze the part latent vectors v."" - - (P) to the values computed by the generalizable network, and then proceed to overfit both fØ, and fö. to the partset P, giving us the overfitted network f0.z. During overfitting, we gradually blend between the original weights wy at the first epoch to the updated weights úi at the last epoch.
Shape editing, Similar to the generalizable representation, edits of the overfitted representation can be performed by modifying cuboid parameters to obtain modified part set Pg. and modified occupancy P.. (z) = fz)((=(P)) As result of our strategy to decouple parts from each other, a transformation T; of a cuboid C; is directly applied to the occupancy of the corresponding part: Pr (2)) = op (p;"((--))) ) for all æ that are closer to cuboid i than to any other cuboid. This accurately preserves geometric detail after edit, but results in discontinuities at the boundaries between edited parts, as shown in Figure 1.
Adaptive Overfitting Our goal is to use the overfitted representation in areas where the overfitted occupancy reliable, and the generalizable representation everywhere else. For shape edits that transform cuboid parameters, the overfitted occupancy in any local region undergoes the same transformation as the nearest cuboid. For human-made shapes such as chairs and tables, this behaviour is desirable in regions that are either close to only cuboid, close to only unedited cuboids. In other regions (near joints between two or more cuboids, or where at least one cuboid has been edited), the occupancy may need to undergo more complex transformations to reflect the new part configurate guniguration.
Given a set of parts Po and an edited version of the parts PE, we formalize the intuition described above as scalar blending field (xx) defining blending factor in [0, 1] between the generalizable and the overfitted representation at each query point min
((x) s(
C (CPo UPPE/C". (I.CC)
where CPo and ch are the subsets of cuboids in the original and edited shape, respectively, that have been changed in PE. CPE is the cuboid in Pg closest to æ. The kernel k is the same triweight kernel defined in Section 3 for part aggregation in the global occupancy network.
Given a blending factor A(z), we finally fuse the two representations by blending between the parameters, weights, and features of the networks:
Tp(x)
16 Σ
of
(()
P.P (x)),
with (x)
P. bo ....... P...P. + .I....., PP.. + bb):
where 0., 0m. wP (x), and pp.P are linearly interpolated between the overfitted and generalizable representation using the blending factor A(r))
0. (1 X(=)) 0, + X(x) 0.. of (z) (1-X(z)) &P (2) (x)= A(z) w (x), (x(x)
(8)
(9)
p P. (1 (z=) h.. + X(x) p. (z).
(10)
When editing shape, typically overfit to the original configuration of the parts, in that case, set P = Po and P = Pg.
4 Results
We evaluate FEUFORM three tasks: shape reconstruction, shape editing, and shape part mixing.
5

---

ground truth
COMEESCE
SEAGHETTI
NEUFORM gencralizable
NEUFORM overfitted
NEUFORM
7000
Figure 3: Shape reconstruction. Comparing reconstructions of PartNet [25] chairs. We show reconstructions of four shapes. COALESCE and the overfitted representation preserve geometric detail, but have more artifacts near joints. SPAGHETTI and the generalizable representation perform better near joints but lose geometric detail. NEUFORM combines the best of both worlds.
Dataset. We use the PartNet [25] dataset for our experiments. PartNet is a dataset of shapes in 24 common categories, including furniture and typical household items. Each shape is annotated with hierarchical part segmentation. We experiment on the chair, 1amp, and table categories and select hierarchy levels that result an average of roughly 8, 4, and 8 parts for chairs, lamps, and tables, respectively. Cuboids are computed as oriented bounding boxes of the segmented parts using Trimesh [36]. We train the generalizable model on each shape category separately and choose a training/test split of 6000/1800, 2100/400, and 3500/500 for chairs, lamps, and tables, respectively. All shapes are centered and the largest bounding box side scaled to 2.
Training details. We train the generalizable model for 1000 epochs using the Adam [17] optimizer with learning rate of le and exponential learning rate decay of 0.994 per epoch. In each epoch, we train on 4096 query points per shape with a batchsize of shape. We sample 12.5% of the points uniformly in the [-1, 1] cube and 87.5% of the points around the surface with Guassian offset (0, 0.05)). The overfitted model is trained for 100 epochs on a single shape using the same training setup. Training the generalizable model takes roughly 33 hours on TitanXp GPU and training the overfitted model takes roughly 25 minutes on a single V100 GPU.
Baselines and ablations. We compare our results to SPAGHETTI [16] as the state-of-the-arr gen- eralizable representation sharing a similar architecture to our generalizable representation, and COALESCE [39], state-of-the-art method generating the joint geometry between parts given (poten- tially re-arranged) part meshes. Additionally, we compare with two ablations of our method: using only the generalizable representation and using only the overfitted representation.
Metrics. As quantitative metrics, we follow prior work in using the Chamfer Distance (CD) and Earth Mover's Distance (EMD) between points sampled on generated shape surface and points sampled on ground truth shape surfaces. For CD, we sample 30k and 10k points uniformly on the shape surfaces away from and near joint regions, respectively. near joint regions for EMD. As volumetric measure, we evaluate the signed distance field (SDF) 25k points away from joint regions and 5k points near joint regions per shape, with the same distribution as the query points, and report the absolute difference between the values of the generated
sample 1024 points away from and

---

ground truth
cuboids offied
generalizable
NeuiFoo
NESFORR overfined
NeuForm
Figure 4: Shape editing. Comparing edits on PartNet chairs when using only the generalizable or only the overfitted representations. We show edits shapes with different coarse structure and fine scale details. The generalizable representation has plausible joint areas, but lacks geometric detail; the overfitted representation preserves detail, but has artifacts near joints (see zoom-ins). NEUFORM combines the two representations to both preserve geometric detail and generate plausible joints.
and ground truth shapes. Since our tasks focus on the joints between shape parts, we separately report these metrics joint regions (A(x) < 0.5; see Eq. 6). non-joint regions, and unweighted average of the two.
(i) Shape Reconstruction. While our main application shape editing, we cannot perform quantitative evaluation on the shape editing task directly, since we have no ground truth for edited joints. Instead, we use reconstruction experiment as a pseudo-editing task. We evaluate the reconstruction performance of NEUFORM compared to baselines and ablations on input that reflects the information available during shape editing, where we have some knowledge of non-joint regions, but no knowledge of joint regions. In this experiment, we demonstrate that the generalizable model is necessary to accurately reconstructtgenneaat joint regions, and the overfitted model necessary to accurately reconstruct detail in non-joint regions, therefore the comparison includes both overfitted and generalizing models. Our overfitted model is trained without ground truth for any the joint
compare our results to all baselines and ablations. The Chamfer Distance is multiplied by 10 SPAGHETTI and our generalizable representation perform well in joint regions, while COALESCE and the overfitted representation perform better in non-joint regions. The adaptive overfitting performed by NEUFORM achieves good performance in both regions, resulting overall a significant improvement over both SPAGHETTI and COALESCE. As would expect, the overfitted representation performs perticularly well on the reconstruction task, but its performance on joint regions drops significantly in shape editing tasks, as demonstrate qualitatively in the following sections.
Table 1: Comparing shape reconstruction performance.
Joint regions
Non-joint regions
All regions
CDL
EMDJ
SDF
CDL
EMD, SDFL
CDJ EMDJ SDF|
SPAGHETTI [16]
0.337
65.54
1.343
1.381
176.27 3.758
0.859 120.96 2.570
COALESCE [39]
0.738
97.51
2.440
0.154
130.20 2.918
0.446 113.86 2.679
NEUFORM generalizable
0.390
84.27
2.109
0.523
117.81
5,208
0.457
101.04 3.659
NEUFORM overfitted
0.318
78.54
2.198
0.157
80.45
2.644
0.238
79.50 2.471
NEUFORM
0.253
78.05
1.814
0.334
88.53
2.538
0.293
83.29
2.176

---

ground truth
ground untt
ceiginal cuboids edited cubcids
COALESCCE
Nou Nou FURM
NauFora
Figure 5: Comparing edits on PartNet chairs to COALESCE [39] and SPAGHETTI [16]. We show two different sets of edits because COALESCE does support edits of more fine-grained parts like bars, while SPAGHETTI does not currently support part scaling in their released code. COALESCE struggles with edit. Here we show screenshots from SPAGHETTI'S editing UI (hence the different color). Blending between the generalizable and overfitted representations using NEUFORM gives edit results, with cleaner joints and detailed part geometry.
extended joint areas and SPAGHETTI's result is significantly noisier after
plausible
areas and we perform the comparison on 64 shapes selected randomly from the test set. COALESCE does support fine-grained parts, thus, for fair comparison, restrict our joint areas those defined by COALESCE in this experiment.
Table I shows quantitative results of this comparison and Figure 3 shows qualitative examples for all methods. SPAGHETTI performs well in joint regions, but since is a generalizable model, lags behind the overfitted model and COALESCE in non-joint regions, giving lower performance overall. COALESCE has the lowest performance in joint regions, as it struggles with larger or more extended joint areas, and has reasonable performance in non-joint areas. While COALESCE uses the ground truth geometry in non-joint areas, some of the joint geometry tends to incorrectly extend into the non-joint areas, lowering the performance. As expected, performs well in joint regions, and misses detail non-joint regions. In this reconstruction task, the overfitted representation performs significantly better in joint regions than in the edit tasks describe in the next sections, since the part configuration of the reconstructed shape is the same overfitted to. In the reconstructions, errors at the joints the missing ground truth in joint regions. NEUFORM combines the advantages of the overfitted- and the generalizable representations. uses the lower-detail, but plausible geometry from the generalizable model in the joint regions, and naturally transitions to the detailed geometry of the
generalizable representation
the part configuration
due to
original
part donur
SEAGHETH mixed
NIuFORM mixed
original
part donor
Sescarm mixed
NEUFORM mixed
Figure 6: Shape mixing. Mixing parts of different PartNet chairs. We replace the highlighted part in the original shape with the highlighted part in the donor shape, and compare our results to SPAGIETTI on re-mixed shapes. Similar to the editing setting, SPAGIETTIT' quality deteriorates on shapes with mixed parts. NEUFORM combines the foreign part more seamlessly into the shape.

---

ground trush
original cuioids edited cubcids
NeSFORRM generalizable
NauFoReM overfited
NeuFFoRo
XXX
Figure 7: Different object categories. Shape edits on PartNet tables and lamps. Similar to chairs, the generalizable model lacks detail and the overfitted model contains artifacts in joint regions, whereas NEUFORM combines the advantages of both.
overfitted representation in non-joint regions. Note that the exact extent of the joint regions is defined by the blending field X defined in Eq.
(ii) Shape Editing. We experiment with shape edits by modifying the parameters of multiple cuboids our shape representation. Editing results of NEUFORM compared to the generalizable and overfitted representations
shown Figure 4. Edits the generalizable representation confirm the trend we saw in the reconstruction task: joints are plausible after edits, but geometric detail is preserved. When editing the overfitted representation, observe significant artifacts near the joints, due to the previously unseen part configuration. Our adaptive overfitting strategy preserves the plausible joints of the generalizable representation well the geometric detail of the overfitted representation.
In Figure 5, compare shape editing COALESCE and SPAGHETTI. Since COALESCE does not supports fine-grained edits, and SPAGHETTI does not support scaling, we compare to each on a separate set of edits. As while SPAGHETTI's geometry deteriorates significantly after an edit.
in the reconstruction, COALESCE struggles with extended joints,
(iil) Shape Mixing. We demonstrate our model's ability to assemble new shapes from the parts of pre-existing ones in Figure 6. We mix and match cuboids and their associated part features from different chairs, and then blend the parts together. For a given query point and its closest part P. we the overfitted representation associated with the shape that P was originally part of. Our method synthesizes much smoother joint connections between parts while preserving their surface details.
Additional shape categories. Figure? shows edit results on tables and lamps, compared the generalizable and overfitted representations. Similar chairs, the generalizable representation is missing shape detail, resulting, for example, artifacts on thin parts, while the overfitted representa- tion struggles with joint areas. In the right-most table, we can clearly see that these artifacts occur both in regions that are joints after the edit, as well as regions that used to joints in the original shape. Adaptive overfitting avoids these artifacts.
Overfitting times. We evaluate the time required for training the overfitted model by measuring reconstruction quality when overfitting for different amounts of time. Times were measured on single V100 GPU. Results are shown in Figure 8. We can see that most regions are sufficiently converged after minutes, showing that the 25 minutes used for the main experiments chosen very conservatively and may be reduced without losing significant quality.

---

overfitting time CD
overfitting time vs. EMD
overfitting time SDF
640
jant reglons
120
joint regions
Jant regins
Chumdir Dicance (x10r)
200 joint reginns
all segions
Divannne
B02- jout regions
Stuell Disence Rold Ezoor
500 reginns
segions
18
600
26 overfinting (niimmes)
dverriiting time (miiuntss)
overfining (minutes)
Overfitting time vs. reconstruction quality, We show the reconstruction quality at different overfitting times, where we measure reconstruction quality in terms of CD, EMD, and SDF error. Note how quality typically converges around minutes.
Figure 8:
5
Conclusions
We have introduced the NEUFORM architecture to enable adaptive mixing of information between generalizable neural neural network, trained a collection of shapes, and an overfitted model, trained on a single shape to capture its idiosyncrasies. We achieved this by designing a network architecture that allows adaptive mixing of networks by carefully blending respective network weights and training history.
Our work is just the first step in the direction of merging overfitted and generalizable models. For example, currently the two models do not have explicit knowledge of each other, adding this knowledge could be interesting future work. For shape editing, this could allow the generalizable network to focus more on joint geometry. Another limitation is the currently non-data-driven blending field. Learning context-based blending factor is a promising next step for facilitating easier and higher quality editing. Shape compression may be a possible future application, where a generalizable prior could efficiently represent the coarse geometry of shape, and overfitted model could be trained to represent details on top of the generalizable prior.
Acknowledgments and Disclosure of Funding
This work was supported by David Cheriton Stanford Graduate Fellowship, ARL grant W91 INF- 21-2-0104, a Vannevar Bush Faculty Fellowship, a PECASE by the ARO, Stanford HAI, and gifts from the Adobe and Snap Corporations.
References
[1] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas J Guibas. Learning representations and generative models for 3d point clouds. ICMI., 2018.
[2] Matan Atzmon and Yaron Lipman. SAL: Sign agnostic learning of shapes from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2565 2574, 2020.
[3] Matan Atzmon and Yaron Lipman. SAL++: Sign agnostic learning with derivatives. arXiv preprint arXiv:2006. 05400, 2020.
[4] Jan Bednarik, Shaifali Parashar, Frhan Gundogdu, Mathieu Salzmann, and Pascal Fua. Shape reconstruction by learning differentiable surface representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4716-4725. 2020.
[5] André Brock, Theodore Lim, James M. Ritchie, and Nick Weston. Generative and discriminative voxel modeling with convolutional neural networks. CoRR, 2016.
[6] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In IEEE Computer Vision and Pattern Recognition (CVPR), 2019.
[7] Angela Dai and Matthias NieBner. Scan2mesh: From unstructured range scans to 3d meshes. In Pruc. Computer Vision and Puttern Recognition (CVPR), IEEE, 2019.
[8] Angela Dai, Charles Ruizhongtai Qi, and Matthias NieBner. Shape completion using enccoder-predictor and shape synthesis. Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2017.
10

---

[9] Thomas Davies, Derek Nowrouzezahrai, and Alec Jacobson. Overfit neural networks as a compact shape representation, 2020.
[10] Theo Deprelle, Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathicu Aubry. Learning elementary structures for shape generation and matching. arXiv preprint urXiv:1908.04770 2019.
[11] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T. Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In ICCV, 2019.
[12] Rohit Girdhar, David F. Fouhey, Mikel Rodriguez, and Abhinav Gupta. Learning a predictable and generative vector representation for objects. CoRR, 0as//66030086333 2016.
[13] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometrie regularization for learning shapes. arXiv preprint 20x::2002.10099 2020.
[14] Thibault Groueix, Matthew Fisher. Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. A papier måché approach learning 3d surface generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 216-224, 2018.
[15] Zekun Hao, Hadar Averbuch-Elor, Noah Snavely, and Serge Belongie. Dualsdf: Semantic shape manipula- tion using a two-level representation, 2020.
[16] Amir Hertz, Or Perel, Raja Giryes, Olga implicit shapes through part aware generation. arXiv preprint arXiv:2201.13168 2022.
and Daniel Cohen-Oa Or. Spaghetti: Editing
[17] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
[18] Gidi Littwin and Lior Wolf. Deep meta functionals for shape representation. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 1824-1833, 2019.
[19] Jemry Liu, Fisher Yu, and Thomas Funkhouser. Interactive 3d modeling with generative adversarial network. International Conference 3D Vision (3DV), 2017.
[20] Julien NP Martel, David B Lindell, Connor Z Lin, Eric Chan, Marco Monteiro. and Gordon Wetzstein. Acorn: Adaptive coordinate networks for neural scene representation. arXiv preprint 1XXi:22005002785 2021.
[21] Lars Meseheder, Michael Occhsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
[22] Ben Mildenhall. Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren neural radiance fields for view synthesis. In European Conference on
Ng. NeRF: Representing Computer Vision, pages 405-421. Springer, 2020.
[23] Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy Mitra, and Leonidas Guibas, Structedit: Learning structural shape variations. arXiv preprint arXiv:190$ 190.:00998.00057... 2019.
[24] Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy Mitra, and Leonidas Guibas. Structurenct: Hierarchical graph networks for 3d shape generation. ACM TOG, 2019.
[25] Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, and Hao Su. PartNet: A large-scale benchmark for fine- grained and hierarchical part-level 3D object understanding, In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[26] Luca Morreale, Noam Aigerman, Paul Guerrero, Vladimir G Kim, and Niloy J Mitra. Neural convolutional surfaces. In Proc. CVPR, 2022.
[27] Luca Morreale, Noam Aigerman, Vladimir G Kim, and Niloy Mitra. Neural surface maps. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4639-4648, 2021.
[28] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 165-174, 2019.
[29] Omid Poursaeed, Matthew Fisher, Noam Aigerman, and Vladimir G. Kim. Coupling explicit and implicit surface representations for generative 3d modeling. ECCV, 2020.

---

[30] Charles R. Qi. Hao Su, Kaichun Mo, and Leonidas J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation, 2016.
[31] Ayan Sinha, Jing Bai, and Karthik Ramani. Deep learning 3d shape surfaces using geometry images. In ECCV, 2016.
[32] Vincent Sitzmann, Julien NP Martel, Alexander W Bergman, David B Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. arXiv preprint arXiv:2006. 09661, 2020.
[33] Hao Su, Haoqiang Fan, and Leonidas Guihas. point generation network for 3d object reconstruction from a single image. CVPR, 2017.
[34] Minhyuk Sung, Zhenyu Jiang, Panos Achlioptas, Niloy J. Mitra, and Leonidas Guibas. Deformsynenet: Deformation transfer via synchronized shape deformation spaces, 2020.
[35] Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3d shapes. In Proc. CVPR. pages 11358- 11367, 2021.
[36] trimesh. Trimesh Ihttps://timmhhorrg// 2022.
[37] Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian. FoldingNet: Point cloud auto-encoder via deep grid deformation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 206-215, 2018.
[38] Wang Yifan, Lukas Rahmann, and Olga Sorkine-Hornung Geometry--cosisseet neural shape representation with implicit displacement fields, 2021.
[39] Kangxue Yin, Zhiqin Chen, Siddhartha Chaudhuri, Matthew Fisher, Vladimir G Kim, and Hao Zhang. Coalesce: Component assembly by learning synthesize connections. In 2020 International Conference on 3D Vision (3DV), pages 61-70. IEEE, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We do not forsee any potential negative socictal impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] Only empirical results.
(b) Did you include complete proofs of all theoretical results? [N/A] Only empirical results.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [No] The data is publicly accessible, well-known dataset. If accepted, we plan release code, including instruction how reproduce the results, one or two months after the notification.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you using existing assets (c.g., code, data, models) or curating/releasing assets...
12

---

(a) If your work uses existing assets, did you cite the creators? [Yes] Dataset was cited.
(b) Did you mention the license of the assets? [No] The dataset is publicly avilable and the license can easily be found.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A] No new assets presented.
(d) Did you discuss whether and how consent was obtained from people whose data you'le using/curating? [N/A] The dataset is publicly available under the MIT license.
(c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A] The dataset synthetic.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A] No human subjects were used as not part of the research.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A] No human subjects were used as not part of the research.
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A] No human subjects were used as not part of the research.
13


Reviews:
[{"review_id": "wZYTZJq_sf", "reviewer": "Reviewer_iV53", "summary": "This paper presents a shape editing method which works by combining generalized and overfit occupancy models. By applying the generalized model to edited regions, the overfit model to unedited regions and smoothly interpolating between them they naturally achieve more attractive shape predictions. With this method they demonstrate superior shape reconstructions, shape edits and and shape mixing. ", "questions": "Why is the overfit model trained without joint ground truth in the reconstruction task? I am not sure it makes sense to evaluate the overfit  or the NEUFORM models in this setting as reconstruction is a generalization task and you are overfitting to shape in both situations. I may be miss-understanding something here though.", "limitations": "Yes , though I mention above a possible addition.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths of this paper is that the idea is presented and works well. I have no qualms with the methodology, presentation or results. \n\nMy main concern is the time scale needed to begin editing a new shape. It is indicated that it takes 25 mins to train an overfitting model, which if I understand will be required for each new shape you wish to edit. This seems prohibitively long and I think should be addressed in the paper somewhere, with a table comparing the time to edit for the various methods you compare to. At the very least it should be mentioned in the limitations. \n\nWith that said the editing and mixing results are impressive, and the idea and results are presented very well.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GE5-ryT7M4L", "reviewer": "Reviewer_WuvA", "summary": "The paper presents NEUFORM, a neural representation that combines the benefits of detail preservation of overfitted representations and edit-ability of generalizable representations. In the generalizable representation the shape is represented as a set of part parameters and an occupancy function models the occupancy field of the shape at any query location. The model consists of a part mixing network, a part query network, and a global occupancy network. An overfitted occupancy function with the same architecture as the generalizable representation is used to capture geometric details. The part latent vectors are frozen before overfitting and only the query network and the occupancy network are updated. A scalar blending field defines a blending factor between the generalizable and overfitted representations at each query point. The two representations are fused by blending between the parameters, weights and the features of the networks given the blending factor. NEUFORM is evaluated on reconstruction, part based shape editing and shape mixing. Qualitative and quantitative experiments demonstrate superiority of NEUFORM over SPAGHETTI [17] and COALESCE [39] as the baselines.\n  ", "questions": "- As commonly done in prior work, the authors train separate models on each category, and they do not discuss whether training a generalizable model jointly on multiple categories help or hurt the performance. The model architecture is not directly tied to a fixed category, and shapes from different categories can share similar-looking parts. Is it possible to share knowledge across categories to further improve reconstruction and editing?  \n- It would be good to have the same surface color for shapes from NEUFORM vs SPAGHETTI in Figures 5 and 6. ", "limitations": "- There is no potential negative societal impact. \n- The authors note limitations of their work in terms of limited explicit knowledge-sharing between over-fitted and generalizable models and non-data-driven blending field. ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality**:\n- The proposed idea of combining overfitted and generalizable representation by adaptively blending corresponding network weights is novel and interesting. \n- The authors notice that the problem of shape editing needs both detail preserving and generalizable representations, and it lends itself well to the proposed adaptive model.\n- The authors carefully design the generalizable and overfitted networks so that they can be combined well in a hybrid model. \n- They use the idea of blending network weights which is trending topic in machine learning but less studied in 3D deep learning.  \n \n**Quality**:\n- The paper is well-motivated and well-written.  \n- Experiments are thorough and convincing and sufficient details are provided for the experiments to be reproducible. \n\n**Clarity**: \n- The proposed approach is described clearly. The authors provide intuitive explanation of their choices as well as mathematical abstraction of the proposed model. \n\n**Significance**:\n- The problem of shape reconstruction and editing is of great practical value and the current state-of-art methods create noticeable artifacts and are far from perfect. \n- The paper provides significant value to the research community as it shows considerable quantitative and qualitative improvements over the state-of-the-art across tasks such as reconstruction, shape editing and shape mixing. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8wkxZQXDAk", "reviewer": "Reviewer_JEir", "summary": "This paper proposes an architecture and training scheme for 3D shape reconstruction that combines the generalizability of models that parameterize an entire shape family with the level of detail of models overfit to a single shape. To this end, a generalizable part-based model is trained for the shape family and another model with the same architecture is overfit to a specific shape. The two models are then combined in order to take advantage of edits via the latent space of the former while keeping the details captured by the latter. As a result, it is possible to perform part-based edits where novel geometry is hallucinated where necessary (i.e., around part joints) while existing geometry (the parts themselves) are faithfully reconstructed.\n\nThe authors show experimental results on sheep reconstruction, shape editing, and shape mixing, and they compare to state-of-the-art part-based deep shape reconstruction methods. ", "questions": "Looking at certain results, such as the first row of Figure 3, it seems that the model is sometimes unable to extrapolate local part texture/style when parts are edited (e.g., the slate on the chair end halfway up the seat on the edited model). \n\nCan this method be leveraged for compression in a superior way than previous similar models? Is there any way to significantly shrink the generalizable model, since much of the burden of representation is now carried by the overfit model?\n\nIs it possible to improve the time required to adapt this approach to every new 3D model (e.g., retraining the overfit model)? This seems to be a bottleneck compared to using a purely generalizable model; could some initialization or fine-tuning alleviate this?\n\nMinor typos:\n\nL2: \"form\" -> \"from\"\n\nL139: $P$ -> $\\mathcal P$\n\nL188: $\\overline P$ -> $\\overline {\\mathcal P}$\n\nL232: $1e-4$ -> \"1e-4\" or \"0.0001\"", "limitations": "Limitations and directions for future work are adequately discussed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think this paper is very clearly written and proposes a reasonable and fairly simple extension of previous shape reconstruction methods. I like that it makes a step towards neural implicit reconstructions that are more usable in practice (thank to the ability to accommodate higher levels of detail) while still retaining the benefits of a data driven approach. The experimental results are pretty convincing, and there is a clear advantage in the comparisons.\n\nOne drawback is the requirement to overfit a model (which takes ~30 min) for each new target set of parts. Additionally, the method is currently not always able to extrapolate local detail when parts are edited, since the overfit model is limited to the known regions of the parts. This detracts a bit from the claim of being able to make semantically reasonable edits at a high level of detail.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept