GenSDF: Two-Stage Learning of Generalizable Signed Distance Functions
Authors: Gene Chou
Ilya Chugunov
Felix Heide
Venue: NeurIPS 2022 Accept (2022)
Keywords: implicit neural representations
3d object reconstruction
signed distance functions
meta-learning
generalization

Abstract: We investigate the generalization capabilities of neural signed distance functions (SDFs) for learning 3D object representations for unseen and unlabeled point clouds. Existing methods can fit SDFs to a handful of object classes and boast fine detail or fast inference speeds, but do not generalize well to unseen shapes. We introduce a two-stage semi-supervised meta-learning approach that transfers shape priors from labeled to unlabeled data to reconstruct unseen object categories. The first stage uses an episodic training scheme to simulate training on unlabeled data and meta-learns initial shape priors. The second stage then introduces unlabeled data with disjoint classes in a semi-supervised scheme to diversify these priors and achieve generalization. We assess our method on both synthetic data and real collected point clouds. Experimental results and analysis validate that our approach outperforms existing neural SDF methods and is capable of robust zero-shot inference on 100+ unseen classes. Code can be found at https://github.com/princeton-computational-imaging/gensdf

GenSDF: Two-Stage Learning of
Generalizable Signed Distance Functions
Gene Chou
Ilya Chugunov Princeton University
Felix Heide Princeton University
Princeton University gchou0princeton. edu
chugunov@princetooncom
fheide@princettn.co
Abstract
We investigate the generalization capabilities of neural signed distance functions (SDFs) for learning 3D object representations for unseen and unlabeled point clouds. Existing methods can fit SDFs handful of object classes and boast fine detail or fast inference speeds, but do not generalize well to unseen shapes. We introduce two-stage semi-supervised meta-learning approach that transfers shape priors from labeled to unlabeled data to reconstruct unseen object categories. The first stage uses an episodic training scheme simulate training on unlabeled data and meta-learns initial shape priors. The second stage then introduces unlabeled data with disjoint classes semi-supervised scheme diversify these priors and achieve generalization. We assess our method on both synthetic data and real collected point clouds. Experimental results and analysis validate that our approach outperforms existing neural SDF methods and is capable of robust zero- shot inference on 100+ unseen classes. Code can be found here.
Introduction
Learned 3D representations form the core of shape recovery [1], robotie manipulation [2], scene understanding [3], and content generation [4] tasks. While simple to interpret, high-quality explicit 3D representations can come with slew of hidden costs. Conventional voxel grids without acceleration suffer from cubical memory requirements [5, 6, 7J. dense point clouds contain large amounts of redundant information make querying local structures computationally expensive [8, 91, and direct optimization of mesh representations is often infeasible [10]. As alternative, works such as [11, 12, 13] propose implicit learned representations; networks trained to functionally approximate some underlying 3D geometry. These methods offer lower memory requirements and faster training and inference speeds [14] than conventional explicit representation methods [5, 8).
Neural signed distance functions (SDFs) [11] are subset of this work which implicitly model the distance from a queried location to the nearest point on a shape's surface - negative inside the shape, positive outside, zero at the surface. Existing methods are capable of approximating diverse synthetic geometry, including multiple objects [11, 15, 16] and varying levels of detail for single objects [17, 18]. However, most successful methods ground truth signed distance values to fit to an object. As ordinary point clouds lack these values, a sully-sueer approach cannot learn from such unlabeled data.
fully supervised -hey require access to
To replace the need for any annotated data, fully unsupervised approaches [19, 20, 21] operate directly on point clouds. These methods leverage pseudo-labels and proxies for ground truth signed distance values, for example the unsigned distance to the nearest point in the point cloud. Unfortunately. existing unsupervised methods introduce just three or four additional additional object categories. As a result, these approaches can also not take advantage of large unlabeled point cloud datasets [22, 23, 24] and their real-world applications are limited. Our proposed method lifts this limitation, and learns from large-scale
only reconstruct handful of shapes, and break down when
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

unlabeled data, successfully reconstructing an order of magnitude more categories in inference than seen during training.
To achieve this we propose a semi-supervised learning approach. Existing semi-supervised methods use labeled and unlabeled data of the same class to improve intra-class performance for tasks such as 3D reconstruction from images [25, 26, 27] and scene reconstruction from point clouds [3, 28]. Although they perform well on seen data, these methods are trained on labeled and unlabeled data of the same class, and so do not by themselves generalize well to unseen classes. We propose a novel cemi-suuereised meta-learning learning approach that learns geometric relations between labeled and unlabeled data of non-overian non-overlapping categories. One which learns an SDF that generalizes to a diverse set of unseen, out-of-distribunt
geometries.
We train in two stages. In the first meta-learning stage, we learn shape priors from the labeled set. Different from existing supervised methods, we introduce episodic training scheme where we split data into subsets with disjoint classes to learn a generalized shape representation. One part of this split is treated as labeled supervised set of data, with ground truth signed distance penalties, while the other is used to emulate unlabeled data samples, with only self-supervised losses, We validate that this first stage provides a generic shape prior which favors expressiveness at the cost of reconstruction quality. In the second, semi-supervised stage, we introduce a new fully disjoint set of unlabeled data to the training scheme. Successful training hinges on our novel self-supervised sign prediction loss, which diverges from the unsigned convention of previous works [19, 20, 21J. This second stage allows the model to ingest large amounts of unlabeled data to diversify the shape model learned in the first stage, which significantly increases the quality of the signed distance predictions over a wide set of unseen shape classes.
We evaluate our method on Acronym [29], a synthetic dataset, and YCB [23]. a collection of real- world point clouds. We validate the generalization capabilities of our approach, and find that outperforms state-of-the-art learned SDF methods and is capable of robust zero-shot inference [30] on 100+ unseen classes and scanned point clouds.
2 Related Work
Learning Implicit Surface Representations. Implicit neural representations form the backbone of many modern 3D reconstruction and view synthesis works [11, 12, 13, 31, 32]. They use neural networks as universal approximators [33] to directly learn a mapping between spatial or spatio-temporal [34] coordinates and scene attributes, which offers fulle-differentiable and space- efficient [28] way to represent 3D geometry. Park et al. [11] first experimented with mapping coordinates to signed distance values- neural signed distance function (SDF)- whereby an object is implicitly represented by the distances between 3D coordinate queries their closest surface point.
With labeled data when all 3D query points are annotated with ground truth signed distance values- predictions of signed distance values can be supervised directly [11, 15]. Existing methods [11, 15] require test-time fine-tuning even for a single class, prohibiting real-time inference or prediction on unlabeled data, or overfit to single objects [17, 18]. Occupancy-baac learn rudimentary shape prior, but still struggle to reconstruct fine features in unseen shapes. The proposed semii -umi-supeeiiee method learns a robust shape prior, and substantially outperforms fully supervised methods when tested set of over one hundred unseen categories.
methods [16, 35, 36] can
With only unlaheled data - un-annotated point clouds approaches try to generate proxies for signed distance values for supervision. Ma et al. [19] look at the unsigned distance to the nearest point in the point cloud as a proxy, and Atzmon and Lipman [20] propose a class of loss functions and geometric initializations such that these unsigned distances converge to signed values. We propose signed nearest neighbor loss and demonstrate that our approach, unlike existing unsupervised methods, is able to reconstruct thin object structures and converge dataset with large object variance.
Semi-supervised Surface Representations. Traditional semi-supervised methods rely on learning from datasets where each category contains labeled and unlabeled samples. During inference they rely on pseudo-labels [37, 38], or employ ensemble techniques to enforce consistency [39]. These methods cannot be directly applied to target data from unseen categories.

---

Meshes X
Labeled Data X₃
Model
Point Cloud
Encoder
11000
Mesh
Point Cloud
Spit
Shape
Decader
Query Points
Concet
Unlabeled Data XelR
Query Points
Point Clouds R
Point Cluid
Pradicted
Values
L
Craap
Stage (Meta)
Stage (Semi)
Both
Figure 1: Our two-stage learning pipeline. The first stage (red) episodically splits the dataset into two subsets of disjoint categories of emulated labeled and unlabeled data, trained with combined supervised Suu and self-supervised Leett loss. The second stage (orange) introduces real unlabeled data with non-overlapping classes in a semi-supervised fashion improve model generalization.
For 3D representation tasks, existing methods [25, 26, 27] use semi-supervision reconstruct geometries from images or point clouds [3, 28]. Elich et al. [25] pretrain their model with labeled data to learn 3D representations, to recover known shapes from RGBD data. Li al. [3] use labeled data as dense boundary values for semi-supervised SDF learning, and fit road scenes from point clouds. While these approaches improve predictions on seen data, they cannot generalize well to unseen classes, even with test-time optimization [3]. Recently, person re-identification and domain adaptation methods perform semi-supervision on non-overlapping labeled and unlabeled classes and have shown to generalize better [40, 41, 42]. This has not been attempted for learning SDFs, and our method introduces meta- leta-learnnng approach to learn shape priors, where we emulate training on unlabeled data with pseudo-labels.
The ability to learn generalizable 3D representations is necessary for real-world applications, real objects exhibit significantly more diversity than any dataset. In learning implicit functions, one approach [43, 44] is to break down shape features to find common properties, but this also requires prior knowledge about target shapes which might not be available.
Generalization.
We propose a meta-learnigg approach that does not make assumptions or require prior information about target data. Meta- Mleaaleerning has been extensively studied in few-shot learning [45, 46], domain generalization [47], and image retrieval [39], 3D reconstruction from 2D or 2.5D images [48, 49, 50], and point cloud classification and segmentation tasks [51, 52, 53]. However, meta-learning has not been extensively investigated in the context of SDFs. The most relevant method is MetaSDF [15], albeit it is fully supervised and exhibits large time and memory complexity due to inner loop gradient descent steps. Furthermore, MetaSDF overfits to single objects and requires ground truth signed distance values during test time. The proposed episodic training approach avoids this costly inner loop, and learns unified network that only requires a raw input point cloud for reconstruction.
3 Semi-supervised Meta-Learning of Implicit Functions
We next formalize the problem we are solving in Sec. 3.1, then describe the proposed learning method and training objectives in Sections 3.2 and 3.3.
3.1 Problem Formulation
Conditional Signed Distance Functions We are interested in learning a generalized representation for shapes. We learn signed distance function (SDF), which is continuous function that represents the surface of a shape approximated by the zero level-set of a neural network. Given a raw input )- with N points and 3D query point SDF Φ. R x (p. E R l== R such that the function & predicts the signed distance value for 3D coordinate conditioned on sparse point cloud. Then a unified function that can be directly applied to any target point cloud.
point cloud P {p: E R
R3, we learn conditional
3

---

P. + P)
(1)
where s denotes the predicted signed distance value between x and the shape described by P.
The surface boundary of shape described by P is its zero-level set (o()))),, which can be expressed as
(o() f2 R ((2, P) = =0..
(2)
Thus, given a trained Q we can visualize the surface of some by drawing its zero-level set.
Problem Setting We assume both labeled and unlabeled datasets are available, denoted X and R. respectively. X = {Px, SDF), where Px is set of point clouds and (DF denotes the ground-truth SDF operator that is defined for all query points z E RR3 R (Pr), where Pn is set of point clouds belonging to classes disjoint with those of Px. Each point cloud is a set of 3D coordinates that describes the surface boundary of an object, and contains no additional information such as normals.
To learn a generalizable conditional SDF, we propose a two-stage semi-supervised meta-learning algorithm. In the first stage, we aim to learn shape priors using X. As illustrated Figure we propose to train our model using an episodic scheme [47]. where every f epochs we randomly split X into two subsets with disjoint categories. In the second stage, as shown in Figure I, we continue to train the pretrained weights from the first stage and train the model on X and R in semi-supervised fashion. We obtain a unified neural network & that learns from one dataset and can generalize to represent SDFs of objects in unseen categories during test time; i.e., zero-shot inference [30].
3.2 Stage 1: Meta-Learning Shape Prior
Our eventual goal is to train on labeled and unla- beled data simultancously, in a meta-learning approach as we learn shape priors from X. Every epochs, split X into two subsets X2 and Xy with disjoint categories. We train X,, with supervised loss Suup. simi- lar to typical supervised approaches on labeled data. We take away ground-truth signed distance values from Xy making "pseudo-unlabelle" and train it with our self-supervised loss Leelt. Our training objective is to minimize
mimic this goal
Outside
Leelt
J/2 2
Loup
Pt
PL
Inside
Figure 2: Our self-supervised (left) loss uses the nearest neighbor of query point to approximate signed distance. The supervised loss (right) in- stead penalizes based exact signed distance.
Lmeta = (Xi.) + Am (Xtr
(3)
where Am controls the strength of the self supervised loss Cuelt- This meta-learning approach intends accomplish two goals. First, the model learns backpropagate with our self-supervised loss, which we need for training on unlabeled data in the second stage. Second, episodic training on non-overlapping categories forces the model to learn a generalized shape representation rather than overfit to geometries within single category.
Compared to directly training on labeled and unlabeled data, this meta-learning process provides flexibility as we can tune the split frequency and ratio of labeled and "pseudo-unlabeled" data. The model repeatedly sees the same point clouds with and without labels, shape features generated during labeled splits are shared with "pseudo-unlabeles" supervised training from scratch is unstable because errors in unlabeled data accumulate (see Tab. 3). We empirically show in our ablation study that our method has stronger representation power for both seen and unseen classes. We provide an example of a training step Alg. and diagram in Fig. 1. Next, we explain our loss functions in more detail.
splits. In contrast, we find that semi-
Supervised Loss Component In the supervised setting, we have the ground truth SDF operator; i.e,, we know the signed distance values for all query points Therefore, the loss function is simply
Coup
[x(xx) SDF(xx)ll.
(4)
K
KEK
where ||
is the LI loss.

---

Algorithm Stage I: Meta-Learning a Shape Prior
1: preprocess: a labeled set X
2: initialize: learning
1J, split frequency f, hyperparameter Am, model & with parameters
3: for in (nungennum__ee ranpe(nnumepochh)::
4:
if (e mod f) = 0:
5:
XL. XU t split(X)
6: Loup 1x Σ
Eq. (4)
(x,SDF)EX2
7:
Laell = |Xul (a.1)EXU
1 Σ
(1
||--I|| (2;; 0*), 1|
x-t
> Eq. (7), simplified
8:
Lmota -aup + Am Leelf
>> Eq. (3)
9:
gc+1 4- -/ Lieta
10: return
Self-supervised Loss Component for Unlabeled Training Samples For each point x E R we use its closest point E P to approximate its projection to the surface, defined as
argmin || pl|2- pEP
(5)
In the literature [54], it is common to use
T=r-VD(z)ee
(6)
to approximate the closest point on the object surface to query point x given some trained SDF function for tasks such as collision detection. We build on this insight: we want predict to approximate t on the point cloud by using the predicted signed distance value. However, directly using Eq. (6) leads to highly inaccurate sign predictions. There are no explicit penalties for predicting the wrong sign and when the query point is close to the surface, an l loss between and can be very low even when the sign is predicted incorrectly. For a surface boundary to be accurately estimated, there must be accurate sign changes between 3D coordinates.
find that
Thus, we substitute V (z: with the direction vector between / and x. For incorrect sign predictions, the vector points in the direction opposite to the surface, guaranteeing larger error. We denote our approach self-supervised because the predictions of the signs are used as labels. This makes it possible leverage ground-truth signs during the meta-learning stage guide training since the sign and absolute distance can be disentangled. In contrast, (=(x) in the direction of maximum distance change. Our training objective is to minimize the distance between ī and I. Furthermore, they have a distance value of 0, which helps with thinner structures. Our final loss function
is calculated as a unit that points
add additional & loss by using the points on point clouds as
Leell TR
ΣΣ
11 4
4
A
1
(P(()))
(7)
REK
PEPanish
where Ao determines the weight of the point cloud predictions and
I-t
((x)
if (x:) 0,
I - E[
(8)
T - 1 -
(xx)
if (x)) < O.
Previous work [19] has predicted by directly calculating Eq. (6) during training, but their resulting sign predictions are substantially less accurate than ours for the reasons we discuss. We validate this with the results shown in Tab. 1. We note that point cloud P is sampled from a surface and does not represent the continuous boundary. Thus, / the nearest neighbor that exists P but not necessarily the true one that exists in the target surface we want to reconstruct. As result there are inaccuracies and previous unsupervised work [19] that use P as a proxy are not robust when training on multiple classes. Our two-stage approach lifts this constraint.

---

Seen Classes
Unseen Classes
DeepSDF
18.26/10.87
33.19/29.74
MetaSDF
4.269/ 11536
4.834/17766
ConvOceNet
1.348/0.713
2.333/ 1.032
Proposed 0.351 /0.166
0.407 /0.250
Chamde Dert 20"1
20td
2014
Figure 3: Mean/median Chamfer Distance (CD) of 20 seen and 166 unseen classes (right, scale 10 +). For method report numbers prior to any test-time refinement. We include a histogram plot for visualizing the per-class CD distribution of the unseen classes (left, log scale).
3.3 Stage 2: Semi-supervision with Unlabeled Training Samples
In our second stage, visualized in Fig. I, we train our conditional SDF, pretrained in the meta-learning stage, on both labeled and unlabeled data in a semi-supervised fashion. As a result of the learned priors, training remains robust even with large amounts of unlabeled data. Furthermore, continuing to train with labeled data provides strong supervision to ensure inaccuracies from unlabeled data do not build up. We show in ablation study (Sec. 4.4) that semi-supervised training from scratch can be unstable. Given labeled set and unlabeled set R, our training objective is to minimize
Leemi = (X) + X Sell (R)
(9)
where X controls the strength of the self supervised loss Leelf.
4 Experiments
In this section, we analyze and validate the proposed representation method for seen and unseen object categories. We begin with a discussion of implementation details and experimental setup.
Implementation For our experiments, we borrow the encoder architecture from [16] which is modified PointNet [8] that uses plane projection to learn local geometric features, and parallel UNets [55] aggregate spatial information. For the decoder we use an 8-layer multi-layer perceptron with 512-dimensional hidden layers, similar to the architecture from [11, 15, 19, 20]. We emphasize, that our two-stage training pipeline is designed to archileecture-annotiic. and we provide results for alternate encoders and decoders in the supplement. For Eqs. (3) and (9) we the set the loss coefficients to Am = A 0.1, and X = 0.01 in Eq. (7). We provide full architecture description and additional training details in the supplemental document.
Experimental Setup We source training and evaluation data from Acronym [29] and YCB [23]. Acronym is post-processed subset of the popular ShapeNeet [56] dataset which contains 8,872 clean, synthetic 3D models in 262 shape categories. In contrast to ShapeNet, all of Acronym's meshes are watertight, which ensures that they have well-defined SDFs. We use the 20 largest classes (2,995 meshes) as labeled dataset X for all fully-supervised methods, including our meta-learning first stage. The next 76 classes (3,408 meshes) form our unlabeled dataset R for semi-supervised training. The remaining 166 classes (1,525 meshes) are withheld for testing. Preprocessing details can be found in the supplemental document. The YCB dataset [23] contains point clouds from 3D scans of common real-world objects and does not contain ground truth meshes. We use these point clouds as additional unlabeled test data to assess SDF predictions
nattof-distribution objects.
Given point cloud that describes the surface of an object, the proposed method returns implicit signed distance function ((P). We construct cube with each dimension ranging from to 1, then discretize it into a set of grids with resolution 2563. We run Marching Cubes [57] at each grid point to reconstruct mesh surface for qualitative assessment. For quantitative evaluation we measure the Chamfer Distance [8] between 30,000 randomly sampled points on the surfaces of the reconstructed and ground truth meshes.
Evaluation
Experiments We conduct experiments to investigate the generalizability and representational power of our method. Specifically, we analyze the effects of reconstructing seen and unseen objects (Sec. 4.1), self-supervision on unlabeled data (Sec. 4.2), performance on real-world point cloud seans (Sec. 4.3), and ablation experiments (Sec. 4.4).

---

Cloud
DeepSDF
MetaSDF
ConvOocNet
Proposed
Ground Truth
Lamp
Desk
Figure 4: Reconstruction results for seen objects objects in the training set. DeepSDF and MetaSDF struggle to recover thin structures and non-convex surfaces due to their lack of learned shape priors. Compared to ConvOccNet our approach produces fewer artifacts in empty space.
DeepSDF
ConvOcoNet
Proposed
Ground
Bunny
Horse
Beskee
Bookcase
Figure 5: Reconstruction results for unseen categories not included during training. Only our method recovers fine detail, such as the gaps Shelf's s surfaces and the curves of Bunny and Horse.

---

4.1 Reconstructing Seen and Unseen Shapes
In this subsection, we compare our method to existing approaches: DeepSDF [11], MetaSDF [15], and Convolutional Occupancy Networks (ConvOccNet) [16] on both seen and unseen shapes. For reconstructing seen data, we evaluate results on the shared classes (see the experimental setup paragraph in Sec. 4). For testing on unseen classes, disjoint with any class used for training. To improve visual results, we apply refinement by fitting the model on the point cloud for a few iterations. Specifically, we employ our self-supervised loss without using any additional information beyond 5,000 raw input points. In contrast, DeepSDF [11] and MetaSDF [15] use ground truth signed distance values for refinement. We include specific refinement details in our supplement. Since ConvOceNet [16] does not perform refinement, we report quantitative results of our proposed method without refinement for fair comparison.
the remaining 166 categories that are
Seen Categories. Fig. 4 shows reconstruction results on seen shapes. We observe that existing methods and our approach are capable of reconstructing seen data. However, DeepSDF and MetaSDF fail to reconstruct thin structures such the lamp poles due to the lack of a shape prior. Quantitatively, reported in the middle column Fig. 3, even though we use the same encoder as ConvOceNet [16] to learn shape priors, our two-stage approach achieves lower CD by an order of magnitude.
Unseen Categories. When tested on unseen categories, as shown in Fig. 5, only our proposed method is able to reconstruct unseen categories, and accurately represents fine details such as narrow gaps in the Shelf example. Quantitatively, as shown in the right column of the table in Fig. 3, the proposed method achieves better performance. Notably, the gap between the CD of seen and unseen classes is lower in our case. While the CD of most methods double between seen and unseen classes, there is only a marginal increase for ours.
4.2 Comparison to Unsupervised Methods on Unlabeled Data
A self/unsupervised method is essential for our semi-supervised stage to train on unla- beled data and for test-time refinement on only the raw input point cloud. In this subsection, we compare to existing unsupervised base- lines: NullalPull [19] and SAL [20] For a fair comparison, we use only our eef-ssupervied loss in Eq. (7) to train directly on unlabeled data and do not perform test-time refinement (Proposed (self) in Tab. 1).
Table 1: Mean/median Chamfer Distance for self- and un-supervised methods when trained on 5 objects from the same QueenBed class and 352 objects from 4 classes (QueenBed, Plant, TrashBin, DeskLamp). We report pre-refinement numbers for our approach, scale is 10-1
Queen Beds
Four Classes
SAL NeuralPull Proposed (self)
1.786 / 1.869
65.55
49.38
1.745 / 1.502
2.419
1.812
Tab. 1 validates that when training a single Queen Bed class, our explicit sign penalty al- lows our self-supervised method to reconstruct sharper details from irregular surfaces. On the other hand, existing methods [20, 19] produce low losses for close approximations the absolute values of the signed distances, rather than accurate sign predictions, resulting in loss of details where there is abrupt change in the surface (see supplement for visualizations). All non-supervised methods we test on, including ours, degrade when training degrade quickly as the number of meshes increases.
0.146 0.139
0.218
0.184
Proposed
0.081
0.077
0.089
0.061
The proposed method with semi-supervision scales well as increase the number of classes. Thus, our method can tackle large-scale operations and take advantage of the availability of large amounts of unlabeled data. We also show in our supplement that using our self-supervised variant from this section leads to significantly more robust training.
4.3 Reconstruction of WorldWorld Raw Point Clouds
Next, we test our model on the YCB [23] dataset, which is a real-world point cloud dataset acquired from multi-view RGBD captures. The fused multi-view point clouds in this dataset resemble
Approaches such as [17, 58, 18] are designed to overfit to single objects, which significantly deviates from the focus of this paper, so do not include comparisons with these methods.
8

---

Point Cloud
ConvOcoNet
Proposed
Paint Cloud
ConvOcoNel
Proposed
Pitcher
Bowl
Figure 6: Reconstruction results for real scanned data from unseen categories in YCB [23]. This is difficult setting, as the data has a different canonicalization, scaling, and distribution. Our proposed method is able to recover structures such as Pitcher's ' handle and the inside of Bowl.
Point Cloud
ConvOozNet
Proposed
Point Cloud
ConvOccNet
Proposed
Drill
Mustard
100
Pitcher
Banana
Figure 7: Reconstructions of YCB [23] after recentering and normalization of input point cloud.
input measurements for a robotic part-picking or manipulation task. We demonstrate robust mesh reconstructions of the measured data, e.g., recovering the "handle" of pitcher in Fig. and 6, which may serve as input complex robotic grasping tasks.
YCB is a different domain from Acronym [29] and we show two experiments to thoroughly test performance of outtof-distribution data. First, we perform scaling or normalization of the input point cloud, shown in Fig. 6. Then, we also recenter and normalize the input point cloud (which we would do in a real-world setting) and show results Fig. We omit results from DeepSDF [1 and MetaSDF [15] since they require ground truth signed distance values during test time. As shown in Fig. 6 and 7, ConvOccNet [16] loses detail such as the handle of the pitcher. Our proposed method able to generate 3D shapes with detail and plausible geometry, even generalizing to different scaling, illustrating its ability of handling in-the-wild and out-of-distribution data.
To further understand what level of noise our method is susceptible to, we gradually add Gaussian noise with mean zero and variance o² to the Pitcher point cloud in YCB [23] and evaluate its CD, Both quantitatively and qualitatively, our method is able to adapt to noise with o² < 0.5, but afterward the reconstruct object degrades severely. We show results in Table 2. In the future, adding more diverse unlabeled data to the training of the proposed model, including different domains, scaling, and noise, may further boost the generalizability to any target data.
Table 2: Adding Gaussian noise with zero mean and variance o² to the Pitcher point cloud in YCB and evaluating Chamfer Distance (CD), scale 10 10-4
0.0
0.01
0.05
0.1
0.15
0.2
CD 0.870
0.911
3.122
5.208
7.429
7.391
4.4 Ablation Experiments
In this subsection, we analyze individual components of our two-stage learning. We show quantitative results in Tab. 3. Only meta refers here to the meta-learning first stage on 20 classes. This outperforms existing fully supervised methods (see Fig. 3) for both seen and unseen data, while using the same amount of data and no additional manual annotations.

---

The favorable results validate the generalization capabilities of our episodic training scheme. Our ceni-supervision stage further improves recon- struction quality by introducing more shapes without requiring annotations. Only semi refers to only training the proposed semi-supervision stage from scratch all classes without transferring the learned representation from the meta-learning stage. Even though this stage has access both labeled and unlabeled data, it has a higher CD compared to our final two-stage approach. This is because errors in the proxies of unlabeled data ac- cumulate and result in training that is less robust. This indicates that our meta-learning stage indeed builds strong foundational prior that ensures the second stage semi-supervised training is robust.
Table 3: Mcan/median Chamfer Distance (CD) for various ablations 20 classes of seen data and 166 classes of unseen data. We report pre- refinement numbers, scale is 10
Seen Classes
Unseen Classes
Proposed
0.351/0.166
0.407/0.255
Only meta
0.503/0.22 10.2
0.515 /0.297
Only semi
0.821 0.821/0.510
0.889/0.61
Sup. 20 cls Sup. 76 cls
0.695/0.468
1.164/1.36)
0.713/0.550
0.908/0.866
Sup. (20 cls) refers to training the labeled set of 20 classes without episodic training and resplitting. Sup. (76 cls) refers to training on 76 classes with ground truth annotations but without episodic training and resplitting. The CD for seen classes is similar to the Only meta setting, but meta- learning method outperforms both baselines by a large margin when comes to reconstructing unseen classes, despite being trained with fewer categories.
5 Conclusion
In this work, we devise two-stage semi-supervised meta-learning approach that achieves strong generalization to unseen object categories. This method able to fit to any target point cloud without additional annotations and outperforms existing SDF methods in both seen and unseen categories. Our method is not without limitations, which we plan to address in the future. Similar to existing SDF representations [16], we do not achieve real-time inference due large number of model parameters. Learning from large-scale multi-object point clouds, with abundant datasets for automotive scenes [59, 60) is also exciting future direction. The proposed approach may also serve representation for inverse rendering methods that tackle vision tasks in an maaysis-by-ssythee by-synthesis approach.
Acknowledgments
We would like to thank Hong- Yuan Liao and Chien- Yao Wang from Academia Sinica, Taiwan for constructive discussions and advice in completing this work. llya Chugunov was supported by NSF Graduate Research Fellowship. Felix Heide was supported by an NSF CAREER Award (2047359), a Sony Young Faculty Award, Project X Innovation Award, and an Amazon Science Research Award.
References
[1] Matthew Berger, Andrea Tagliasacchi, Lee M. Seversky, Pierre Alliez, Gaël Guennebaud, Joshua A. Levine, Andrei Sharf, and Claudio T. Silva. survey of surface reconstruction from point clouds. Comput. Graph. Forum, 36(1))301-32,, 2017.
[2] Danny Driess, Jung-Su Ha, Mare Toussaint, and Russ Tedrake. Learning models as functionals of higned-disance fields for manipulation planning. In 5th Annual Conference Robot Learning. 2021.
[3] Pengfei Li, Yongliang Shi, Tianyu Liu, Hao Zhao, Guyue Zhou, and Ya-Qin Zhang. Semi- supervised implicit scene completion from sparse lidar, 2021. 2,3
[4] Roy Or-EI, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Kemelmacher- Shlizerman. Stylesdf: High-resolution 3d-consistent image and geometry generation. arXiv e-prints, pages arXiv-2112, 2021.
10

---

[5] Daniel Maturana and Sebastian Scherer. Voxnet: A 3d convolutional neural network for real- time object recognition. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 922-928, 2015.
[6] Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox. Octree generating networks: Efficient convolutional architectures for high-resolution 3d outputs. In Proceedings of the IEEE international conference on computer vision, pages 2088-2000 2096, 2017.
[7] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 15651-15663. Curran Associates, Inc., 2020.
[8] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017. 1,
[9] Charles Ruizhongtai Qi. Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
[10] Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan Russell, and Mathieu Aubry. AtlasNet: Papier-Mäché Approach Learning 3D Surface Generation. In Proceedings IEEE Conf. on Computer Vision and Puttern Recognition (CVPR), 2018. 1
[11] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 165-174, 2019. 1, 6, 8,9
[12] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings IEEE Conf.
on Computer Vision and Pattern Recognition (CVPR), 2019. 1,
[13] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020. 1,2
[14] Thomas Davies, Derek Nowrouzezahrai, and Alec Jacobson. On the effectiveness of weight- encoded neural implicit 3d shapes. arXiv preprint WwXix:2009.09808 2020.
[15] Vincent Sitzmann, Eric R. Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein. Metasdf: Meta-learning signed distance functions. In Proc. NeurIPS, 2020. 1, 3, 6, 8,9
[16] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolutional occupancy networks, In European Conference on Computer Vision, pages 523-540. Springer, 2020. 1, 2, 6, 10
[17] David B Lindell, Dave Van Veen, Jeong Joon Park, and Gordon Wetzstein. Bacon: lamd-limited coordinate networks for multiscale scene representation. arXiv preprint S.tiv:2112.04644 2021. 2, 8
[18] Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1358-11367, 2021. 1,
[19] Baorui Ma, Zhizhong Han, Yu-Shen Liu, and Matthias Zwicker. Neural-pull Learning signed distance function from point clouds by learning to pull space onto surface. Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference Machine Learning. volume 139 of Proceedings of Machine Learning Research, pages 7246-7257. PMLR, 18-24 Jul 2021. 1, 2, 5, 6,
11

---

[20] Matan Atzmon and Yaron Lipman. Sal: Sign agnostic learning of shapes from raw data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2020. 2,2,6, 6, 8
[21] Matan Atzmon and Yaron Lipman. Sald: Sign agnostic learning with derivatives. arXiv preprint 1xXi::2006005400 2020. 1,2
[22] Timo Hackel, N. Savinov, L. Ladicky, Jan D. Wegner, K. Schindler, and M. Pollefeys. SEMAN- TIC3D.NET; A new large-scale point cloud classification benchmark. In ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, volume IV-1-W1, pages 91-98, 2017.
[23] Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. In 2015 international conference on advanced robotics (ICAR), pages 510-517. IEEE, 2015. 1,
2, 9.9
[24] Mikaela Angelina Uy Quang-Hieu Pham, Binh-Son Hua, Duc Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: new benchmark dataset and classification model on real-world data. In International Conference on Computer Vision (ICCV), 2019.
[25] Cathrin Elich, Martin R. Oswald, Marc Pollefeys, and Joerg Stueckler. Semi-supervised learning of multi-object 3d scene representations, 2021. 2,3
[26] Atul Kanaujia, Cristian Sminchisescu, and Dimitris Metaxas. Semi-supervised hierarchical models for 3d human pose reconstruction. In 2007 IEEE Conference on Computer Vision and Pattern Recognition, pages 1-8, 2007. 3,3
[27] Jingtan Piao, Chen Qian, and Hongsheng Li. Semi-supervised monocular 3d face reconstruction with end-to-end prepe-eeeee domain transfer. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9397-9406, 2019. 2,3
[28] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-7473, 2020. 2,
[29] Clemens Eppner. Arsalan Mousavian, and Dieter Fox. ACRONYM: A large-scale grasp dataset based on simulation. In Under Review at ICRA 2021, 2020. 6, 9
[30] Yongqin Xian, H. Christoph Lampert, Bernt Schiele, and Zeynep Akata, Zero-shot learning comprehensive evaluation of the good, the bad and the ugly. TPAMI, 2018. 2,4
[31] Ilya Chugunov, Yuxuan Zhang, Zhihao Xia, Cecilia Zhang, Jiawen Chen, and Felix Heide. The implicit values of a good hand shake: Handheld multi-frame neural depth refinement. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[32] Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovit- skiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7210-7219, 2021. 2
[33] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 255):359-366, 1989. 2
[34] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865-5874, 2021. 2
[35] Julian Chibane. Thiemo Alldieck, and Gerard Pons-Moll. Implicit functions in feature space for 3d shape reconstruction and completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6970 6981, 2020. 2
[36] Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, Thomas Funkhouser, et al. Local implicit grid representations for 3d scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6001-6010, 2020.
12

---

[37] Jun Wang, Sanjiv Kumar, and Shih-Fu Chang. Semi-supervised hashing for farge-scale search. IEEE transactions on pattern analysis and machine intelligence, 34(12):2393-244) 2012. 2
[38] Yu Wu, Yutian Lin, Xuanyi Dong, Yan Yan. Wei Bian, and Yi Yang. Progressive learning for person re-identification with one example. IEEE Transactions on Image Processing, 28(6):2872- 2881, 2019. 2
[39] Guan' an Wang. Qinghao Hu, Jian Cheng, and Zengguang Hou. supervsee generative adversarial hashing for image retrieval. In Proceedings of the European conference on computer vision (ECCV), pages 669,485,440 2018. 2, 3
[40] Yun-Chun Chen, Chao-Te Chou, and Yu-Chiang Frank Wang. Learning to learn in a semi- supervised fashion. European Conference on Computer Vision, pages 460-478. Springer, 2020. 3
[41] Xiaomeng Xin, Jinjun Wang, Ruji Xie, Sanping Zhou, Wenli Huang, and Nanning Zheng. Semi-supervised person re-identification using multi-view clustering, Pattern Recognition, 88:285-297, 2019.
[42] Xiaomeng Xin, Xindi Wu, Yuechen Wang, and Jinjun Wang. Deep self-paced learning for semi-supervised person re-identification using multi-view pelf-paced clustering. In 2019 IEEE International Conference on Image Processing (ICIP), pages 2631-2635. IEEE, 2019. 3
[43] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Carsten Stoll, and Chris- tian Theobalt. PatchNets: Patch-Based Generalizable Deep Implicit 3D Shape Representations. European Conference on Computer Vision (ECCV), 2020. 3
[44] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Local deep implicit functions for 3d shape. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4857-4866, 2020. 3
[45] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
[46] Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1 1208, 2018. 3
[47] Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain generalization using meta-regularization. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. 3,4
[48] Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Joshua B Tenenbaum, William T Freeman, and Jiajun Wu. Learning to Reconstruct Shapes From Unseen Classes. In Advances in Neural Information Processing Systems (NeurIPS), 2018. 3
[49] B. Wallace and Hariharan. Few-shot generalization for single-image 3d reconstruction via priors. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 3817-3826, Los Alamitos, CA, USA, nov 2019. IEEE Computer Society. 3
[50] Mateusz Michalkiewicz, Sarah Parisot, Stavros Tsogkas, Mahsa Baktashmotlagh, Anders Eriks- son, and Eugene Belilovsky. Few-shot single-view 3-d object reconstruction with compositional priors. In Computer Vision - ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXV, page 614-630, Berlin, Heidelberg, 2020. Springer- Verlag. 3
[51] Ali Cheraghian, Shafin Rahman, and Lars Petersson. Zero-shot learning of 3d point cloud objects. In International Conference on Machine Vision Applications (MVA), 2019. 3
[52] Charu Sharma and Manohar Kaul. Self-supervised few-shot learning on point clouds. Advances in Neural Information Processing Systems, 33:7212-7221, 2020. 3
13

---

[53] Na Zhao, Tat-Seng Chua, and Gim Hee Lee. Few-shot 3d point cloud semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
2021. 3
[54] Miles Macklin, Kenny Erleben, Matthias Müller, Nuttapong Chentanez, Stefan Jeschke, and Zach Corse. Local optimization for robust signed distance field collision. In Proc. ACM Comput. Graph. Interact. Tech. 1, 2020. 5
[55] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Compuling and Compuler-Assisted In- tervention MICCAI 2015, pages 234-241, Cham, 2015. Springer International Publishing.
[56] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song. Hao Su, et al. Shapenet: An information rich 3d model repository. arXiv preprint arXiv: 151 12 2.03022,222 2015.
[57] William Lorensen and Harvey E. Cline. Marching cubes: A high resolution 3d surface construction algorithm. In Proceedings of the 14th Annual Conference Compuler Graphics and Interactive Techniques, SIGGRAPH '87, page 163-169, New York, NY, USA, 1987. Association for Computing Machinery. 6
[58] Julien N. P. Martel, David B. Lindell, Connor Z. Lin, Eric Chan, Marco Monteiro, and Gordon Wetzstein. Acorn: Adaptive coordinate networks for neural scene representation. ACM Trans. Graph. (SIGGRAPH), 40(4), 2021. 8
[59] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pages 3354-3361. IEEE, 2012. 10
[60] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo. Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 2446-24464422 2454, 2020. 10
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Experiments (Sec. 4) and Conclusion (Sec. 5).
(c) Did you discuss any potential negative societal impacts of your work? [No] We do not believe that our work has potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] We will release code and pretrained models. Full reproduction details are in supplement.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Implementations details can be found in supplement.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [No]
14

---

(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Provided in supplement.
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [Yes] We will release code.
(d) Did you discuss whether and how consent obtained from people whose data you't 're using/curating? [No] We use datasets that are publicly available. References are provided.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] We did not use human subjects.
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "NEQ8I_5NU-4", "reviewer": "Reviewer_39TN", "summary": "The main contribution of this paper is to improve the generalization capabilities of implicit representations learning. This paper introduce an episodic training scheme where we split data into subsets with disjoint classes to learn a generalized shape representation at the first stage. The second stage allows the model to ingest large amounts of unlabeled data to diversify the shape model learned in the first stage.", "questions": "(1) In the first stage, every f epochs, this paper splits data X into two subsets with disjoint categories, and then train both data with self-supervised loss and supervised loss respectively. According to my understanding, training the network with ground-truth signed distance values supervision through the supervised loss will give more accurate shape priors. I do not understand the role played by the self-supervised loss here. Even though the paper gives a vague explanation: “episodic training on non-overlapping categories forces the model to develop generalized representations as it cannot overfit to the data.” If the self-supervised loss simply prevents overfitting to the data, there are many common training strategies in the field of machine learning  that can prevent overfitting. A more convincing proof of principle for the self-supervised loss is needed here.\n\n(2) I am puzzled by the results of Table 1, whether such a large improvement can be achieved using only unlabeled point cloud data, and what is the reason for the improvement? How is the prediction of /phi(x) guaranteed to be correct if it does not depend on the labeled point cloud. The loss function of Eq. 7 can be equally reduced if it is predicted /phi(x) to all positive or negative signs in Eq. 8. Both SAL and NeuralPull give proofs that they can predict the correct sign, and this paper is unconvincing in this part.", "limitations": "I don't foresee any potential negative societal impacts.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths：\n\n(1) This paper proposes a training scheme that can learn implicit representations from both labeled and unlabeled point cloud data, and the method produces effective experimental results.\n\n(2) Enhancing the implicit representation learning capability of the network using both labeled and unlabeled point cloud data simultaneously is a meaningful research problem, and from the experimental results, the paper also proposes a framework that can combine both kinds of data effectively.\n\nWeaknesses：\n\nEven though I am knowledgeable about works related to implicit representation learning, I have difficulty understanding the reasons why the framework proposed in the paper works outperforms existing neural SDF methods. No convincing explanation is given in the paper for this proposed framework to achieve such a large effect of enhancement.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "LBD4iquI7Ir", "reviewer": "Reviewer_84Fy", "summary": "The paper considers the task of learning to reconstruct shapes (via the SDF) from \"unlabelled\" point clouds (no mesh and no normals) using a single network that must generalize for unseen shapes (and shape classes). The network is allowed to train on both labelled data (meshes) and unlabelled data (unoriented point clouds). The proposed method trains with a standard supervised loss on labelled data, and a self supervised loss that builds upon Neural Pull [19] (but corrects it to properly penalization incorrect signs) for unlabelled data. They further train the supervised loss with the self-supervised loss on psudo-unlabelled data, claiming it greatly helps generalize, with ablations studies demonstrating this.", "questions": "Why does adding the semi-supervised training decrease the Chamfer distance on seen classes so much (0.503 to 0.351)? Shouldn't the network have been trained enough to memorise/overfit those shapes/shape classes? What do you believe the semi-supervised training is adding here (we don't need object level generalisation to perform well on seen shapes)?", "limitations": "A short discussion of limitations is given in the conclusion. No negative societal impacts are discussed, but I agree that there are no direct negative impacts.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- A data driven approach to shape reconstruction with a strong focus on generalization\n- Specifically allows to learn to generalize from a large corpus of unlabelled shapes.\n- Fixes an important weakness in the NeuralPull training objective\n- Proposes introducing unsupervised training into the the initial supervised training step that shows significant improvement in generalization\n- Shows large improvement over baseline methods for both unsupervised and supervised methods, without further test time refinement\n\nWeaknesses\n- The baselines for this task could be better. \n\t- Section 4.4 says that \"No split\" is essentially ConvOccNet but has lower CD due to a different choice of decoder, but the difference from this choice as decoder is really large. As a result one of the baselines for Section 4.1 should be ConvOccNet with the same decoder as you use.\n\t- The method is technically given access to 76 more classes via unsupervised training for it to learn to generalize on (since the other methods do not have an unsupervised learning method), while supervised methods only get 20. It would be better to try supervised training on more diverse classes (but a few in each class such that the overall number of supervised meshes to train on is the same) to better emulate a supervised training approach that is trying to generalize to other shape classes. \n\t- It would be interesting to have a baseline that does the same setup as your approach (Phase 1 and 2) but change the self-supervised loss to be the same as Neural Pull, which would show how important your change is.\n\t- For unsupervised methods (Section 4.2), it would be good to compare to IGR without normals (the autodecoder setting), which has a better unsupervised loss (Eikonal term) than SAL and doesn't have the defect that NeuralPull does. For an even stronger baseline, PHASE+FF [* 1] or DiGS [* 2] would be even better (again with the autodecoder setting).\n- No quantitative comparison to single network per shape methods (which you call Single-object methods in the supp material), e.g. IGR, SIREN. Although, as you argue in the supp material, this is a different setting, such quantitative results (which you have given a qualitative result for) would serve as a good upper bound for seeing how close your method can generalize.\n\n[* 1] https://arxiv.org/abs/2106.07689\n[* 2] https://arxiv.org/abs/2106.10811", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GC8PjZrarz6", "reviewer": "Reviewer_GFH2", "summary": "Given multiple-category labeled (with GT SDF) and unlabeled point cloud data,  this paper introduces a simple yet effective meta-learning inspired pretraining stage before the conventional semi-supervised stage in order to train a generalizable SDF function model.  Their strategy frequently randomly splits the labeled data into a self-supervsied and GT-supervised group with different categories, which creates diversity of different tasks and encourage a shared embedding network for better model initialization, and better leverage labeled multi-category data as guidance, assisted by self-supervision loss on unlabeled data.\n\n", "questions": "1. What if in stage 1, we simply first train on all the labeled data directly with GT/SSL loss, what will be the results on the next stage of semi-supervised training? The no split in Table 2 seem to be the desired one, but I am not sure whether it is followed by semi-supervised training, and have the exactly same training setting except for the splitting;\n\n2. How to decide the split ratio between labeled and unlabeled dataset, would smaller number of labeled data (like only 5 classes instead of 20/1) lead to big degrade to the methods? How is the split frequency being decided, will it be a big influence factor to the pretraining performance?\n\n3. A recent paper 'Shape As Points: A Differentiable Poisson Solver' can also reconstruct surface from complete point cloud by learning a general geometry equation solver, is it possible to also compare with this paper? Although quantitative results would be better, comparison in text is valid if the experiments become too complex.", "limitations": "The main limitations are two-fold as mentioned by the authors:\n1. It requires multi-category data with ground truth for training;\n2. It does not consider partial scans, and may fail on partial and noisy data;", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\\\n[originality]\nIt is novel to apply episodic scheme on randomly split hybrid training for SDF learning as pretraining;\nThey further introduce a self-supervised loss regularization item[Section 3.2], which is well considered for the target task;\n\n[quality & clarity] \nThe paper is written with a smooth logic flow, and clear explanation on the methods and experiments, with a lot of high-quality visualization and comparison, like Figure 5 in main paper, and figures in the supps. \n\n[significance]\nThis simple yet effective training paradigm is helpful and inspiring for a lot of 3D learning tasks, the achieved results[Figure 3, Table 1] are \nmuch better than compared baselines.\n\n**Weakness**\n1. This paper gives me an illusion of as long as you can keep randomly splitting your labeled multi-category dataset\nand train with two different losses(GT-supervised and Self-supervised), you can get a well-generalized model with good pretraining weights;\nI am surprised at this conclusion, in which the author might want to give more details like training curves, failure cases, to really show the full potential and limitation of this proposed framework, while also explain better on the intuition behind;\n\n2.  No ablation study on the discussed 'Self-supervised Loss Component' in line 163-182, we want experiments to support the authors' argument on this alternative formulation to avoid incorrect sign predictions; \n\n3. As mentioned by the author in the supp., it seems current work only supports complete point cloud, it doesn't generalize to real-world partial scans, which is the major and crucial application scenerios for semi-supervised learning/Meta-learning to demonstrate its effectiveness. Even though current work is not designed for partial scans, the author should at least show some qualitative results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DzreTeINLPh", "reviewer": "Reviewer_8obm", "summary": "This work proposes GenSDF, a method for object surface estimation, parameterized by SDF, from 3D point cloud inputs. The main distinctions of GenSDF to prior work is its ability for high quality generalization to novel classes not seen during training, and the ability to take advantage of \"unlabeled\" point cloud data (without know SDFs) as supervision during training. GenSDF achieves this by two well designed training phases, first a meta learning procedure to train a general shape prior, by sampling labelled and unlabeled subsets of the labelled training set, and a second semi-supervised step where the fully labelled training set is used in addition to an additional entirely unlabeled set. GenSDF also proposes a well designed loss for using unlabeled point clouds as supervision for an SDF prediction model.\n\nEvaluation is performed on Acronym (a subset of ShapeNet), split into 20 labeled training classes, 70 unlabeled classes for semi-supervised training, and 166 classes for testing. GenSDF significantly outperforms SDF-supervised and unsupervised . GenSDF can also directly generalize to point clouds from scans of the YCB dataset. Ablation studies directly demonstrate the benefit of the proposed two-stage training procedure.", "questions": "1. It is clear that GenSDF is better at generalization to unseen categories than prior methods. However, the shape similarity of what are supposed to be different categories in ShapeNetSem affects these claims to some extent. Can the authors please comment on how the fact that there are semantically very similar categories that may end up being split in the seen and unseen sets affects the claims that can be made about generalization ability?\n2. Regarding the point under weaknesses in significance, can the authors please provide a bit more context about how they design the evaluation task relative to potential real world applications of GenSDF? Including this in the introduction of the paper would help the reader understand the significance of what the model can do.", "limitations": "The authors have discussed some limitations of their work throughout the draft. The paper itself doesn't have high potential for negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Originality\n### Strengths\nDirect generalization to unseen categories in the domain of surface estimation from point clouds has not been investigated before, and is an important application (see Significance). This method proposes a novel and well designed two stage training procedure, along with a loss function for using unlabeled data samples. The model's design is supported by high quality ablation studies and significantly higher performance than baselines in terms of chamfer distance.\n\n## Quality\n### Strengths\n1. _Simple, elegant and effective implementation_: The high level idea of learning a general shape prior that can be used for semi supervised training with unlabeled data makes sense, and paper addresses this using a simple but effective approach of repeatedly partitioning the training set into labeled and unlabeled classes. Further, in order to train models on point clouds only, where only the distance to the nearest point in the point set is available, it's necessary to use a loss that is a proxy for SDFs. The proposed loss is a straightforward and improved proxy between nearest point unsigned distance and SDF values relative to prior methods, and includes an additional term to encourage precision closer to the 0-level set.\n2. _High performance relative to baselines_: The performance of GenSDF vs. the closest baseline in terms of chamfer distance is 0.351 vs 1.348 for seen and 0.407 vs 2.333 for unseen. This further highlights the poor generalization ability of prior methods.\n\n### Weaknesses\n1. _Evaluation to unseen classes_: The Acronym dataset is based on ShapeNetSem. While ShapeNetSem does contain 270 different categories, a lot of these are distinct but semantically similar. For example, Chair, OfficeChair, Stool, SideChair, Barstool, AccentChair are all similar chair-like objects that are distinct categories in ShapeNetSem, and Table, RoundTable, EndTable, DiningTable, CoffeeTable, AccentTable are also all similar table-like objects that are distinct categories in ShapeNetSem. If the dataset contains categories that are distinct, but very similar according to their shape, this may potentially affect to what extent claims can be made about generalization.\n2. _Further evaluation on YCB_: The paper shows two qualitative examples on YCB. It would be very helpful to see multiple instances of real world generalization capability, especially of different types of objects. The data is not re-scaled to match the distribution of the original training data. While this makes sense to showcase generalization ability, not doing this pre-processing and showing the results removes a valuable evaluation of the model.\n\n## Clarity\nThe paper is very well written and easy to follow. The visualizations of the reconstructions and the diagrams are high quality. The tables are clearly formatted.\n\n## Significance\n### Strengths\nIt is useful to develop object surface estimation methods that generalize, so that they can be used in cases where lots of ground truth meshes (necessary to derive SDF for supervision) are not available.\n\n### Weaknesses\nLack of provided context for real world applicability. There is a multitude of ways in which 3D point clouds can be obtained for which we would like to get a detailed and high quality mesh estimate. However, the characteristics, such as the amount and kinds of noise,  or number of points of the point clouds depend on the method they were obtained (e.g. multiple depth sensor readings vs. multi-view stereo). It would be very useful if the paper included some text about how the tasks designed and used for evaluation (e.g. 5K points with a specific kind of Gaussian noise) in the paper are a good proxy for the real world utility of models like GenSDF.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept