Voint Cloud: Multi-View Point Cloud Representation for 3D Understanding 
Authors: Abdullah Hamdi
Silvio Giancola
Bernard Ghanem
Venue: ICLR 2023 poster (2023)
Keywords: multi-view
point cloud
3D understanding

Abstract: Multi-view projection methods have demonstrated promising performance on 3D understanding tasks like 3D classification and segmentation. However, it remains unclear how to combine such multi-view methods with the widely available 3D point clouds. Previous methods use unlearned heuristics to combine features at the point level. To this end, we introduce the concept of the multi-view point cloud (Voint cloud), representing each 3D point as a set of features extracted from several view-points. This novel 3D Voint cloud representation combines the compactness of 3D point cloud representation with the natural view-awareness of multi-view representation. Naturally, we can equip this new representation with convolutional and pooling operations. We deploy a Voint neural network (VointNet) to learn representations in the Voint space. Our novel representation achieves state-of-the-art performance on 3D classification, shape retrieval, and robust 3D part segmentation on standard benchmarks ( ScanObjectNN, ShapeNet Core55, and ShapeNet Parts). Further analysis shows that VointNet improves the robustness to occlusion compared to other methods.

Published as a conference paper at ICLR 2023
VOINT CLOUD: MULTI-VIENS POINT CLOUD REP- RESENTATION FOR 3D UNDERSTANDING
Abdullah Hamdi
Silvio Giancola
Bernard Ghanem
King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia
fabdullah. hamdi, silvio.giancola, bernard. bernard.ghanem/ Saatnnemm@@kustt..oo. @kaust.edu....
ABSTRACT
Multi-view projection methods have demonstrated promising performance on 3D undorstanding tasks like 3D classification and segmentation. However, it remains unclear how to combine such multi-view methods with the widely available 3D point clouds. Previous methods use unlcarned hcuristics to combine features the point level. To this end, we introduce the concept of the multi-view point cloud (Voint cloud), representing each 3D point as a set of features extracted from several view-points. This novel 3D Voint cloud representation combines the compactness of 3D point cloud representation with the natural view-awareness of multi-view representation. Naturally, we can equip this new representation with convolutional and pooling operations. We deploy a Voint neural network (VointNet) to learn representations in the Voint space. Our novel representation achieves fateeof-the-artterffrme on 3D classification, shape retrieval, and robust 3D part segmentation on standard benchmarks ScaaObbeecNN,, ShapeNet Core55, and ShapeNet Parts).'
INTRODUCTION
A fundamental question in 3D computer vision and computer graphics is how to represent 3D data (Mescheder al., 2019; Qi et al., 2017a; Maturana Scherer, 2015). This question becomes particularly vital given how the success of deep lcarning in 2D computer vision has pushed for the wide adoption of deep learning in 3D vision and graphics. In fact, deep networks already achieve impressive results in 3D classification (Hamdi et al., 2021), 3D segmentation (Hu al., 2021), 3D detection (Liu et al., 2021a), 3D reconstruction (Mescheder et al., 2019), and novel view synthesis (Mildenhall vision networks either rely on dírect 3D representations, indirect 2D projection on images, or mixture of both. Direct approaches operate on 3D data commonly represented with point clouds (Qi et al., 2017a), moshes (Feng et al., 2019), or voxcls (Choy et al., 2019). In contrast, indirect approaches commonly render multiple 2D views of objects scenes (Su et al., 2015), and process each image with a traditional 2D image-
al., 2020). 3D computer
architecture. The human visual system is closer such a multi-view indirect approach for 3D understanding, as it receives streams of rendered images rather than explicit 3D data.
Tackling 3D vision tasks with indirect approaches has three main advantages: (i) mature and transferable 2D computer vision models (CNNs, Transformers, etc. ), (ii) large and diverse labelod image datascts for pre-training (e.g. ImageNet (Russakovsky et al., 2014)), and (iii) the multi-view images give context-rich features based on the viewing angle, which are different from the goometric 3D neighborhood features. Multi-view approaches achiove impressive performance in 3D shape classification and segmentation (Wei et al., 2020; Handi et al., 2021; Dai & Nießner, 2018). However, the challenge with the multi-view representation (especially for dense predictions) lies in properly aggregating the per-view features with 3D point clouds. The appropriate aggregation is necessary to obtain representative 3D point
'The code is available

---

Published as a conference paper at ICLR 2023
Multi-View Renderings
View-Features
Voint
Voint Cloud
Figure 1: 3D Voint Clouds. We propose the multi-view point cloud (Voint cloud), a novel 3D representation that is compact and naturally descriptive of view projections of 3D point cloud. Each point in the 3D cloud is tagged with Voint, which accumulates view-features for that point. Note that not all 3D points are visible from all views. The set of Voints constructs a Voint cloud.
clouds with single feature per point suitable for typical point cloud processing pipelines. Previous multi-view works rely on heuristics (e.g. average or label mode pooling) after mapping pixels to points (Kundu et al., 2020; Wang et al., 2019a), multi-view fusion with voxels (Dai & Nießner, 2018). Such setups might not be optimal for a few reasons. (i) Such heuristics may aggregate information of mislcading projections that obtained from arbitrary view-points. For example, looking at an object from the bottom and processing that view independently can carry wrong information about the object's content when combined with other views. (ii) The views lack geometric 3D information.
To this end, we propose new hybrid 3D data structure that inherits the merits of point clouds (i.e. compactness, flexibility, and 3D descriptiveness) and leverages the benefits of rich perceptual features of multi-view projections. We call this new representation multi-view view point cloud (or Voint cloud) and illustrate it in Figure 1. A Voint cloud is a set of Voints, where cach Voint set of view-dopendent fcatures (view-fcatures that correspond same point in the 3D point cloud. The cardinality of these view-features may differ from one Voint to another. In Table 1, we compare some of the widely used 3D representation and our Voint cloud representation. Voint clouds inherit the characteristies of the parent explicit 3D point clouds, which facilitates learning Voint representations for a variety of vision applications (e.g. point cloud classification and segmentation). To deploy deep learning the new Voint space, we define basic operations on Voints, such as pooling and convolution. Based on these operations, we define practical way of building Voint neural networks that we dub VointNet. VointNet takes a Voint cloud and outputs point cloud features for 3D point cloud processing. We show how learning this Voint cloud representation leads strong performance and gained robustness for the tasks of 3D classification, 3D object retrieval, and 3D part segmentation on standard benchmarks like ScanObjectNN (Uy et al., 2019), and ShapeNet (Chang et al., 2015).
the
Contributions: (i) We propose a novel multi-view 3D point cloud representation (denoted as Voint cloud), which represents each point (namely a Voint) as a set of features from different view-points. (ii) We define pooling and convolutional operations at the Voint level to construct a Voint Neural Network (VointNet) capable of learning to aggregate information from multiple views in the Voint space. (iii) Our VointNet reaches state-of- several 3D understanding tasks, including 3D shape classification, retrieval, and robust part segmentation. Further, VointNet achieves robustness improvement to occlusion and rotation.
the-artperformance
2

---

Published as a conference paper at ICLR 2023
3D Representation
EExplicttness
View-Based
Main Use
3D Expressiveness
Point Clouds
Explicit
x
3D Understanding
Medium
Multi-View Projections
Implicit
/
3D Understanding
Low
Voxels
Explicit
X 3D Understanding
X
Medium
Mesh
Explicit
3D Modeling
High
NeRFs
Implicit
Novel View Synthesis
Medium
Voint Clouds (ours)
Explicit
3D Understanding
Medium
Table 1: Comparison of Different 3D Representations. We compare some of the widely
used 3D representations to our proposed Voint cloud. Note that our Voint cloud shares the view-dependency of NeRFs (Mildenhall et al., 2020) while inheriting the merits of 3D point clouds.
2 RELATED WORK
Learning on 3D Point Clouds. 3D point clouds are widely used for 3D representation in computer vision due to their compactness, flexibility, and because they can be obtained naturally from sensors like LiDAR and RGBD cameras. PointNet (Qi et al., 2017a) paved the way as the first doep learning algorithm to operate directly on 3D point clouds. It computes point features independently and aggregates them using an order-invariant function like max-pooling. Subsequent works focused on finding neighborhoods of points to define point convolutional operations (Qi et al., 2017b; Wang et al., 2019c; Li el. al., 2018; Han et al., 2019). Several recent works combine point cloud representations with other 3D modalities like vaxels (Liu al., 2019b; You et al., 2018) or multi-view images (Jaritz al., 2019). We propose a novel Voint cloud representation for 3D shapes and investigates novel architectures that aggregate view-dependent fcatures at the 3D point level.
Multi- View Applications. The idea of using 2D images to understand the 3D world was initially proposed in 1994 by Bradski el. (Bradski & Grossberg, 1994). This intuitive multi-view approach was combined with deep learning for 3D understanding in MVCNN (Su et al., 2015). A line of works continued developing multi-view approaches for classification and retrieval by improving the aggregation of the view-features from each image view (Kanezaki et al., 2018; Esteves et al., 2019; Cohen & Welling, 2016; Wei et al., 2020; Hamdi et al,, 2021). In this work, we fuse the concept of multi-view into the 3D structure itself, such that every 3D point would have an independent set of view-fcatures according to the view-points available in the setup. Our Voints are aligned with the sampled 3D point cloud, offering a compact represent ation that allows for efficient computation and memory usage while maintaining the dieeddependent component that facilitates view-based learning for vision.
Hybrid Multi-Vien with 3D Data. On the task of 3D semantic segmentation, a smaller number of works tried to follow the multi-view approach (Dai & Nießner, 2018; Kundu et al., 2020; Wang et al., 2019a; Kalogerakis et al., 2017; Jaritz et al., 2019; Liu et al., 2021b; Lyu at al., 2020). A problem arises when combining view fcatures to represent local points/voxcle while preserving local geometric features. These methods tend to average the view-features (Kundu et al., 2020; Kalogerakis et al., 2017), propagate the labels only (Wang et al., 2019a), learn from reconstructed points in the neighborhood (Jaritz et. al., 2019), order points a single grid (Lyu et al., 2020), or combine the multi-view features with 3D voxel features (Dai & Nießfner, 2018; Hou et al., 2019). To this end, our proposed VointNet operates the Voint cloud space while preserving the compactness and 3D descriptiveness of the original point cloud. VointNet leverages the power of multi-view features with learned aggregation on the vicw-features applied to each point independently.
3 METHODOLOGY
The primary assumption in our work is that surface 3D points are spherical functions, i.e. their representations depend the viewing angles observing them. This condition contrasts with most 3D point cloud processing pipelines that assume a view-independent representation of 3D point clouds. The full pipeline is illustrated in Figure 2.
3

---

Published as a conference paper at ICLR 2023
View-Points (Mx2)
Pixel-to-Paint Mapping B (MxHxW)
Visiblitt (N x M)
Point Cloud
Renderer
Mutti-View
2D Backbone
Multi-View
Unprojection
Voint Cloud
VointNet
Point Features (Nxd)
Renderings (MXHXWX3)
Features
(Nx3)
(MXHXWxd)
(N (NXMxd)
Figure 2: Learning from Voint Clouds. To construct 3D Voint cloud &, renderer R renders the point cloud from view-points and image features are extracted from the generated images via a 2D backbone C. The image features are then unprojected to the Voint cloud by dg and passed to VointNet F. learn both and F, 3D loss on the output points used with an optional auxiliary 2D loss on C.
3.1 3D VOINT CLOUD
From Point Clouds to Voint Clouds. 3D point cloud is a compact 3D representation composed of sampled points on the surface of a 3D object a scene and be obtained by different sensors like LiDAR (Chen et al., 2017) or as result of reconstruction (Okutomi & Kanade, 1993). Formally, we define the coordinate function for the surface gx(x) IR3 RR as the Sign Distance Function (SDF) in the continuous Euclidean space (Park et al., 2019; Mescheder et al., 2019). The 3D iso-surface is then defined as the sct of all points that satisfy the condition (x) = O. We define surface 3D point cloud X' E Rx3 as set of N 3D points, where each point x; E R3 represented by its 3D coordinates (x), Mis =.) and satisfics the iso-surface condition follows: X' fxx E R (x,)) 0} 0} work, aim to fuse the diewdeeped to 3D point. Inspired by NeRFs (Mildenhall et al., 2020), we assume that surface points also depend on the view direction from which they are being observed. Specifically, there exists continuous implicit spherical function g(x,u) u): R >> Rd that defines the features of each point x depending on the view-point direction Given of M view-dependent features of size d for the sphere centered at point x as follows.
1-1- In this
of M view-point directions U E M Voint & E Rxx is
* = (x,, u,) E Rd
X, E EEXI M
(1)
The Voint cloud X E NxMx = 1&,,,,1 is the set of all N Voints X; corresponding to the parent point cloud X. Note that we typically do not have access to the underlying implicit function g and we approximate it with the following three steps.
1- Multi- View Projection. As mentioned earlier, a Voint combines multiple riew-features of the same 3D point. These view- features come from multi-view projection of the points by a point cloud renderer R. IR Nx3 + IR Mx H x x3 that renders the point cloud X' from multiple view-points l into M images of size H x W 3. In addition to projecting the point cloud into the image space, R defines the index mapping B E {0,.., NYA, between each pixel to the N points and background it renders. Also, R outputs the visibility binary matrix V {0, 13 for each point from each view. Since not all points appear in all the views due to pixel discretization, the visibility score Vi,, defines if the Voint x; is visible in the view uj. The matrix B is crucial for unprojection, while V is needed for defining mcaningful operations on Voints.
2- Multi- Feature Extraction. The rendered images are processed by function C: MxH + MRMMXHWW xd that extracts image features, as shown in Figure C is the identity function, all the view-features would typically the RGB value of the corresponding point. However, the function can be a 2D network dedicated to the downstream task and can extract useful global and local features about each view.
3- Multi- View Unprojection. We propose a module ΦB : XMMHxWXx + Nxxx that unprojects the 2D features from each pixel to be 3D view-features at the corresponding voint. Using the mapping B created by the renderer, ΦB forms the Voint cloud features X.
4

---

Published as a conference paper at ICLR 2023
To summarize, the output Voint cloud is described by Eq (1), where (xx, uy) @e (C (X,,), and the features are only defined for view j of Voint x if ViJ
3.2 OPERATIONS ON 3D VOINT CLOUDS
We show in the Appendix that a functional form of max- max-pooled individual view-features of a set of angles can approximate any function in the spherical coordinates. We provide theorem that extends PointNet's theorem of point cloud functional composition (Qi et al., 2017a) and its Universal Approximation to spherical functions underlying Voints. Next, we define sct of operations on Voints as building blocks for Voint neural networks (VointNet).
VoxntMax. We define VointMax as max- max-pooling on the visible view-features along the views dimension of the voint &. For all i E 1, 2, ..., N and j E 2,,,... M..,,,,,,, M,
VointMax(XE) J Ri s.t. Vis= =1
(2)
Voint Conv. We define the convolution operation by R + Nx any learnable function that operates on the Voint space with shared weights on all the Voints and has the view-features input size d and outputs view-features of size d' and consists of ly layers. A simple example of this VointConv operation is the shared MLP applied only on the visible fiew-featuree We provide further details for such operations in Section 4.2, which result in different non-exhaustive variants of Vet.
3.3 LEARNING ON 3D VOINT CLOUDS
Netn The goal of the VointNet model obtain multi-view point cloud features that can be subsequently used by any point cloud processing pipeline. The VointNet modulc F: NxAxx - Nxd is defined as follows.
F(x) hp (VointMax (hv (2)))
(3)
where hp is any point convolutional operation (e.g. shared MLP or EdgeConv). VointNet F transforms the individual view-features using the learned VointConv by before VointMax is applicd on the view-fcatures to obtain point fcatures.
VointNet Pipeline for 3D Point Cloud Processing. The full pipeline is described in Figure 2. The loss for this pipeline can be described as follows:
arg min
Σ
v.).
(4)
Oc: O
where L a Sross-Enttoss (CE) loss defined on all the training points X, and {y:) N defines the labels of these points. The other components (R, LB,,, C) all defined before. The weights to be jointly learned are those of the 2D backbone (0c) and those of the VointNet (0p) using the same 3D loss. An auxiliary 2D loss on Oc can be optionally added for supervision at the image level. For classification, the entire object be treated as single Voint, and the global features of each view would be the view-feat ures of that Voint. We analyze different setups in detail in Section 6.
i=1
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datascts. We benchmark VointNet on the challenging and realistic ScanObjectNN dataset for 3D point cloud classification (Uy et al., 2019). The dataset has three variants, includes background and occlusion, and has 15 categories and 2,902 point clouds. For the shape retrieval task, we benchmark on ShapeNet Core55 as a subset of ShapeNet (Chang et al., 2015). The dataset consists of 51,162 3D mesh objects labeled with 55 object classes. We follow the MVTN's setup (Hamdi et al., 2021) in sampling 5,000 points from each mesh object obtain point cloud. On the other hand, for the task of shape part segmentation,
5

---

Published as a conference paper at ICLR 2023
Classification Overall Accuracy
Method
Data Type OBJ BG OBJ LONOY
Hardest
PointNet (Qi et al., 2017a)
Points
73.3
79.2
68.0
SpiderCNN (Xu et al. 2018)
Points
77.1
79.5
73.7
PointNet ++ ++ et al., 2017b)
Points
82.3
84.3
77.9
PointCNN (Li et al., 2018)
Points
86.1
85.5
78.5
DGCNN (Wang et al. 2019c)
Points
82.8
86.2
78.1
SimpleView (Goyal et al., 2021)
M-View
79.5
BGA-DGCNN (Uy et al., 2019)
Points
79.7
BGA-PN++ (Uy et al., 2019)
Points
80.2
MVTN (Hamdi al., 2021)
M-View
92.6
92.3
82.8
VointNet (ours)
Voints
93.7
94.0
85.4
Table 2: 3D Point Cloud Classification on ScanObjectNNN We report the accuracy of VointNet in 3D point cloud classification on three different variants of ScanObjectNN (Uy al., 2019). Bold denotes the best result its setup. Note that the Hardest variant includes rotated and translated objects, which highlights the benefits of Voints on challenging scenarios.
we test on ShapeNet Parts (Yi ct al., 2016), a subset ShapeNet (Chang al., 2015) that consists of 16,872 point cloud objects from 16 categories and 50 parts. For occlusion robustness, we follow MVTN (Hamdi et al., 2021) and test on ModelNet40 (Wu el al., 2015), which is composed of 40 classes and 12,311 3D objects.
Metrics. For 3D point cloud classification, we report the overall accuracy, while shape retrieval is evaluated using mean Average Precision (mAP) over test queries (Hamdi et al., 2021). 3D semantic segmentation is evaluated using mean Intersection over Union (mloU) points. For part segmentation, we report Instanco-averaged mloU (Ins. mloU).
Baselines, We include PointNet (Qi et al., 2017a), Point.Net++ (Qi et al., 2017b), DGCNN (Wang et al., 2019c), baselines that use point clouds. We also compare against multi-view classification approaches like MVCNN (Su et al., 2015), SimpleView (Goyal et al., 2021), and MVTN (Hamdi et al., 2021) as baselines for classification and retrieval and adopt some of the multi-view segmentation baselines (e.g. Label Fusion (Wang et al., 2019a) and Mean Fusion (Kundu et al., 2020)) for part segmentation.
4.2 VOINTNET VARIANTS
VointNet in Eq (3) relies on the VointConv operation by as the basic building block. Here, we briefly describe three examples of by operations VointNet uses.
Shared Multi-Layer Perceptron (MLP). It is the most basic VointConv formulation.
For layer 1, the features of Voint i at view j are updated to layer 1+1 as: h = p (h W,), where P is the shared MLP with weights W; followed by normalization and a nonlinear function (e.g. ReLU). This operation is applied on all Voints independently and only involves the visible views-fcatures for cach Voint. This formulation extends the shared MLP formulation for PointNet (Qi al., 2017a) to work on Voints' view-features.
Graph Convolution (GCN). We define fully connected graph for each Voinl by creating a virtual center node connected all the view-features aggregate their information (similar to "cls" token in ViT (Dosovitskiy al., 2021)). Then, the graph convolution can be defined as the shared MLP (as described above) but on the edge features between all view features, followed by max pool on the graph neighbors. An additional shared MLP is used before the final output.
Graph Attention (GAT). A graph attention operation can be defined just like the GCN operation above but with learned attention weights on the graph neighbor's 's features before averaging them. A shared MT.P computes these weights.

---

Published as a conference paper at ICLR 2023
Results
MVCNN (Su al., 2015)
ViewGCN (Kanezaki et al., 2018) (Wei et al., 2020)
RotNet
MVTN (Hamdi et al., 2021)
VointNet (ours)
Shape
Retr. mAP
73.5
77.2
78.4
82.9
83.3
Table 3: 3D Shape Retrieval. We report 3D shape retrieval mAP ShapeNet Core55 (Chang al., 2015; Sfikas al,, 2017). VointNet achieves state-of-the the-art results on this benchmark.
Part Segmentation (Rotated)
Method
Data Type (Unrotated)
PointNet (Qi
al., 2017a)
Points
80.1
36.6 10.2
DGCNN (Wang et al., 2019c)
Points
80.1
37.1 10.2
CurveNet (Xiang et al., 2021)
Points
84.9
32.3 #0.0
Label Fuse (Wang et al., 2019a)
M-View
80.0
61.4 10.2
Mean Fuse (Kundu et al., 2020)
M- M-View
77.5
62.0 +0.2
VointNet (ours)
Voints
81.2
62.4 =0.2
Table 4: Robust 3D Part Segmentation on ShapeNet Parts. We compare the Inst. mloU of VointNet against other methods in 3D segmentation on ShapeNet Parts (Yi et al., 2016). At test time, we randomly rotate the objects and report the results over ten runs. Note how Voint performance largely exceeds the point baselines in the realistic rotated scenarios, while exceeding multi-view baselines on the unrotated benchmark. All the results reproduced in our setup.
4.3 IMPLEMENTATION DETAILS
Rendering and Unprojection. We choose the differentiable point cloud renderer R from Pytorch3D (Ravi et al., 2020) in our pipeline for its speed and compatibility with Pytorch libraries (Paszke et al., 2017). We render point clouds on multi-viww images with size 224 x 224 x 3. We color the points by their normals' values or keep them white if the normals are not available. Following similar procedure to (Wei et al., 2020; Hamdi et al., 2021), the view-points sctup is randomized during training (using M = views) and fixed to spherical views in testing (using M = 12 views).
A Architectures. For the 2D backbone C, we use ViT-B (Dosovilskiy et al., 2021) (with pretrained weights from TIMM library (Wightman, 2019)) for classification and DeepLabV3 (Chen et al., 2018) for segmentation. We use the 3D CE loss on the 3D point cloud output and the 2D CE loss when the loss is defined on the pixels. The feature dimension of the Voint.et architectures is = 64, and the depth lv = 4 layers in fv. The main results based VointNet (MLP), unless otherwise specified as in Section 6, where we study in details the effect of VointConv by and C.
Training Setup. We train our pipeline in two stages, where start by training the 2D backbone on the 2D projected labels of the points, then train the entire pipeline end-to-end while focusing the training on the VointNet part. We the AdamW optimizer (Loshchilov & Hutter, 2017) with an initial learning rate of 0.0005 and a step learning rate schedule of 33.3% every 12 epochs for 40 epochs. The pipeline trained with one NVIDIA Tesla V100 GPU, We do not use any data augmentation. More details about the training setup (loss and rendering), VointNet, and the 2D backbone architectures can be found in the Appendix.
5 RESULTS
The main test results of our Voint formulations are summarized in Tables 2.3, 4, and 5. We achieve taate-ofthh-aattpeffrmaanc in the task of 3D classification, retrieval, and robust 3D part segmentation. More importantly, under the realistic rotated setups of ScanObjectNN and ShapeNet Parts, we improve over 7.2 % Acc. and 25% mloU respectively compared to point baselines Qi et al. (2017a); Wang et al. (2019c). Following common practice Hamdi et al. (2021), we report the best results out of four runs in benchmark tables, but detailed results are provided in the Appendix.

---

Published as a conference paper at ICLR 2023
Ground Truth
VointNet (ours)
Mean Fuse (Kundu et al., 2020)
Figure 3: Qualitative Comparison for Part Segmentation. We compare our VointNet 3D segmentation predictions to Mean Fuse (Kundu et al., 2020) that is using the same trained 2D backbone. Note how VointNet distinguishes detailed parts (e.g. the car window frame).
5.1 3D SHAPE CLASSIFICATION
Table reports the classification accuracy on the 3D point cloud classification task on ScanObjecdNN Uy et al. (2019). It benchmarks VointNet against, other recent and strong baselines Hamdi et al. (2021); Goyal et al. (2021); ]; Hamdi et al. (2021). Voint demonstrates on all the variants, including the challenging Hardest (PB T50 RS) variant that includes challenging scenarios of rotated and translated objects. The increase in performance (+2.6%) is significant on this variant, which highlights the benefits of Voints on challenging scenarios, with further affirming results in Section 5.4. We follow exactly the same procedure as MVTN Hamdi et al. (2021).
5.2 3D SHAPE RETRIEVAL
Table 3 benchmarks the 3D shape retrieval mAP on ShapeNet Core55 Chang et al. (2015). VointNet achieves
ShapeNet Core55. Baseline results
reported from Hamdi et al. (2021).
5.3 ROBUST 3D PART SEGMENTATION
Table 4 reports the Instance-averaged segmentation mloU VointNet compared with other methods on ShapeNet Parts Yi et al. (2016). Two variants of the benchmark are reported : unrotated normalized setup, and the rolated realistic setup. For the rolated setup, we follow the previous 3D literature Liu et al. (2019a): Hamdi et al. (2021; 2020) by testing the robustness of trained models by perturbing the shapes in ShapeNet Parts with random rotations at test time (ten runs) and report the averages in Table 4. Note VointNet's improvement over Mean Fuse Kundu et al. (2020) and Label Fuse Wang et al. (2019a) on unrotated sctup despite that both basclines use the same trained 2D backbone as VointNot. Also, for rotated setups, point methods don't 't work as well. All the results in Table 4 are reproduced by our code in the same setup (see the code attached in supplementary material). Figure 3 shows qualitative 3D segmentation results for Net and Mean Fuse Kundu el. al. (2020) as compared to the ground truth.
5.4 OCCLUSION ROBUSTNESS
One of the aspects of the robustness of 3D classification models that have been recently studied is their robustness to occlusion, detailed in MVTN Hamdi et al. (2021). These simulated occlusions are introduced at test time, and the average test accuracy is reported each cropping ratio. We benchmark our VointNet against recent baselines in Table 5. PointNet Qi et al. (2017a) and DGCNN Wang et al. (2019c) are used as point- based baselines, and MVTN Hamdi et al. (2021) as multi-view baseline.

---

Published as a conference paper at ICLR 2023
Occlusion Ratio
Method
Data Type
0.1
0.2
0.3
0.5
PointNet (Qi et al., 2017a)
Points
89.1
88.2
86.1
81.6
53.5
DGCNN (Wang al., 2019c)
Points
92.1
77.1
74.5
71.2
30.1
PCT (Guo et al., 2021)
Points
93.3
92.6
91.1
88.2
61.9
MVTN (Hamdi al., 2021)
M-View
93.8
90.3
89.9
88.3
67.1
VointNet (ours)
Voints
92.8
91.6
91.2
89.1
66.1
Table 5: Occlusion Robustness for 3D Classification. We report the test accuracy ModelNee140 (Wu et al., 2015) for different occlusion ralios of the data robustness of different 3D methods.
occlusion
80
79
mlou
78
Mean
Label
VointNet (GAT)
75
4
9
10
11
12
Number of Views (M)
Figure 4: Effect of the Number of Views. We plot Ins. mIoU of 3D segmentation the mumber of views (M) used in inference ShapeNet Parts. Note VointNet's consistent improvement over Mean Fuse (Kundu et al., 2020) and Label Fuse (Wang et al., 2019a). Both baselines use the same trained 2D backbone as VointNet and are tested on the same unrotated setup.
ANALYSIS AND INSIGHTS
Number of Views. We study the effect of the number of views M on the performance of 3D part segmentation using multiple views. We compare Mean Fuse (Kundu et al., 2020) and Label Fuse (Wang et al., 2019a) to our VointNet when all of them have the same trained 2D) backbone. The views are randomly picked, and the experiments are repeated four times. Ins. mloU with confidence intervals are shown Figure 4. We observe a consistent improvement with Vet over the other two baselines across different numbers of views.
2D Backbone
VointConv MLP GCN GAT
Results Inst. mIoU
FCN DeepLabV3
78.8 0.2
77.6 + 0.2
77.1 0.2
/
80.6 + 0.1
77.2 0.4
80.4 + 0.2
Table Ablation Study 3D Segmentation. We ablate different components VointNet (2D backbone and VointConv choice) and report Ins. mloU performance on ShapeNet Parts.
Choice Backbones. We ablate the choice of the 2D backbone and the Voint Conv operation used in VointNet and report the segmentation Ins. mloU results in Table 6. Note how the 2D backbone greatly affects performance, while the VointConv operation type does not. This ablation highlights the importance of the 2D backbone in VointNet pipeline and motivates the use of the simplest variant of VointNet (MLP). We provide a detailed study of more factors well compute and memory costs in the Appendix.

---

Published as a conference paper at ICLR 2023
7 LIMITATIONS AND ACKNOWLEDGMENTS
One aspect limiting the performance of Voints is how well-trained the 2D backbone is for the downstream 3D task. In most cases, the 2D backbone must be pretrained with enough data to learn meaningful information for VointNet. Another aspect that limits the capability of the Voint cloud is how to properly select the view-points for segmentation. Addressing these limitations is an important direction for future work. Also, exlending Voint learning on more 3D tasks like 3D scene segmentation and 3D object detection is left. for future work.
Acknowledgments. This work was supported by the King Abdullah University of Science and Technology (KAUST) Office of Sponsored Research through the Visual Computing Center (VCC) funding and the SDAIA-KAUST Center of Excellence in Data Science and Artificial Intelligence (SDAIA-KAUST AI)
REFERENCES
Gary Bradski and Stephen Grossberg. Recognition of 3-d objects from multiple 2-d views by self-organizing neural architecture. In From Statistics to Neural Networks, pp. 349-375. Springer, 1994.
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Rich 3D Model Repository. Technical Re- port MmXiv:1512.03011 GR), Stanford University - Princeton University - Toyota Technological Institute at Chicago, 2015.
Liang- Ching Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-dccoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 801 818, 2018.
Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognation, pp. 1907-1915, 2017.
Christopher Choy, Jun Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognation, pp. 3075-3084, 2019.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International 2016.
conference on machine learning, pp.
Angela Dai and Matthias Nicßner. 3dmv: Joint 3d-multi-viww prediction for 3d semantic scene segmentation. In Proceedings of the Europeon Conference on Computer Vision (ECCV), pp. 452 468, 2018.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostala Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Usækoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
Carlos Esteves, Yinshuang Xu, Christine Alen-Bllachhett, and Kostas Daniilidis. Equivariant multi-view networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1568-1577, 2019.
Yutong Feng, Yifan Feng Haoxuan You, Xibin Zhao, and Yue Gao. Meshnet: Mesh neural network for 3d shape representation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 8279 8286, 2019.
Ankit Goyal, Hei Law, Bowei Liu, Alejandro Newell, and Jia Deng. Revisiting point cloud shape classification with a simple and effective baseline. In ICML, 2021.
Meng-Haa Hao Guo, Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pet: Point cloud transformer. Computational Visual Media, 7(2):187-119)) 2021.
10

---

Published as a conference paper at ICLR 2023
Abdullah Hamdi, Sara Rojas, Ali Thabet, and Bernard Ghanem. Advpe: Transferable
adversarial perturbations on 3d point clouds. In Computer Vision - ECCV 2020, pp. 241-251 Cham, 2020. Springer International Publishing. ISBN 978-3-030-58610-2.
Abdullah Hamdi, Silvio Giancola, and Bernard Ghanem. Mvtn: Multi-view transforma- tion network for 3d shape recognition. In Proceedings of the IEEE/CVF International Conference on Compuler Vision (ICCV), pp. 1 1-11, October 2021.
Zhizhong Han, Xiyang Wang, Yu-Shen Liu, and Matthias Zwicker. Multi-angle point cloud-vae: Unsupervised feature learning for 3d point clouds from multiple angles by joint self-reconstructionn and half-to-half prediction. In 2019 IEEE/CVP International Conference on Computer Vision (ICCV), pp. 10441- 10450. TEEE, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.0360 1.03855, 2015. URL http: /atxiv///xvvvv/// org/ abss//12. 03385.
Ji Hou, Angela Dai, and Matthias NicBner. 3d-sis: 3d semantic instance segmentation of rgb-d scans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4421 4430, 2019.
Wenbo Hu, Hengshuang Zhao, Li Jiang, Jiaya Jia, and Ticn-Tsii Wong. Bidirectional projec- tion network for cross dimension scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recognition, pp. 14373 14382, 2021
Maximilian Jaritz, Jiayuan Gu, and Hao Su. Multi-view pointnet for 3d scene understanding. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 0 0, 2019.
Evangelos Kalogerakis, Melinos Averkiou, Subhransu Maji, and Siddhartha Chaudhuri. 3d shape segmentation with projective convolutional networks. In proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3779 3788, 2017.
Asako Kanezaki, Yasuyuki Matsushita, and Yoshifumi Nishida. Rotationnet: Joint object categorization and pose estimation using multiviews from unsupervised viewpoints. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5010-5019, 2018.
Abhijit Kundu, Xiaoqi Yin, Alireza Fathi, David Ross, Brian Brewington, Thomas Funkhouser, and Caroline Pantofaru. Virtual multi-view fusion for 3d semantic seg- mentation. In European Conference Computer Vision (ECCV), pp. 518 535. Springer, 2020.
Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointenn: Convolution on x-transformed points. In Advances neural information processing systems (NIPS), pp. 820-830, 2018.
Yongcheng Liu, Bin Fan, Shiming Xiang, and Chunhong Pan. Relation-shape convolutional noural network for point cloud analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8895-8904 2019a.
Ze Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong. Group-free 3d object detection via transformers. arXiv preprint 2rXii:2104.0060
2021a.
Zhengzhe Liu, Xiaojuan Qi, and Chi-Wing Fu. 3d-to-24 2d distillation for indoor scene parsing. In Proceedings of the IEFE/CV /CVF Conference on Computer Vision and Pattern Recognition, pp. 4464-4474, 2021b.
Zhijian Liu, Haotian Tang, Yujun Lin, and Song Han. Point-voxel cnn for efficient 3d deep learning. Tn Advances in Neural Information Processing Systems, pp. 065-975. 2019b.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3444 3440, 2015.
11

---

Published as a conference paper at ICLR 2023
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 71110110,, 2017.
Yecheng Lyu, Xinming Huang, and Ziming Zhang. Learning to segment 3d point clouds in 2d image space. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition, pp. 12255 122644, 2020.
Ricardo Martin-Brualle Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf the wild: Neural radiance fields for uncon- strained photo collections. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7210 7219, 2021.
Daniel Maturana and Sebastian Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In 2015 IEEE/RS International Conference on Intelligent Robots and Systems (IROS), pp. 922 928. IEEE, 2015.
Leonard McMillan and Gary Bishop. Plenoptic modeling: An image-bascd rendering system. In Proceedings of the 22nd annuual conference on Computer graphics and interactive lechniques, pp. 39 46, 1995.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEFE/CV /CV Conference on Compuler Vision and Pallern Recognition, pp. 1460-4470, 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pp. 405-421. Springer, 2020.
Masatoshi Okutomi and Takeo Kanade. A maseiiee baseline stereo. IEEE Transactions on pattern analysis and machine intelligence, 15(4):353 -33,333, 1993.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 165-174, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS- W. 2017.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10318- 2021.
Charles Qi, Hao Su, Kaichun Mo, and Leonidas J Guíbas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the TEFE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 652 660, 2017a
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet+ Deep hicrarchical fcature learning on point sets in a metric space. In Advances in neural information processing systems (NIPS), pp. 5099 5108, 2017b.
Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yon Lo, Justin John- son, and Georgia Gkioxari. Accelcrating 3d deep learning with pytorch3d. arXiv:2007. 085001, 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Sathoesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. CoRR, URL http://arxiv. wwwps/swww.rrrab//////es//
2014.
0555
12

---

Published as a conference paper at ICLR 2023
Konstantinos Sfikas, Theoharis Theoharis, and Ioannis Pratikakis. Exploiting the PANORAMA Representation for Convolutional Neural Network Classification and Re- trieval. In loannis Pratikakis, Florent Dupont, and Maks Ovsjanikov (eds.), Eurogrophics Workshop on 3D Object Retrieval, pp. 1-7. The Eurographics Association, 2017. ISBN 978-3--3866--03-- doi: 10.2312/3dor 20177145.
Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. Multi-view couvolutional neural networks for 3d shape recognition. In Proceedings of the IEEE internalional conference compater vision, pp. 945 953, 2015.
Hugues Thomas, Charles R Qi, Jean-Emmanuul Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 6411-6420, 2019.
Mikaela Angelina Uy, Quang- Hieu Pham, Binh-Son Hua, Due Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model real-world data. In International Conference on Computer Vision (ICCV), 2019.
Brian H Wang, Wei-Lun Chao, Yan Wang, Bharath Hariharan, Kilian Q Weinberger, and Mark Campbell. Ldls: 3-d object segmentation through label diffusion from 2-d images. JEEE Robolics and Aulomalion Lellers, 4(3):2902-200) 2209, 2019a.
He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Lconidas Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recogrittio,, pp. 2642 2651, 2019b.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon, Dynamic graph enn for learning on point clouds. ACM Transactions on Graphics (TOG), 2019c.
Xin Wei, Ruixuan Yu, and Jian Sun. View-gen: View-based graph convolutional network for 3d shape analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1850-1859, 2020.
Ross Wightman.
Pytorch image models.
http:////gttuub..o///gggggg
pytorch- aecch--mmagess image-models. 2019.
Zhirong Wu, S. Song, A. Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and J. Xiao. 3d shapenets: deep representation for volumetric shapes. In 2015 IEFE Conference on Computer Vision and Pattern Recognation (CVPR), pp. 1912-1920, 2015.
Tiange Xiang, Chaoyi Zhang, Yang Song, Jianhui Yu, and Weidong Cai. Walk in the cloud: Learning curves for point clouds shape analysis. In Proceedings of the IEEE/CVF International Conference on Compuler Vision (ICCV), pp. 915 924, October 2021.
Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, and Yu Qiao. Spiderenn: Deep learning point sets with parameterized convolutional filters. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 87 102, 2018
Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections. ACM Transactions on Graphics (ToG), 35(6):1 12, 2016.
Haoxuan You, Yifan Feng, Rongrong Ji, and Yue Gao. Pynet: A joint convolutional network of point cloud and multi-view for 3d shape recognition. In Proceedings of the 26th ACM international conference on Multimediaa, pp. 1310-1318, 2018.
Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng. and Angjoo Kanazawa. PlenOctroes for real-time rendering of neural radiance fields. In ICCV, 2021.
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip Torr, and Vladlen Koltun. Point transformer. ar Xie preprint 2rXix:2012.091 2020.
13

---

Published as a conference paper at ICLR 2023
APPENDIX
A DETAILED FORMULATIONS
A.1 TOY EXAMPLE
In the toy 2D example in Figure 5, the center point (represented by circular function g) is viewed from various view-points Uj that are agnostic to the underlying function itself. In many applications, is desired have a single fcature representing each point in the point cloud. When the projected values of from these 4j view-points are aggregated together pool) to get a constant representation of that point, the underlying properties of g are lost. We build our Voint representation to keep the structure of intact by laking the full set ((g,,(uu)))
(e.g. by max/meat
in learning the aggregations.
A.2 FUNCTIONAL FORM OF VOINTNET
We can look at simplified setup to decide on the functional form of the deep neural network that operates in the Voint space. In this simplified sctup, we consider a 2D example (instcad of 3D Voints) and assume that circular function describes & point at the center. The center point will assume its value according to the angle u. The following Theorem proves that for any continuous set function f that operates any set of M angles {uI,... uM}, }. there exists an equivalent composite function consisting of transformed max-pooled individual view-features. This composition is the functional form we describe later for Voint neural networks
Theorem 1 Suppose f S + R is continuoous set function operaling on angles set S == u E [0] The conlinuuity of f is based on the Hausdorff distance da belween two sets of angles, where (S, max,çes' min,,es dA (u,, uf), and da is the smallest positive angle between two angles d u') min(ju 2T 2 - |u - z'l). Then, for every e > O, and u = [uI,.... um] C, there erisls a continuous function h and symmetric function g(ui, uM) = MAX, such thal:
|SCO - (MAX (h (u.)..((m))) <c
(5)
where is a continuous function, and MAX is an dement-wise vector mar operator.
Proof. By the continuity of f, we take & that [f(()) f(44) for any u.U S if du U') < d Define K =2 /o,, which split [0,2T] into K intervals evenly and define an auxiliary function that maps an angle to the beginning of the interval it lies in:
[Ku]
(u) K
Let ú - o()) : E U, then
(s()) 1(1) <E
(6)
Let hx(u) = e d(uul( kl) be a soft indicator function where d duu,
k-1
K
R
min (d, (u, xx da (u, )) is the distance between angle to interval h(u) = [u); hx then h R2 R
k-1
Let
K
Let Øj (u,,..., UM) = max (hy (ui)...., by (uN)), indicating the occupancy of the j-th interval by angles in 4. Let q = [qn...;; :4k), then q: [0, 2m MM - {0, 1) is symmetric function, indicating the occupancy of each interval by angles in U.
Define ç: (0, 1] - S as ((q) {x====------ qu 2 1} which maps the occupancy vector to a set which contains the left end of cach angle interval. It is straightforward to show:
(q((2)) = ú
(7)
Let r R + R be continuous function such that 7(q) f(((q)) for q E 1) Then from Eq (6) and Eq (7),
(
(8)
= (2)) <e
14

---

Published as a conference paper at ICLR 2023
u
U3
-1
+1
-1
U2
-1
+1
-1
1
+1
-1
U1
-1
+1
-1
+1
+1
U5
-1
g(u) = sign( cos u
Figure 5: A Toy 2D Example of Voints. Voints assume view-dependency for every 3D point. Here, we look at a single 2D point at the center with a circular function g(u) = sign (cos u) from five arbitrary view-points {uy]ā-1- Trying to reduce g to single value based on My projections undermines the underlying structure of g. We take the full set ((,))) as representation g and learn a set function f on these features for a more informative manner of representation aggregation.
Note that 7 (q (4)) can be rewritten as follows:
? (2)) 7 (q (uu....,,s))))
(MAX (h (u1). h (ux)))
(9)
(7 o MAX) (h (ui),..., huM))
Since yo MAX is symmetric function and from Eq (8) and Eq (9), we reach to the main result in Eq (5). This concludes the proof.
A.3 3D VOINT CLOUD
Plenoptic and Spherical Coordinate Functions. The Plenoptic function was first introduced by McMillan and Bishop (McMillan & Bishop, 1995) 1995 as general function that describes the visible world. The Plenoptic function P is continuous spherical function that describes the visibility at any Euclidean 3D point in space (V., Vy, Vz) when looking into any direction (8, o) across wavelength À at time t. It is defined as p = P (0, o, X, Vz, Vg, Va,t), t). Such a remarkable and compact formulation covers all the images observed as just samples of the function P. For fixed time and wavelength, the reduced Plenoptic function P becomes p P (00,0, 0, Vz, Vy. Va,) which can describe any field in 3D space. This shortened formulation what Neural Radiance Fields (NeRFs) (Mildenhall et al., 2021) try to learn with MLPs to describe the radiance and RGB values in the continuous Euclidean space with a dependency on the view direction (0, Φ). In the same spirit of the Plenoptic function and NeRFs, the Voint cloud representation relies on the viewing angles (8, Φ) to define the view-features. The problem with the plenoptic functions P, and subsequently NeRFs, is that they are very high dimensional, and any attempt to densely represent. the scene with discrete and fixed data will cause memory and compute issues (Yu et al., 2021; Pumarola et al., 2021). Unlike NERF' (Mildenhall et al., 2020) that define dense 3D volumes, we focus only on the surface of the 3D shapes with our Voint clouds representation. Our Voints are in the order of the sampled point cloud, offering compact representation that allows for efficient computation and momory while maintaining the view-dependent component that facilitates visw-basod learning.
al., 2020; Pumarola et al., 2021;
From Point Clouds to Voint Clouds. Implicit representation of 3D surfaces typically aims to learn an implicit function (x(x) R3 R that define the Sign Distance Function
15

---

Published as a conference paper at ICLR 2023
(SDF) the occupancy in the continuous Euclidean space (Park et al., 2019; Mescheder et al., 2019). The 3D iso-surface is then defined as the set all points x that satisfy the condition (x(x) = 0 (assuming (x(x) SDF hereafter). We define surface 3D point cloud X' E Nx as a set of N 3D points, where each point x; E R$ is represented by its 3D coordinates (zi, Vis z:) and satisfy the iso-surface condition follows.
== = x; E R (x.))) - 0)
(10)
Here, assume that surface points also depend on the view direction from which they are being observed. Specifically, there exists a continuous implicit spherical function g(x, u) R + Rd that defines the features at each point depending a set of M view-point directions E IR Mx2, a Voint X E R afd is a set of M view-dependent features of size for the sphere centered at point x. The Voint cloud & E Rxxxxx is the of all N Voints X. x; = (g, uy) E RR x, E 23
the view direction Given
(11)
x = & E Mxdd N i=1
Note that we typically do not have access to the underlying implicit function g and we approximate it by 2D projection, feature extraction, and then un-projection as we show next.
1- Multi-View Projection. As mentioned earlier, a Voint combines multiple view-feat ures of the same 3D point. These fiew-features come from a multi-view projection of the points by a point cloud renderer R Nx + IR Mx xW x3 that renders the point cloud X' from multiple view-points U1 into M images of size H W x 3. In addition projecting the point cloud into the image space, R defines the mapping B E NJ...,......... between each pixel to the N points and background renders. Also, R outputs the visibility binary matrix V E {0, 1IWNM for each point from each view. Since not all points appear in all the views due to pixel discrelization, the visibility score Vs,, defines if the Voint &; is visible in the view u, The matrix B is crucial for unprojection, while V is needed for defining meaningful operations on Voints.
2- Miew Feature Extraction. The rendered images are processed by function C: MxH + MxHW xd that extracts image features. If is the identity function, all the view-features would be identical for each Voint (typically the RGB value of the corresponding point). However, the C function can be a 2D network dedicated to the downstream lask and can extract useful global and local features about each view.
3- Multi-View Unprojection. We propose a module O MMMMMMXXWW. - Nxxx that unprojects the 2D features from each pixel be 3D view-features at the corresponding Voint. This is performed by using the mapping B created by the renderer to form the Voint cloud features X. Note that the points are not necessarily visible from all the views, and some Voints that are not visible from any of the M views will not receive any features. We post-process these empty points (~ 0.5% of points during inference) to be filled with nearest 3D neighbors features. The output Voint cloud features would be described as follows.
x; (gij, E
EXX, Vi = 1
g.j = tB (C(R(X) B)
(12)
2 = (8, E R M
1-1
A.4 VOINT OPERATIONS
Voxn In order to learn a neural network in the Voint space in the form dictated by Theorem 1, need to define some basic differentiable operations on the Voint space. The
16

---

Published as a conference paper at ICLR 2023
Voint
Shared MLP
Graph Convolution
Virtual View-Feature
MIP
Voint Graph
Voint
Voint
Figure 6: VointNet Variants. We propose three variants VointNet that use three different examples of VointConv operation h:: shared MLP (MLP), Graph Convolution (GCN), and Graph Attention (GAT). Here highlight the main difference between VointNet (MLP) that shares the MLP all the view-features and VointNet (GCN) that creates fully connected graph on view-features and learn an MLP on the edge fiat-features. VointNet (GAT) is similar to VointNet (GCN) in addition to learning attention weights for each view-feature weighted average aggregation.
max operation on the Voint cloud can be defined as follows.
Vax max Ri.j. Vi,j
(13)
s.t. E 1,2, 1...... N1 jE 1,2, M .V. =1
Equivalently, VointMax(X) maxy (R.. xo V..)) where V is the complement of V.
Conv. We define the convolution operation hy Nxxxd + R Nxxd any learnable function that operates on the Voint space with shared weights on all the Voints and has the view-features input size d and outputs view-features of size d' and consists of ly layers. Examples of this VointConv operation include the following operations applied only on the visible view-features: a shared MLP, graph convolution, and a graph attention. We detail these operations later in Section A.6, which result in different non-exhaustive variants of VeintNet.
A.5 LEARNING ON 3D VOINT CLOUDS
VointNet. Typical 3D point cloud classifiers with a fcature max pooling layer work as in Eq (14), where hllp and hpcony are the MLP and point Convolutional (1 x 1 or edge) layers, respectively. This produces K-class classifier F.
F(X)= hmip (max (hreone (xx))) x.EX
(14)
Here, F: NX3 + R produces the logits layer of the classifier with size K. On the other hand, the goal of the VointNet model is to get multi-view point cloud features that be used after which by any point cloud processing pipeline. The VointNet module F; NxAxxx Nxd as follows.
f(X) = hp (VointMax (hr
(15)
A.6 VOINNNET VARIANTS
We define the convolution operation hy: Nxxxd NxMxd in VointNet from Eq (15) any learnable function that operates the Voint space with shared weights all the
17

---

Published as a conference paper at ICLR 2023
Voints and has the view-features input size d and outputs view-features of size d' and consists of lv layers. Examples of this VointConv operation inchude the following:
Shared MLP. It is the most basic Voint neural network. For layer I, the features of Voint i at view j is updated as follows to layer / 1
h h - p (hi, Vi,j
(16)
s.t. 1,,,,,22... N j,,,,,.... M Vu, -1
where p is the shared MLP with weights W, followed by normalization and nonlinear function ( e.g. ReLU) applied all Voints independently at the visible views features for each Voint. This formulation extends the shared MLP formulation for NointNet (Qi et. al., 2017a) to make the MLP shared across the Voints and the views-features.
Graph Convolution (GCN). Just like how DGCNN (Wang et al., 2019c) extended PointNet (Qi et al., 2017a) by taking the neighborhood information and extract edge features, we extend the basic Net formulation in Eq (15). We define a fully connected graph for each Voint along the views dimension by creating center virtual node connected to all the view features ( similar to the classification token in VIT (Dosovitskiy et al., 2021)). This center virtual view-feature would be assigned the index j and can be initilized with veros as the "cls' token in ViT (Dosovitskiy et al., 2021). Then, Voint graph convolution operation can be defined as follows to update the activations from layer I to 1+1
h = (07=~(((m)) ((mmax v ((h) k(w))))) )
k
(17)
Vi,j s.t. E 1,2,... N. j E 0, 1,...... M E 1E.,,....MM kk + Vi =1
where p. v are two different shared MLPs as in Eq (16). The difference between VointNet (MLP) and VointNet (GCN) is highlighted in Figure 6.
Graph Attention (GAT). Similar to how Point Transformer (Zhao et al., 2020) extended the graph convolution by adding attention to DGCNN (Wang et al., 2019c), extend the basic Voint GraphConv formulation in Eq (17). Voint graph attention operation can be defined as follows update the activations from layer to 101+1
h h#:1=/ = (
M
Σ
net ((h), M(x))))))) (Wo
k=0,
(18)
Vi,j s.t. iE 1,2, j E 0, I..,, M Vij=
Tk = < (hl,
where p, S, are three different shared MLPs in Eq (16), and ma the learned attention weights for each neighbor view-feature.
B DETAILED EXPERIMENTAL SETUP
B.1 DATASETS
ScanObjectNN: 3D Point Cloud Classification. We follow the literature (Goyal et al., 2021: Hamdi al., 2021) on testing 3D classification in the challenging ScanObjeetNN (Uy et al., 2019) point cloud dataset, since it includes background and considers occlusions. The dataset is composed of 2902 point clouds divided into 15 object categories. We use 2048 sampled points per object for Voint learning. We benchmark on its variants: Object only, Object with Background, and the Hardest perturbed variant (PB T50 RS variant). Visualization is provided in Figure 7 of some of the renderings used in training the 2D backbone in our pipeline.
ShapeNet Core55: 3D Shape Retrieval. The shape retrieval challenge SHREC (Sfikas et al., 2017) uses ShapeNet Core55 is a subset of ShapeNet (Chang et al., 2015) for bench- marking. The dataset consists of 51,162 3D mesh objects labeled with 55 object classes. The
18

---

Published as a conference paper at ICLR 2023
training, validation, and test sets consist of 35764, 5133, and 10265 shapes. We create dataset of point clouds by sampling 5000 points from each mesh object as in MVTN (Hamdi et al., 2021).
ShapeNet Parts: 3D Part Segmentation. ShapeNet Parts is a subset of ShapeNet (Chang et al., 2015) that consists of 13,998 point cloud objects for train and 2,874 objects for the test from 16 categorics and 50 parts. It is designed for the part segmentation task (Yi et al., 2016). Visualization provided in Figure 10 of some of the renderings used in training the 2D backbone in our pipeline colored with the ground truth segmentation labels.
4odelNet 3D Shape Classification Occlusion Robustness. ModelNet40 (Wu et al., 2015) is composed of 12,311 3D objects (9,843/2,468 in sraining/testing/) labelled with 40 object classes. We sample 2048 points clouds from the objects following previous works (Qi et al., 2017b; Zhao et al., 2020). Visualization is provided in Figure 8 of some of the renderings used in training the 2D backbone our pipeline.
B.2 METRICS
Classification Accuracy. The standard evaluation metric in 3D classification is accuracy. We report overall accuracy (percentage of correctly classified test samples) and average per-class accuracy (mean of all true class accuracies).
Retrieval mAP. Shape retrieval is evaluated by mean Average Precision (mAP) over test
queries. For every query shape S from the lest set, AP is defined AP = CTF Σn 1S..), where GTP is the number ground truth positives, N is the size of the ordered training set, and (S,,,) 1 if the shape Sn, is from the same class label of query Sq. We average the retrieval AP over the test set to measure retrieval mAP.
Segmentation mloU. Semantic Segmentation is evaluated by mean Intersection over Union (mloU) over pixels or points. For every class label, measure the size of the intersection mask between the ground truth points of that label and the predicted points that label. Then, divide by the size of the union mask of the same label to get ToU. This procedure is repeated all the labels, and averaging the ToUs gives mIoU. We report two types of ms:Us: instance-averagee mloU (averages all mloUs across all objects and Category-averagel mloU (averages all mloU from shapes of the same category, and then average those across object categories).
B.3 BASELINES
Point Cloud Networks. We include PointNet (Qi et al., 2017a), PointNet++ (Qi et al., 2017b), DGCNN (Wang et al, 2019c), PVNet (You et al., 2018), and KPConv (Thomas et al., 2019), Point Transformer (Zhao et al., 2020) and CurveNet (Xiang et al., 2021) as baselines that use point clouds. These methods leverage different convolution operators point clouds by aggregating local and global point information.
Multi-View Networks. We also compare against multi-viow classification approaches like MVCNN (Su et al., 2015) and MVTN (Hamdi et al., 2021) as basclines for classification and available multi-view pipeline for 3D part segmentation, we adopt some of the multi-view segmentation baselines (e.g. Label Fusion (Wang et al., 2019a) and Mean Fusion (Kundu et al., 2020)) for part segmentation work in the Voint space.
retrieval. Since there
B.4 IMPLEMENTATION DETAILS
Rendering and En-Projection. We choose the differentiable point cloud renderer R from Pytorch3D (Ravi et al., 2020) in our pipeline for its speed and compatibility with Pytorch libraries (Paszke et al., 2017). We render multi-view images with size 224 x 224 x 3. We color the points by their normals' values or keep them white if the normals are not available. Following similar procedure to (Wei et al., 2020; Hamdi et al., 2021), the view-point setup randomized during training (using M views) and fixed to spherical views in testing (using M 12 views).
19

---

Published as a conference paper at ICLR 2023
Object Only
With Background
PB T50 RS (Hardest)
Figure 7: ScanObjectNN Variants. We show examples of point cloud renderings of different variants of the ScanObjectNN (Uy et al., 2019). These renderings are used in training VointNet for 3D point cloud classification.
VIT (Dosovitskiy et al., 2021) (with pretrained weights from TIMM library (Wightman, 2019)) for classification and DeepLabV3 (Chen et al., 2018) for segmentation. We used parallel heads for cach object category for part segmentation since the task is solely focused on parts. We use the 3D cross-entropy loss on the 3D point cloud output and the 2D cross-entropy loss when the loss is defined the pixels. When used, the linear tradeoff coefficient of the 2D loss term is set to 0.003. To balance the frequency of objects part segmentation, we multiply the loss by the frequency of the object class of each object we segment. The feature dimension of the VointNet architectures d = 64, and the depth is lv layers hv. The main results are based on VointNet (MLP) variant unless otherwise specified. The coordinates can be optionally appended to the input view-features X, which can improve the performance but reduce the rotation robustness we show later in Section C.1 and Table 9.
Architectures. For the 2D backbone,
Training Setup. We train our pipeline in two stages, where we start by training the 2D backbone the 2D projected labels of the points, then train the full pipeline end-to-end while focusing the training on the VointNet part. We use the AdamW optimizer (Loshchilov & Hutter, 2017) with initial learning rate of 0.0005 and step learning rate schedule of 33.3% every 12 epochs for 40 epochs. The pipeline trained with one NVIDIA Tesla V100 GPU. We do not any data augmentation.
20

---

Published as a conference paper at ICLR 2023
Figure 8: ModelNet40. We show some examples of point cloud renderings of ModelNet-40
(Wu al., 2015) used for 3D classification robustness our setup.
Figure ShapeNet Core55. We show some examples of point cloud renderings of ShapeNet Core55 (Chang et al., 2015) used for 3D shape retrieval in our setup.
21

---

Published as a conference paper at ICLR 2023
+XX
+
Figure 10: ShapeNet Parts. We show some examples of point cloud renderings of ShapeNet Parts (Yi et al, 2016) colored with ground truth segmentation labels. We use these renderings as 2D ground truth to trein the 2D backbone for 2D segmentation before training VointNet's s pipeline for 3D segmentation.
22

---

Published as a conference paper at ICLR 2023
Classification Data Type ModelNet40 ShapeNet Core
Shape Retrieval
Method
PointNet (Qi et al., 2017a)
Points
89.2
PointNet- (Qi et al., 2017b)
Points
91.9
DGCNN (Wang et al., 2019c)
Points
92.2
XPCony(Thommn
al., 2019)
Points
92.9
PCT et al., 2021)
Points
93.3
CurveNet et al., 2021)
Points
93.8
ReVGG (Sfikas et al., 2017)
M-View
74.9
MVCNN (Su et al., 2015)
M-View
90.1
73.5
ViewGCN (Wei et al., 2020)
M-View
93.3
78.4
MVTN (Handi et al., 2021)
M-View
93.8
82.9
VointNet (ours)
Voints
92.8
83.3
Table 3D Shape Classification and Retrieval. We report VointNet's classification accuracy on ModelNet40 (Wu et al., 2015) and its 3D shape retrieval mAP on Shape Core55 (Chang et al., 2015; Sfikas et al., 2017). Baseline results are reported from (Hamdi et al., 2021; Zhao et al., 2020; Xiang et al., 2021).
Rotation Perturbations Range
Method
0
9
1180
PointNet (Qi et al., 2017a)
88.7
42.5
38.6
PointNet ++ (Qi et al., 2017b)
88.2
47.9
39.7
RSCNN (Liu et al., 2019a)
90.3
90.3
90.3
MVTN (Hamdi et al., 2021)
91.7
90.8
91.2
Net (ours)
91.5
90.9
91.1
Table 8: Rotation Robustness for 3D Classification. At test time, we randomly rotate objects in NodelNet40 (Wu et al., 2015) around the Y-axis (gravity) with different ranges and report the overall accuracy.
C ADDITIONAL. RESULTS
C.1 MODEL ROBUSTNESS
Rotation Robustness for 3D Classification. We follow the standard practice 3D shape classification literature by testing the robustness of trained models to perturbations at test time (Liu et al., 2019a; Hamdi et al., 2021). We perturb the shapes with random rotations around the Y-axis (gravity-axis) contained within +900 and +180 and report the test accuracy over ten runs in Table 8.
Rotation Robustness for 3D Segmentation. We follow the previous 3D literature by testing the robustness of trained models to perturbations at test time (Liu et al., 2019a; Hamdi et al., 2021; 2020). We perturb the shapes in ShapeNet Parts with random rotations in SO(3) at test time (len runs) and report. Ins. mloU in Table 9. Note how our VointNet performance largely exceeds the baselines in this realistic unaligned scenario. We can augment the training with rotated objects for the baselines, which improves their robustness, but loses performance on the unrated setup. Adding zyz coordinates the view-features of VointNet improves the performance on an unrotated sctup but negatively affects the robustness to rotations. The discrepancy between the Voint results and the results of some point cloud methods is that Voints heavily depend the underlying 2D backbone and inherit all its biases, especially those from pretraining. Hence, the 2D backbone limits what the performance can reach with VointNet. We study the effect of the backbone in detail in Section C.2. Figure 11 shows qualitative 3D segmentation results for VointNet and Mean Fuse (Kundu et al., 2020) as compared to the ground truth.
23

---

Published as a conference paper at ICLR 2023
Ground Truth
VointNet (ours)
Mean Fuse (Kundu et al., 2020)
Figure 11: Qualitative Comparison for 3D Part Segmentation. We compare VointNet 3D segmentation prediction to Mean Fuse (Kundu et al., 2020) that using the same trained 2D backbone. Note how VointNet distinguishes detailed parts (e.g. the car window frame). Beware that visualization colors can shift if an extra label is predicted (e.g. the motorbike labels are correct).
24

---

Published as a conference paper at ICLR 2023
Segmentation Under Rotation
Method
Unrotated
Rotated
PointNet (Qi et al., 2017a)
80.1
36.6 =0.2
DGCNN (Wang et al., 2019c)
80.1
37.1 =0.2
PointNet + Aug.
65.8
65.8 #0.1
DGCNN + Aug,
60.7
#0.2
Mean Fuse (Kundu et al., 2020)
79.1
61.6 =0.1
Label Fuse (Wang et al., 2019a)
78.9
61.0 + 0.1
VointNett (w/o zyz) VointNet (w/o ryz) + Aug. VointNet (w/ ryz)
79.6
63.4 0.1
68.0
68.5 + 0.1
81.2
61.5 #02
Table 9: Rotation Robustness for 3D Part Segmentation. At test time, randomly rotate objects from ShapeNet Parts (Yi et al., 2016) and report the Ins. mloUs of our VointNet compared to trained PointNet (Qi et al., 2017a) and DGCNN (Wang et al., 2019c). Note how VointNet's performance largely exceeds the baselines in realistic unaligned scenarios, highlighting the benefit of view dependency. If we rotation augmentation in training for the baselines, the rotated performance improves, but the unrolated performance drops.
C.2 DETAILED ANALYSIS
Effect of Pretraining. We study the effect of pretraining the 2D backbone C for 3D classification on ModelNet40. Training a ViT with Mean Fuse for 3D classification on ModelNet40 obtains 92.2 test Acc. with ImageNet pretraining and 80.0 test Acc. from scratch. Other multi-view networks, e.g. MVCNN (Su al., 2015), ViewGCN(Wei et al., 2020), and MHTN(Hamdi et al., 2021) all use ImageNet pretraining, which is not unique to Voints.
Classification Backbone. We study the effect of ablating the 2D backbone C for 3D classification on ModelNet40. We show in Table the performance of VointNet (MLP) when Vit-B (Dosovitskiy et al., 2021) and ResNet-18 (He al., 2015) are used. We also show that following the per-point classification setup instead of the per-shape for 3D) shape classification leads to worse performance for VointNet and the naive multi-view. This is why we used the per-shape approach when adopting VointNet for 3D) classification (using one Voint for the entire shape).
Number of points and visibility. Table 11 studies the effect of point number on 3D part segmentation performance, when different numbers of views used. The visibility ratio is also reported in each case.
We colored the points with ground truth normals as in Figure 16, when they are available (ShapeNet Parts), and we used white colors as in Figure 9, when other baselines do not use normals. We ablate the color of the points on VointNet (MLP) with normals colors, white color, and NOCs colors (Wang et al., 2019b). We obtain the following segmentation mloU results: (normals: 80.6), (white: 74.7), and (NOCs: 57.9).
Points color.
Time and Memory Requirements. To assess the contribution of the Voint module, we take macroscopic look at the time and memory requirements of each component in the operations (GFLOPs) and the time of
pipeline. We record the number of floating-pote a forward pass for a single input sample. In Table 12, the VointNet module contributes negligibly the memory requirements compared to multi-view and point networks.
Feature Size (d). We study the effect of the feature size d on the performance of VointNet (MLP) in 3D part segmentation with confidence intervals) in Figure 12. We note that the performance peaks at d = 128, but is close to what
ShapeNet Parts (Yi et al., 2016) and plot the results (
in the main results (d 64).
25

---

Published as a conference paper at ICLR 2023
View Aggregation
2D Backbone
ResNet18 (per-shape)
ViT-B (per-shape) (per-point)
DeepLabV3
Nei
91.2
92.8
10.2
Table 10: Ablation Study for 3D Classification. We study the effect of different 2D backbone for ModelNet40 3D classification task. We compare VointNet's performance to naive multi-view (e.g. MVCNN (Su et al., 2015) Mean Fuse (Kundu et al., 2020)) using the same 2D backbone. Note that using the per-point classification setup instead of the per-shape for 3D shape classification leads to worse performance for VointNet and the naive multi-view.
Number of Views
Points #
Metric
2
12
visibility
99.1
99.9
100
100
500
mloU
69.2
73.9
76.0
76.4
visibility
98.0
99.7
100
100
1000
mIoU
69.5
74.3
76.5
77.1
visibility
95.7
99.2
99.8
99.9
2000
mloU
69.7
75.0
77.7
78.5
Table 11: Analysis on Number of Points and Visibility. We show the Instance mloUs and visibility ratio (1 (1- amply 1% of our VointNet on ShapeNet Parts when varying points # and number of views.
Model Depth (I.). We study the effect of the model depth ly on the performance of VointNet (MLP) in 3D part segmentation ShapeNet Parts (Yi et al., 2016) and plot the results ( with confidence intervals) in Figure 13. We note that model depth of VointNet does not enhance the performance significantly. Our choice of 4, =4 balances the performance and the memory/computatinnn
requirements of Nei (MLP).
Distance to the Object. We study the effect of distance the object in rendering as in Figure 17 to the performance of VointNet (MLP) in 3D part segmentation on ShapeNet Parts (Yi al., 2016) and plot the results ( with confidence intervals) in Figure 14. We note that our default choice of 1.0 is actually reasonable. This choice of distance shows the object entirely ( as illustrated in Figure 17), but also cover the details needed for small parts segmentation (see Figure 11).
Image Size (H, (). We study the effect of the image size H&W on the performance of Mean Fuse (Kundu et al., 2020) baseline when training the 2D backbone for 3D part segmentation. We plot the results ( (with confidence intervals) in Figure 15.
Number of Views Classification. We study the effect of the number of views (M) on classification accuracy on ModelNet40 Wu et al. (2015) of Voint.et and report results in Table 13.
Unprojection Operation Speed. We evaluate the speed of the unprojection operation ΦB and report average latency of 10,000 runs (in ms) in Table 14.
Unprojection Operation Speed. We evaluate the speed of the point cloud renderer I used in Voint pipeline from Pytroch3D Ravi et al. (2020) and report average latency of 1,000 renderings (in ms/image) in Table 15.
C.3 VISUALIZATIONS
In Figure 16 and 17, we visualize the multi-view renderings of the point clouds along with the 2D learned features based on the DeepLabV3 (Chen et al., 2018) backbone. These features are then unprojected and transformed by VointNet to obtain 3D semantic labels.
26

---

Published as a conference paper at ICLR 2023
Network
GFL.Pss
Time (ms)
Parameters # (M)
MVCNN (Su et al., 2015)
43.72
39.89
11.20
ViewGCN (Wei al., 2020)
44.19
26.06
23.56
ResNet 18 (He et al., 2015)
3.64
3.70
11.20
ResNet 50 (He ct al., 2015)
8.24
9.42
23.59
ViT-B (Dosovitskly et al., 2021)
33.70
12.46
86.57
VIT-L (Dosovitskiy
al., 2021)
119.30
29.28
304.33
FCN (Long et al., 2015)
53.13
10.34
32.97
DeeplabV3 (Chen al., 2018)
92.61
20.62
58.64
PointNet (Qi et al., 2017a)
1.78
4.24
3.50
DGCNN (Wang et al., 2019c)
10.42
0.95
16.350
MVTN (Hamdi et al., 2021)
1.78
4.24
3.5
Veint (MLP)
1.90
2.90
0.04
Net (GCN)
16.18
32.10
0.05
VointNet (GAT)
32.05
68.71
0.07
Full Voint pipeline
94.51
23.50
58.68
Time and Memory Requirements. We assess the contribution of the Voint module to the time and memory requirements in the multi-view and point cloud pipeline. Note that VointNet (shared MLP) is almost 100 times smaller than PointNet (Qi et al., 2017a).
Table 12:
80.5
VointNet (MLP)
lest miou (7)]
80.0
79.5
79.0
50
100
150
200
250
Featrure Size (d)
Figure 12: The Effect of Feature Size d. We plot Ins. mloU of 3D segmentation us. the feature size d used in training on ShapeNet Parts (Yi al., 2016). We note that the performance peaks at d = 128, but it is close to what use in the main results (d = 64).
27

---

Published as a conference paper at ICLR 2023
VointNet (MLP)
80
Test mloU (%)
79
78
2
4
6
8
10
12
Model Depth (l)
Figure 13: The Effect of Model Depth 4,. We plot Ins. mloU of 3D segmentation the model depth 4, used in training on ShapeNet Parts (Yi et al., 2016). We note that model depth of VointNet does not enhance the performance significantly. Our choice of L, balances the performance and the memory/computatioon requirements of VointNet (MLP).
80.0
VointNet (MLP)
lest miou (%)
77.5
75.0
72.5
70.0
0.6
0.8
1.0
1.2
1.4
Distnace to the Object (unit)
Figure 14: The Effect of Distance to the Object. We plot Ins. mloU of 3D segmentation vs. the distance to the object used in inference on ShapeNet Parts (Yi et al., 2016). We note that our default choice of 1.0 is actually reasonable. This choice of distance shows the object entirely ( as illustrated Figure 17), but also cover the details needed for small parts segmentation (see Figure 11).
77.5
Mean Fuse
lest miou (76)
75.0
72.5
70.0
67.5
65.0
180
200
220
240
260
280
Image Size
Figure 15: The Effect of Image Size H. W. We plot Ins. mloU of 3D segmentation es. the image size used in inference on ShapeNet Parts (Yi et al., 2016).
28

---

Published as a conference paper at ICLR 2023
(INPUT)
(PRED 2D)
(PRED 3D)
(GT)
(INPUT)
(PRED 2D)
(PRED 3D)
(GT)
Figure 16: Multi-view Projected Segmentation We show how, after rendering points, we can segment in the image space. For each example, we show (INPUT): the projections of the points (colored with normals) used in training with random view-points. (PRED 2D): the segmentation prediction of the 2D backbone (DeepLabV3) (Chen et al., 2018)). (PRED 3D): the unprojected 3D segmentation prediction. (GT): the 3D segnentattio ground truth.
29

---

Published as a conference paper at ICLR 2023
(INPUT)
(PRED 2D)
(PRED 3D)
(GT)
(INPUT)
(PRED 2D)
(PRED 3D)
(GT))
Figure 17: Multi-view Projected Segmentation 2. We show how, after rendering points, we can segment in the image space. For each example, we show (INPUT): the projections of the points (colored with normals) used in training with random view-points. (PRED 2D): the segmentation prediction of the 2D backbone (DeepLabV3) (Chen et al., 2018)). (PRED 3D): the unprojected 3D segmentation prediction. (GT): the 3D segnentattio ground truth.
30

---

Published as a conference paper at ICLR 2023
Number of Views
Method
4
10
VointNet (Cls. Acc.) 90.3
90.8
92.0
92.3
Table 13: Effect of the Number of Views on Classification. report classification accuracy of NeintNet vs. the number of views (M) used in the training on ModelNet40..
Number of Views
Method
1
2
4
8
10
12
Features Unprojection
3.0
5.3
11.45
15.7
17.2
29.7
24.0
Labels Unprojection
2.6
2.5
3.4
3.1
3.0
3.2
3.6
Unprojection Operation Speed. We report the average latency (in mus) over 10,000 runs of the unprojection operation with its two forms: features unprojection (used in mean) and labels umprojection (used in mode).
Table 14:
Number of Points
Criteria
1e2
1e3
1e4
1e5
1e6
Point Rendering Speed (ms/image /imsee
7.2
7.6
7.7
10.4
37.7
Table 15: Point Rendering Speed. We report the average rendering speed (in ms/image) over 1,000 renderings of the point cloud renderer Ravi et al. (2020) used Voint clouds.
31


Reviews:
[{"review_id": "uFgRgVX6POE", "reviewer": "Reviewer_KkFB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a novel 3D representation called \"Voint Cloud.\" The authors argue against existing heuristic-based aggregation, which could misleadingly incorporate representation from an arbitrary viewpoint. Then they attempted to address the research question of how to aggregate the per-view representation (in a multi-view setting) for a given 3D point cloud. In the process, they proposed their 3D representation, Voint, which is a view-dependent representation associated with each 3D point in a multi-view representation learning setting. The paper derived and described how their Voint Cloud (the proposed multi-view point cloud) could be used to learn representation end-to-end fashion to solve various 3D scene understanding tasks. Experimentally the authors demonstrated the usefulness of the proposed representation on three different 3D understanding tasks: i) 3D point cloud classification (on ScanObjectNN dataset), ii) 3D shape retrieval (on ScanNet Core55), and iii) 3D part segmentation (on ScanNet). The proposed method attained state-of-the-art performance in 3D point cloud classification and 3D object retrieval.", "summary_of_the_review": "The paper has some weaknesses that have been discussed in the limitation section; however, it introduced a new idea (novel 3D representation) and a new methodology (VointNet and its corresponding operations).", "strength_and_weaknesses": "Strengths:\n+ New 3D representation which seamlessly integrates view-dependent representation for each 3D point\n+ Proposed new pooling and convolution operation on the Voint Cloud leading to a new deep neural network called VointNet\n+ state-of-the-art performance on 3D point cloud classification and 3D shape retrieval tasks using the proposed Voint Cloud representation\n\nWeaknesses:\n+ Generalizability to outdoor 3D scene understanding task: All experiments were done on indoor 3D scene understanding tasks (ScanObjectNN, ScanNet). To make a more generalizable claim for Voint Cloud representation, at least one experiment should have been done on an outdoor 3D scene understanding task, e.g., the KITTI dataset. Is there any specific reason for not doing that? \n\n\n+ 3D part segmentation: The proposed method achieved better results against other baselines on a more challenging setup (rotated); however, CurveNet still retained the best performance 84.9 % mIoU on the \"Unrotated setup.\" The paper did not discuss what could be the possible reason for this slight discrepancy.  The ablation study in Table 6 shows different backbones and different variations of VointNet. It is unclear which result from that table corresponds to the one reported in Table 4. I also acknowledge the fact that the paper mentions that unless otherwise stated, the default method for VointNet is shared MLP. \n\n+ Backbone in 3D part segmentation: The backbones used in the 3D part segmentation task are widely used methods (FCN, DeepLabv3). The newer backbone might be more effective and possibly could improve the results. Please refer to the model \"HRNet-W48\" which was used in the following work: \"MSeg: A Composite Dataset for Multi-domain Semantic Segmentation - Lambert, John, et al. CVPR'20\"\n\n+ Number of views: The authors showed an ablation study for the effect of the number of views on the performance of the Voint Cloud. However, it was shown only for the 3D part segmentation task. How about the 3D point cloud classification and 3D object retrieval tasks? \n\n+ 3D point cloud classification and 3D shape retrieval experiment: The proposed Voint Cloud-based representation achieved state-of-the-art results. It would be interesting to show the comparison against methods with deformable convolution on Point Cloud, such as \n\"KPConv: Flexible and Deformable Convolution for Point Clouds - Hugues Thomas et al. ICCV'19\"\n\n\n+ VointNet variations: The paper shows three different variations of VointNet, the simplest of which is a shared MLP-based architecture. Figure 6 (in the appendix) could be included in the main paper to describe the method clearly. \n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "drRMk-gq0Tq", "reviewer": "Reviewer_LsWo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a multi-view 3D point cloud representation called Voint cloud. The multi-view features are extracted from 2D networks and then aggregated into 3D point clouds using correspondence. Visibility of the points in each view is also attached. The proposed method reaches state-of-the-art-performance on several 3D understanding tasks, including 3D shape classification, retrieval, and robust part segmentation.", "summary_of_the_review": "The idea is somewhat novel and results show the effectiveness of proposed method. Incremental change compared to previous method. It'd be nice to see the latency analysis.", "strength_and_weaknesses": "Strength:\n- The idea of extracting low-level features from multi-view 2D images first and then aggregating back into 3D is novel. This takes advantage of mature 2D networks and still maintain the 3D geometry.\n- The paper is well-written with extensive experiments and ablation studies.\n\nWeakness:\nI would like to see latency analysis of each component of proposed method and also in comparison with other methods. Looks like unprojection requires a lot of irregular memory access.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "aT0dXpfu-qO", "reviewer": "Reviewer_14aj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a novel point cloud backbone. The backbone renders point clouds into 2D multi-view renderings and extracts 2D features with 2D backbones. These features are then unprojected back onto 3D point clouds where they are fused into final point features. \nThe backbone is very compact and efficient.\nThe authors have performed extensive experiments including 3D point cloud classification, shape retrieval, and 3D part segmentation, and shown good performance on all benchmarks.", "summary_of_the_review": "I'm leaning toward acceptance due to the impressive performance and efficiency of the method. Adding more discussions and completing the details suggested above could further strengthen my recommendation.", "strength_and_weaknesses": "Strength\n- Though counter-intuitive, the idea of extracting features in 2D and fusing in 3D is interesting, clean, and neat. The method achieves very impressive performance on all tasks while keeping a compact model size and fast inference efficiency.\n- Experiments have shown that the proposed backbone is less prone to partial observation and change of orientation.\n\nWeaknesses\n- The authors have not studied how noise will affect the backbone performance. It could be much harder to extract useful information in 2D than in 3D for noisy point clouds. This could potentially be a case where 3D methods take advantage.\n- The authors seem to deliberately use different view numbers (8 vs. 12) and sampling methods (random vs. spherical) during training and testing. This is not explained and studied in the paper.\n- The authors should include more details about the renderer e.g. speed, color?, point size, etc.\n\n\n- If it is possible to render point clouds from different distances (say 0.5, 1.0, and 2.0) and use them as multi-scale features? Many 2D and 3D backbones employ similar ideas explicitly. Any discussion will be appreciated.\n\nMinor\n- Table 3: SahpeNet -> ShapeNet\n- Figure 14: Distnace -> Distance\n- Figure 12-15 appear to be very large and not nicely rendered (14,15)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "K9jDGR13gym", "reviewer": "Reviewer_aJdJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose the use of image based features coupled with 3D points to train a network to perform various 3D perception tasks, such as classification, retrieval or segmentation, on point clouds.  The authors demonstrate state of the art performance on synthetic several datasets.", "summary_of_the_review": "Although I have some concerns about the overall terminology and limited novelty of the paper, I think that the method is well executed and the results support the direction. I recommend the paper for publication. ", "strength_and_weaknesses": "The paper is well written. The approach is well evaluated and the results are strong. The proposed idea is a well-executed derivation of existing art.\n\nWeaknesses:\n\n1) The novelty of the paper is limited. Essentially, the authors create am image based viewing track of a 3D point (common in SFM and SLAM) and modify pointNet to work in a higher dimensionality space.\n\n2) There is too much effort from the authors to establish the term Voint (It is not clear if there is a fundamental difference between max pooling and convolution with VointMax and VointConv other than applying the operation on higher dimension. )\n\n3) The method is assuming that the render is able to project 3D points to 2D with perfect occlusion detection. Have the authors considered how VointCloud can be extended to more complex point-clouds where mesh reconstruction from a renderer may not guarantee high occlusion detection accuracy?\n\n4) How efficient is VointCloud and VointNet in terms of computational and space complexity? \nMinor comments \n1) Un-projection: The correct term is back-projection. \n2) Eq2 M,N are undifined symbols \n3) Table 3 ShapeNet ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster