Title: SAMPLE-BASED POINT CLOUD DECODER NETWORKS

Abstract: Point clouds are a flexible and ubiquitous way to represent 3D objects with arbitrary resolution and precision. Previous work has shown that adapting encoder networks to match the semantics of their input point clouds can significantly improve their effectiveness over naive feedforward alternatives. However, the vast majority of work on point-cloud decoders are still based on fully-connected networks that map shape representations to a fixed number of output points. In this work, we investigate decoder architectures that more closely match the semantics of variable sized point clouds. Specifically, we study sample-based point-cloud decoders that map a shape representation to a point feature distribution, allowing an arbitrary number of sampled features to be transformed into individual output points. We develop three sample-based decoder architectures and compare their performance to each other and show their improved effectiveness over feedforward architectures. In addition, we investigate the learned distributions to gain insight into the output transformation. Our work is available as an extensible software platform to reproduce these results and serve as a baseline for future work.

1. INTRODUCTION:

Point clouds are an important data type for deep learning algorithms to support. They are commonly used to represent point samples of some underlying object. More generally, the points may be extended beyond 3D space to capture additional information about multi-sets of individual objects from some class. The key distinction between point clouds and the more typical tensor data types is that the information content is invariant to the ordering of points. This implies that the spatial relationships among points is not explicitly captured via the indexing structure of inputs and outputs. Thus, standard convolutional architectures, which leverage such indexing structure to support spatial generalization, are not directly applicable.

A common approach to processing point clouds with deep networks is voxelization, where point clouds are represented by one or more occupancy-grid tensors (Zhou & Tuzel (2018) , Wu et al. (2018) ). The grids encode the spatial dimensions of the points in the tensor indexing structure, which allows for the direct application of convolutional architectures. This voxelization approach, however, is not appropriate in many use cases. In particular, the size of the voxelized representation depends on the spatial extent of the point cloud relative to the spatial resolution needed to make the necessary spatial distinctions (such as distinguishing between different objects in LIDAR data). In many cases, the required resolution will be unknown or result in enormous tensors, which can go beyond the practical space and time constraints of an application. This motivates the goal of developing architectures that support processing point cloud data directly, so that processing scales with the number of points rather than the required size of an occupancy grid.

One naive approach, which scales linearly in the size of the point cloud, is to 'flatten' the point cloud into an arbitrarily ordered list. The list can then be directly processed by standard convolutional or fully-connected (MLP) architectures directly. This approach, however, has at least two problems. First, the indexing order in the list carries no meaningful information, while the networks do not encode this as a prior. Thus, the networks must learn to generalize in a way that is invariant to ordering, which can be data inefficient. Second, in some applications, it is useful for point clouds to consist of varying numbers of points, while still representing the same underlying objects. However, the number of points that can be consumed by the naive feedforward architecture is fixed.

PointNet (Qi et al., 2017) and Deepsets Zaheer et al. (2017) exhibit better performance over the MLP baseline with a smaller network by independently transforming each point into a high-dimensional representation with a single shared MLP that is identically applied to each individual point. This set of derived point features is then mapped to a single, fixed-sized dense shape representation using a symmetric reduction function. As such the architectures naturally scale to any number of input points and order invariance is built in as an architectural bias. As a result, these architectures have been shown to yield significant advantages in applications in which point clouds are used as input, such as shape classification.

The success of PointNet and DeepSet style architectures in this domain shows that designing a network architecture to match the semantics of a point cloud results in a more efficient, and better performing network. Since point clouds are such a useful object representation, it's natural to ask how we should design networks to decode point clouds from some provided shape representation. This would allow for the construction of point cloud auto-encoders, which could serve a number of applications, such as anomaly detection and noise smoothing. Surprisingly, the dominant approach to designing such a differentiable point cloud decoder is to feed the dense representation of the desired object through a single feedforward MLP whose result is then reshaped into the appropriate size for the desired point cloud. This approach has similar issues as the flat MLP approach to encoding point clouds; the decoder can only produce a fixed-sized point cloud while point clouds are capable of representing objects at low or high levels of detail; the decoder only learns a single deterministic mapping from a shape representation to a point cloud while we know that point clouds are inherently random samples of the underlying object.

The primary goal and contribution of this paper is to study how to apply the same lessons learned from the PointNet encoder's semantic congruence with point clouds to a point cloud decoder design. As such, we build on PointNet's principles to present the 'NoiseLearn' algorithm-a novel, simple, and effective point cloud decoding approach. The simplicity of the decoding architectures and the increase in performance are strong indicators that sample-based decoders should be considered as a default in future studies and systems. In addition, we investigate the operation of the decoders to gain insight into how the output point clouds are generated from a latent shape representation.

2. RELATED WORK:

Point cloud decoders are a relatively unexplored area of research. Among the works which describe an algorithm that produces a point cloud, the majority focus their efforts on learning a useful latent shape representation that is then passed to a MLP decoder. PU-Net (Yu et al., 2018) is one such example, in which they design a novel point cloud upsampling network which uses a hierarchical approach to aggregating and expanding point features into a meaningful latent shape representation. To decode the learned shape representation into a point cloud, the latent vector is then passed through a feedforward MLP to produce a fixed number of points. This implies that the network would need to be retrained to allow for a different upsampling rate, which unlikely to be a desired property of an upsampling algorithm.

TopNet (Tchapmi et al., 2019) recognizes the data inefficiency of using a single MLP to decode a point cloud and instead reorganizes their MLP into a hierarchical tree structure in which MLPs at the same level share the same parameters. Their results show that addressing this inefficiency allows for better performance with a smaller parameter count. Similarly, in "Learning Localized Generative Models for 3D Point Clouds via Graph Convolution" Valsesia et al. (2019) augments their decoder by assuming a graph structure over the decoded point cloud and employing graph convolutions. However, despite improved performance neither approach addresses the other issues that come with using MLPs to decode entire point clouds, namely the fixed-size output.

"Point Cloud GAN" (Li et al., 2018) and PointFlow (Yang et al., 2019) take a different approach to producing a point set in a generative setting. Instead of learning a single mapping from any latent vector directly to its decoded point cloud, they learn a function parameterized by the latent vector which transforms low-dimensional Gaussian noise to a 3D point on the surface of the object described by the latent shape representation. This sampling based approach is more in line with the semantics of point clouds. First, an arbitrary number of points can be drawn from the Gaussian noise to produce a point cloud consisting of that number of points without requiring any changes to or retraining of the algorithm. Second, every individual point is decoded independently and identically, which avoids the data inefficiency issues that come with using MLPs to process set data.

While this sampling approach has several desirable properties and appears promising, it's unclear whether the algorithm is applicable outside of the GAN settings these two papers inhabit, if they require specific bespoke loss functions to be trained effectively, or if they are capable of outperforming the baseline MLP approach according to other metrics.

3. POINT-CLOUD DECODING ARCHITECTURES:

A point cloud is a set of n 3D points C = {p 1 , . . . , p N }, where each p i ∈ R 3 . In general, each p i may have additional auxiliary information associated with it via non-spatial dimensions. While all of our architectures easily generalize to include such information, in this paper, we focus on point clouds that exclusively encode shapes without auxiliary information.

A point cloud auto-encoder takes a point cloud C with n points and outputs a point cloud Ĉ with m points that is intended to represent the shape described by C. While often n = m, we are interested in the general case when n and m may be different, which corresponds to up-scaling or downscaling C. Each auto-encoder will be comprised of an encoder E(C), which takes an input point cloud and outputs a latent shape representation h in R l , and a decoder D(h) which maps a latent representation to an output point cloud of the appropriate size. Thus, given an input point cloud C, the auto-encoder output is given by Ĉ = D(E(C)).

In this paper, we focus on the Chamfer distance as the measure of auto-encoder quality. Intuitively this loss function measures how well Ĉ matches C in terms of the nearest neighbor in Ĉ to each point in C and vice versa. Specifically, if dist(p, Ĉ) gives the distance between point p and the nearest neighbor in point cloud Ĉ, our loss function is defined by

L(C, Ĉ) = 1 n p∈C dist(p, Ĉ) + 1 m p∈ Ĉ dist(p, C).

Since the focus of this paper is on point-cloud decoders, all of our architectures use the same pointcloud encoder architecture, while varying the decoder architecture. Below, we first overview the common PointNet-style encoder used followed by a description of the four decoders considered in our experimental analysis, which include three sample-based decoders.

3.1. POINTNET ENCODER ARCHITECTURE:

PointNet (Qi et al., 2017) handles unordered input by recognizing that a symmetric function g (such as element-wise max or sum) produces the same result regardless of the order of its inputs. PointNet thus learns a single transformation function f that maps individual points to an l-dimensional representation and then combines those representations via g. That is, the latent encoding produced by PointNet for a point cloud C = {p 1 , . . . , p n } is the l dimensional vector

E(C) = g (f (p 1 ), . . . f (p N )).

As desired, E(C) is invariant to the ordering of points in C and applies to any number of points.

We learn an MLP representation of f , with input space R 3 , encoding points, and output space R l , encoding the latent representation or point feature. We use max as the reduction function g to map the arbitrary number of resulting point features to a single fixed-size latent shape representation. The hidden layers and size of the latent shape representation for each instantiation of this encoder architecture can be found in Table 1 .

3.2. SAMPLE-BASED DECODERS:

Most prior work has used MLP decoders, which we consider here as a baseline approach. An MLP decoder is a fully connected network that takes the latent shape representation as input and outputs an m × 3 output vector, which represents the m output points. Accordingly, MLP decoders are parameterized by the number and size of their fully connected layers. In our experiments, each fully connected layer consists of parameterized ReLU units with a batch normalization layer.

Our main focus is on sample-based decoders, which allow for an arbitrary number of outputs points to be produced for a latent shape representation. In particular, given a latent shape representation h, each of our decoders is defined in terms of a point feature sampling distribution S(h), where the decoder produces a point-cloud output by sampling m point features from S(h).

Once we have a set of M independently sampled point features from our sampling distribution S(h) we need to transform each one into a triple representing that point's location. Note that we are now in an identical but opposite situation as the point cloud encoder. Whereas the encoder had to transform independent point samples of some underlying object into corresponding high-dimensional representations, our decoder now has to transform independently sampled high-dimensional point representations into a point in space on the surface of the target object. Therefore, we can simply apply the same style of PointNet encoding mechanism with different input and output tensor sizes to implement an effective point feature decoder. The sizes of the hidden layers in our decoder network can be seen in Table 1 . By applying the shared MLP point decoder to each sampled point feature, we can directly decode point clouds of arbitrary size.

Below we describe three architectures for S, which are compared to each other and the baseline MLP decoder in our experiments.

NoiseAppend Decoder. NoiseAppend is similar to the sampling approach described in "Point Cloud GAN" by Li et al. (2018) . They sample point features by simply sampling from a multivariate Gaussian distribution with zero mean and unit variance before appending the sampled noise to the latent shape vector. That is, S(h) = concat (h, N (0, I)).

However, this requires us to decide how many elements of noise should be appended to the latent shape representation. Li et al. (2018) state that the size of the appended noise vector should be 'much smaller than' the size of the latent shape representation, but it's not clear how much noise is necessary to allow the decoder to fully represent the shape. Ultimately this is an additional hyperparameter that needs to be investigated and tuned.

NoiseAdd Decoder. NoiseAdd builds on the concept of adding unit Gaussian noise to the latent shape vector with the goal of avoiding the additional hyperparameter that NoiseAppend introduces. This can be easily accomplished by treating the entire latent vector as the mean of a Gaussian distribution with unit variance. That is, S(h) = N (h, I).

However, this violates the claim by Li et al. (2018) that the amount of noise introduced to the resulting point feature samples should be much smaller than the size of the latent shape representation itself. Therefore, it may be the case that uniformly adding noise to every element of the latent vector obscures the crucial information it represents.

NoiseLearn Decoder. NoiseLearn attempts to instead learn a small separate function V (h) which predicts the log-variance of the desired point feature distribution. Specifically, S(h) = N h, e V (h)/2 I . We define V (h) as a small MLP, the size of which can be seen in Table 1 .

By allowing the network to choose the amount and location of noise to be added to the latent shape vector, we hope that it will learn both to add an appropriate amount of noise for the target shape while All code and infrastructure for "push-button" replication of our experiments open-source (Github/Gitlab location removed for anonymous review -code will be privately made available to reviewers through a comment approximately a week after submission).

4.1. PERFORMANCE COMPARISON:

Quantitative Results. Figure 3 shows the validation loss along the learning curves for the 2M parameter instantiation of each architecture. The relative ordering of the architectures is consistent after the initial phase of training, with all curves flattening out by 100 epochs.

First, the large jumps in the MLP training (due to unstable training runs) show that it was much less stable to training compared to the sample-based architectures. While effort was spent trying to specialize the training configuration for the MLP, stability remained an issue. 1 In contrast the runs for each sample based architecture were stable and similar. Ignoring the MLP stability, it performs similarly to worst performing sample-based architectureby the end of the learning curve.

The three sample based architectures show rapid progress early in learning and then settle into a consistent ordering with NoiseLearn performing best, followed by NoiseAppend, and then NoiseAdd. This suggests that NoiseAdd's approach of adding uniform noise to the latent representation may be obscuring information needed for accurate reconstruction, compared to NoiseAppend, which separates noise from the shape representation. On the other hand, we see that while NoiseLearn also adds noise to the latent representation, it is able to outperform NoiseAppend. This indicates the importance of being able to intelligently select how much noise to add to different components of the representation. Apparently, this allows NoiseLearn to avoid obscuring critical information in the latent representation needed for accurate reconstruction.

Figure 4 shows the average test set performance after 100 epochs of training of each size instantiation of the four architectures (note the log scale). The appendix also shows more detailed results broken down for each of 5 selected object classes. The three sample based architectures show relatively consistent improvement in performance as the sizes grow by orders of magnitude. Rather, the MLP shows initial improvement, but then performance decreases significantly past 100K parameters. We looked further into the behavior of the MLP architecture for the larger parameter sets. We observed that the larger MLPs showed a similar decrease in performance on even the training data, indicating that the problem is not necessarily overfitting but also difficulty of the optimization. It is possible that with substantially more epochs the MLP performance would improve, but at great cost. This indicates that the MLP architecture is much less efficient at exploiting larger network sizes than the more structured sample-based architectures. It is possible that the architecture and training hyperparameters could be tweaked to improve the large MLP networks' performance, such as by adding additional regularization via weight decay or other mechanisms. However, we consider this tweaking to be outside the scope of this work, and note that none of the sampling based architectures required any such tweaking to achieve competitive performance at all parameter counts. Overall the results give strong evidence that the sample-based architectures encode a bias that is much better matched to decoding the class of shapes in ModelNet40 compared to MLPs. The structured sample-based architectures, compared to the MLP, result in more stable learning and the ability to continually improve as the architectures grow in size. Further, we see that the NoiseLearn architecture, which avoids the need to specify hyperparameters to control the amount of noise performs the best, or near best, across all network sizes and number of epochs.

Illustrative Qualitative Results. Figure 5 shows each network's performance three test objects not seen in the training data. The point cloud decoded by the MLP network appears to be more evenly distributed spatially, while the sampling-based approaches are better able to capture finer detail in the target shape, such as the stool's thin legs and crossbars. Among the sample-based approaches, no single approach is clearly dominant in terms of visual quality across the objects. It is interesting that all of the sample-based architectures tend to miss the same type of object details, e.g. the jets on the plane or the leg cross bars on the chair, which may be due to limitations of the PointNet encoders sized and/or architecture. Nevertheless, it is quite interesting that a single relatively small latent vector representation is able to encode the level of detail exibited in these results.

4.2. NOISE EXAMINATION:

Each sampling architecture defines a function from the latent shape representation to a point feature distribution. The underlying latent representation inherently defines the manifold of the encoded shape. Rather, the injected noise (either via appending or addition) can be viewed as playing the role of indexing locations on the manifold for the generated point. Effectively, the primary difference between the sample-based architectures is how they use the noise to index locations and traverse the manifolds. Below we aim to better understand this relationship between noise and spatial indexing and how the architectures differ in that respect.

In Figure 6 we demonstrate how each architecture uses noise by controlling the variance introduced to a trained network in two different ways. To examine how the decoder's output is influenced by individual elements of noise we show the output of these networks when all but one of the noise elements is held at a constant near-zero value. In the lower plots, we show the decoder's behavior when it only receives the union of the noise elements above. This demonstrates both how the network learns to exploit individual elements of noise and how the decoder combines those elements to produce a point cloud that spans the entire shape.

For NoiseAppend all of the noise is of equal magnitude, so we just examine the first five elements of noise in its noise vector. NoiseLearn predicts individual variances for each element in the dense shape encoding, enabling us to select the five elements of noise with the highest variance, and therefore presumably the biggest contribution to the decoded point cloud. The appendix contains additional examples of noise manipulation.

The plots shown in Figure 6 give us some insight into how the networks use noise to complete the decoded shape. Each individual element of noise appears to correspond to a learned path along the surface of the learned shape. The final point cloud then seems to be produced by 'extruding' along those paths. NoiseLearn's use of only four significant elements of noise suggests that in this domain only three or four elements of noise is sufficient to achieve good coverage of the target shape. Figure 8 shows how individual noise channels change when the NoiseAppend architecture is modified to only append one, two, and three noise elements.

With only one element of noise, we can see that the network effectively has to learn a single path that spans as much of the target shape as possible. With two elements of noise, the network instead seems to learn individual 'loops' around the object which are transformed and rotated as necessary. Once the network has access to three elements of noise, we see the same behavior as the functional networks of learning small paths on the object's surface.

If too little noise can seriously hurt NoiseLearn's performance, does adding too much noise do the same? Figure 7 shows the NoiseAppend architecture trained with different amounts of added noise to see if the same performance dropoff is present at both extremes. It appears that even when the noise vector is much larger than the dense shape representation, the decoder's overall performance is not impacted. However, note that adding large amounts of noise does significantly increase the parameter count, so there is a nontrivial cost to doing this.

5. CONCLUSION:

In this work, we evaluated and compared several realizations of a sample-based point cloud decoder architecture. We show that these sampling approaches are competitive with or outperform the MLP approach while using fewer parameters and providing better functionality. These advantages over the baseline suggest that sample based point cloud decoders should be the default approach when a network needs to produce independent point samples of some underlying function or object. To further this this area of research, we provide a complete open-source implementation of our tools used to train and evaluate these networks. 

:

We attempted to rectify this by adding batch normalization to layers in the MLP decoder to make it more similar to the architecture of the sampling approaches. While this stabilizied training, it also prevented the MLP architecture from achieving competitive performance, as its loss more than doubled the next best architectures'.

Figures and Tables:

Figure 1: Diagrams demonstrating the operation of the PointNet style encoder network shared by all architectures we evaluate, and the parallel decoder network shared by all the sampling approaches.

Figure 2: Diagrams of the different approaches to deriving a distribution from the latent shape representation h.

Figure 3: Average validation losses of each architecture in the 2M family during training.

Figure 4: Test performance of each architecture at different parameter counts. Error bars show the 95% confidence interval of the mean.

Figure 5: Examples of networks' auto-encoding results on several previously unseen objects.

Figure 6: Behavior of NoiseAppend and NoiseLearn when the noise is manipulated. The ith figure in the top row shows the decoded point clouds when all but the ith variance is set to zero. The same figure in the bottom row shows the decoded point clouds when all but the first i variances are set to zero. The faint red points show the network's input.

Figure 8: Contribution of individual channels of noise when the NoiseAppend architecture is modified to only append N elements of noise to the shape representation.

Figure 7: Average final validation loss achieved by NoiseAppend networks when varying the number of elements of noise appended.

Detailed description of architectures evaluated. MLP is not present in the 10k family as it is cannot be reduced to that parameter count while still producing 1024 points. evaluated each decoding architecture by training several instantiations of each architecture on a point cloud auto-encoding problem derived from the ModelNet40 dataset, which consists of over 12,000 3D models of 40 different common object classes. The dataset has a prescribed train/test split, with approximately 9800 models in the training dataset and 2500 in the test dataset. We randomly select 10% of the training data to use for validation during training.

Table content: <table><tr><td/><td>Network Architecture</td><td>Encoder Hidden</td><td>Latent</td><td>Appended</td><td>V (h)</td><td>Decoder Hidden</td><td>Parameter Count</td></tr><tr><td/><td/><td/><td>Vector</td><td>Noise</td><td>Hidden</td><td/></tr><tr><td/><td>MLP</td><td>64, 64, 256, 256, 512</td><td>1024</td><td/><td/><td>1024, 512, 512, 512</td><td>4433600</td></tr><tr><td>4M</td><td>NoiseAppend NoiseAdd</td><td>64, 64, 256, 256, 512 64, 64, 256, 256, 512</td><td>1024 1024</td><td>32</td><td/><td>2048, 512, 512, 256, 256, 128 2048, 512, 512, 256, 256, 128</td><td>4468291 4402756</td></tr><tr><td/><td>NoiseLearn</td><td>64, 64, 256, 256, 512</td><td>1024</td><td/><td>128, 32</td><td>2048, 512, 512, 256, 256, 128</td><td>4573059</td></tr><tr><td/><td>MLP</td><td>64, 64, 256, 256, 512</td><td>256</td><td/><td/><td>256, 512, 512, 256</td><td>1739456</td></tr><tr><td>2M</td><td>NoiseAppend NoiseAdd</td><td>64, 64, 256, 256, 512 64, 64, 256, 256, 512</td><td>256 256</td><td>32</td><td/><td>256, 512, 512, 256 256, 512, 512, 256</td><td>1675843 1643076</td></tr><tr><td/><td>NoiseLearn</td><td>64, 64, 256, 256, 512</td><td>256</td><td/><td>128, 32</td><td>256, 512, 512, 256</td><td>1688963</td></tr><tr><td/><td>MLP</td><td>64, 128, 128</td><td>64</td><td/><td/><td>256, 256, 128</td><td>548032</td></tr><tr><td>500k</td><td>NoiseAppend NoiseAdd</td><td>64, 128, 128 64, 128, 128</td><td>64 64</td><td>16</td><td/><td>512, 512, 256, 128 512, 512, 256, 128</td><td>507459 499268</td></tr><tr><td/><td>NoiseLearn</td><td>64, 128, 128</td><td>64</td><td/><td>32</td><td>512, 512, 256, 128</td><td>503555</td></tr><tr><td/><td>MLP</td><td>64, 256</td><td>64</td><td/><td/><td>32</td><td>138048</td></tr><tr><td>100k</td><td>NoiseAppend NoiseAdd</td><td>64, 256 64, 256</td><td>64 64</td><td>16</td><td/><td>256, 128, 64 256, 128, 64</td><td>97923 93828</td></tr><tr><td/><td>NoiseLearn</td><td>64, 256</td><td>64</td><td/><td>32</td><td>256, 128, 64</td><td>98115</td></tr><tr><td/><td>NoiseAppend</td><td>64, 32</td><td>32</td><td>16</td><td/><td>64, 64, 16</td><td>12595</td></tr><tr><td>10k</td><td>NoiseAdd</td><td>64, 32</td><td>32</td><td/><td/><td>64, 64, 16</td><td>11572</td></tr><tr><td/><td>NoiseLearn</td><td>64, 32</td><td>32</td><td/><td>(none)</td><td>64, 64, 16</td><td>12659</td></tr><tr><td colspan="8">conserving the information necessary to accurately reconstruct it without introducing any additional</td></tr><tr><td colspan="2">hyperparameters.</td><td/><td/><td/><td/><td/></tr><tr><td colspan="2">4 EVALUATION</td><td/><td/><td/><td/><td/></tr><tr><td colspan="8">Before training, each object model in the ModelNet40 dataset is used to generate a uniformly-</td></tr><tr><td colspan="8">sampled point cloud with 4096 points which is then scaled to fit within the unit sphere. For all auto-</td></tr><tr><td colspan="8">encoder network models, at each iteration of training, the point clouds are randomly downsampled to</td></tr><tr><td colspan="8">1024 points before being used to update the network parameters. The helps reduce the computational</td></tr><tr><td colspan="8">cost of training and also encouraging better generalization. During training, each decoded point</td></tr><tr><td colspan="3">cloud consists of 1024 points.</td><td/><td/><td/><td/></tr><tr><td colspan="8">We use the Chamfer distance as the loss function due to its relative speed and capability to directly</td></tr><tr><td colspan="8">compare point clouds of unequal sizes without modification. Each network is trained for 100 epochs</td></tr></table>

Average loss of 2M family architectures (± stddev) on test dataset

Table content: <table><tr><td/><td>airplane</td><td>bathtub</td><td>bed</td><td>bench</td><td>bookshelf</td></tr><tr><td>MLP-2m</td><td>3.39 ± 1.89</td><td>5.67 ± 1.78</td><td>5.67 ± 1.41</td><td>5.05 ± 2.73</td><td>6.42 ± 2.40</td></tr><tr><td>NoiseAppend-2m</td><td>3.72 ± 0.88</td><td>6.10 ± 1.37</td><td>5.84 ± 1.35</td><td>5.10 ± 1.54</td><td>6.74 ± 1.62</td></tr><tr><td>NoiseAdd-2m</td><td>4.08 ± 1.10</td><td>6.23 ± 1.50</td><td>6.05 ± 1.42</td><td>5.36 ± 1.72</td><td>6.90 ± 1.78</td></tr><tr><td>NoiseLearn-2m</td><td>2.97 ± 0.73</td><td>5.31 ± 1.30</td><td>5.15 ± 1.28</td><td>3.87 ± 1.27</td><td>5.95 ± 1.51</td></tr><tr><td/><td>bottle</td><td>bowl</td><td>car</td><td>chair</td><td>cone</td></tr><tr><td>MLP-2m</td><td>2.69 ± 1.49</td><td>6.95 ± 3.26</td><td>5.96 ± 1.28</td><td>7.04 ± 2.34</td><td>7.02 ± 49.08</td></tr><tr><td>NoiseAppend-2m</td><td>4.13 ± 1.40</td><td>10.10 ± 4.46</td><td>6.34 ± 1.08</td><td>6.78 ± 2.25</td><td>5.22 ± 2.62</td></tr><tr><td>NoiseAdd-2m</td><td>4.03 ± 1.70</td><td>9.43 ± 4.23</td><td>6.46 ± 1.15</td><td>7.29 ± 2.42</td><td>5.21 ± 2.86</td></tr><tr><td>NoiseLearn-2m</td><td>3.35 ± 1.24</td><td>7.46 ± 3.27</td><td>5.83 ± 1.04</td><td>5.15 ± 1.85</td><td>4.32 ± 2.09</td></tr><tr><td/><td>cup</td><td>curtain</td><td>desk</td><td>door</td><td>dresser</td></tr><tr><td>MLP-2m</td><td>7.77 ± 2.84</td><td>2.90 ± 1.55</td><td>10.78 ± 4.04</td><td>2.64 ± 1.88</td><td>5.95 ± 4.77</td></tr><tr><td>NoiseAppend-2m</td><td>8.67 ± 2.83</td><td>3.68 ± 1.04</td><td>7.53 ± 1.97</td><td>3.18 ± 0.82</td><td>7.13 ± 1.74</td></tr><tr><td>NoiseAdd-2m</td><td>9.08 ± 2.83</td><td>3.53 ± 1.24</td><td>8.14 ± 2.20</td><td>3.13 ± 1.05</td><td>7.24 ± 1.77</td></tr><tr><td>NoiseLearn-2m</td><td>7.63 ± 2.70</td><td>2.85 ± 0.88</td><td>6.00 ± 1.70</td><td>2.52 ± 0.61</td><td>6.32 ± 1.48</td></tr><tr><td/><td>flower pot</td><td>glass box</td><td>guitar</td><td>keyboard</td><td>lamp</td></tr><tr><td>MLP-2m</td><td>9.15 ± 4.87</td><td>5.81 ± 1.70</td><td>2.65 ± 0.89</td><td>2.18 ± 0.49</td><td>13.41 ± 31.22</td></tr><tr><td>NoiseAppend-2m</td><td>9.83 ± 3.96</td><td>7.16 ± 1.79</td><td>2.17 ± 0.72</td><td>2.75 ± 0.68</td><td>8.87 ± 11.19</td></tr><tr><td>NoiseAdd-2m</td><td>10.02 ± 3.95</td><td>7.39 ± 1.94</td><td>2.05 ± 0.74</td><td>2.69 ± 0.66</td><td>8.41 ± 7.15</td></tr><tr><td>NoiseLearn-2m</td><td>8.07 ± 3.35</td><td>6.59 ± 1.63</td><td>1.45 ± 0.53</td><td>2.26 ± 0.50</td><td>6.49 ± 7.05</td></tr><tr><td/><td>laptop</td><td>mantel</td><td>monitor</td><td>night stand</td><td>person</td></tr><tr><td>MLP-2m</td><td>3.79 ± 0.65</td><td>7.29 ± 2.16</td><td>6.51 ± 3.35</td><td>7.26 ± 2.43</td><td>5.55 ± 2.85</td></tr><tr><td>NoiseAppend-2m</td><td>4.57 ± 0.85</td><td>6.65 ± 1.41</td><td>6.36 ± 1.97</td><td>7.53 ± 1.96</td><td>5.21 ± 1.68</td></tr><tr><td>NoiseAdd-2m</td><td>4.73 ± 0.98</td><td>6.78 ± 1.38</td><td>6.53 ± 2.08</td><td>7.67 ± 1.94</td><td>5.08 ± 2.00</td></tr><tr><td>NoiseLearn-2m</td><td>3.25 ± 0.50</td><td>5.59 ± 1.18</td><td>5.33 ± 1.75</td><td>6.35 ± 1.80</td><td>3.87 ± 1.38</td></tr><tr><td/><td>piano</td><td>plant</td><td>radio</td><td>range hood</td><td>sink</td></tr><tr><td>MLP-2m</td><td>10.70 ± 2.85</td><td>10.06 ± 5.18</td><td>5.73 ± 1.57</td><td>7.37 ± 2.84</td><td>9.16 ± 4.50</td></tr><tr><td>NoiseAppend-2m</td><td>8.67 ± 2.10</td><td>9.87 ± 4.82</td><td>5.94 ± 1.90</td><td>6.83 ± 1.53</td><td>6.85 ± 2.09</td></tr><tr><td>NoiseAdd-2m</td><td>8.82 ± 2.15</td><td>10.45 ± 5.21</td><td>6.06 ± 1.87</td><td>7.03 ± 1.53</td><td>7.05 ± 2.18</td></tr><tr><td>NoiseLearn-2m</td><td>7.22 ± 1.77</td><td>8.38 ± 4.20</td><td>5.12 ± 1.83</td><td>5.76 ± 1.25</td><td>5.77 ± 1.77</td></tr><tr><td/><td>sofa</td><td>stairs</td><td>stool</td><td>table</td><td>tent</td></tr><tr><td>MLP-2m</td><td>6.47 ± 1.73</td><td>11.44 ± 6.03</td><td>7.00 ± 3.15</td><td>4.56 ± 1.90</td><td>7.42 ± 2.47</td></tr><tr><td>NoiseAppend-2m</td><td>6.60 ± 1.62</td><td>8.16 ± 3.29</td><td>6.08 ± 2.35</td><td>4.19 ± 1.32</td><td>7.17 ± 1.74</td></tr><tr><td>NoiseAdd-2m</td><td>6.79 ± 1.68</td><td>8.37 ± 3.52</td><td>6.78 ± 2.85</td><td>4.48 ± 1.59</td><td>7.16 ± 1.84</td></tr><tr><td>NoiseLearn-2m</td><td>5.84 ± 1.39</td><td>6.18 ± 2.46</td><td>4.75 ± 1.71</td><td>3.15 ± 1.08</td><td>6.05 ± 1.72</td></tr><tr><td/><td>toilet</td><td>tv stand</td><td>vase</td><td>wardrobe</td><td>xbox</td></tr><tr><td>MLP-2m</td><td>7.92 ± 2.57</td><td>6.92 ± 2.36</td><td>5.92 ± 3.49</td><td>5.53 ± 2.49</td><td>6.12 ± 3.03</td></tr><tr><td>NoiseAppend-2m</td><td>8.21 ± 1.81</td><td>6.79 ± 1.78</td><td>7.60 ± 3.75</td><td>6.24 ± 1.24</td><td>6.59 ± 1.46</td></tr><tr><td>NoiseAdd-2m</td><td>8.51 ± 1.89</td><td>7.01 ± 1.84</td><td>7.73 ± 4.62</td><td>6.42 ± 1.38</td><td>6.67 ± 1.47</td></tr><tr><td>NoiseLearn-2m</td><td>6.83 ± 1.54</td><td>6.01 ± 1.66</td><td>6.25 ± 2.97</td><td>5.54 ± 1.13</td><td>5.75 ± 0.92</td></tr></table>

Average loss of 100k family architectures (± stddev) on test dataset

Table content: <table><tr><td>airplane</td><td>bathtub</td><td>bed</td><td>bench</td><td>bookshelf</td></tr></table>

Average loss of 4M family architectures (± stddev) on test dataset

Table content: <table><tr><td/><td>airplane</td><td>bathtub</td><td>bed</td><td>bench</td><td>bookshelf</td></tr><tr><td>MLP-4m</td><td>8.45 ± 2.38</td><td>10.04 ± 2.81</td><td>9.42 ± 2.72</td><td>10.31 ± 4.79</td><td>9.19 ± 3.94</td></tr><tr><td>NoiseAppend-4m</td><td>3.68 ± 0.90</td><td>6.06 ± 1.42</td><td>5.84 ± 1.37</td><td>5.14 ± 1.45</td><td>6.63 ± 1.59</td></tr><tr><td>NoiseAdd-4m</td><td>4.07 ± 1.11</td><td>6.27 ± 1.53</td><td>6.11 ± 1.43</td><td>5.58 ± 1.79</td><td>6.89 ± 1.79</td></tr><tr><td>NoiseLearn-4m</td><td>2.95 ± 0.72</td><td>5.30 ± 1.29</td><td>5.11 ± 1.26</td><td>3.81 ± 1.26</td><td>5.82 ± 1.54</td></tr><tr><td/><td>bottle</td><td>bowl</td><td>car</td><td>chair</td><td>cone</td></tr><tr><td>MLP-4m</td><td>4.59 ± 2.58</td><td>14.06 ± 4.90</td><td>9.01 ± 2.77</td><td>13.51 ± 5.09</td><td>9.45 ± 6.15</td></tr><tr><td>NoiseAppend-4m</td><td>3.93 ± 1.40</td><td>9.61 ± 3.89</td><td>6.39 ± 1.12</td><td>6.62 ± 2.18</td><td>5.32 ± 2.70</td></tr><tr><td>NoiseAdd-4m</td><td>4.03 ± 1.62</td><td>9.49 ± 4.40</td><td>6.44 ± 1.17</td><td>7.39 ± 2.46</td><td>5.21 ± 2.56</td></tr><tr><td>NoiseLearn-4m</td><td>3.22 ± 1.19</td><td>7.39 ± 3.09</td><td>5.86 ± 1.04</td><td>5.08 ± 1.82</td><td>4.28 ± 2.02</td></tr><tr><td/><td>cup</td><td>curtain</td><td>desk</td><td>door</td><td>dresser</td></tr><tr><td>MLP-4m</td><td>12.07 ± 3.68</td><td>5.65 ± 5.26</td><td>15.68 ± 6.05</td><td>5.09 ± 4.61</td><td>10.15 ± 3.54</td></tr><tr><td>NoiseAppend-4m</td><td>8.58 ± 2.93</td><td>3.72 ± 1.01</td><td>7.64 ± 2.00</td><td>3.18 ± 0.81</td><td>6.96 ± 1.68</td></tr><tr><td>NoiseAdd-4m</td><td>9.21 ± 2.85</td><td>3.64 ± 1.26</td><td>8.35 ± 2.27</td><td>3.06 ± 0.82</td><td>7.21 ± 1.80</td></tr><tr><td>NoiseLearn-4m</td><td>7.50 ± 2.68</td><td>2.81 ± 0.89</td><td>5.90 ± 1.64</td><td>2.49 ± 0.63</td><td>6.27 ± 1.48</td></tr><tr><td/><td>flower pot</td><td>glass box</td><td>guitar</td><td>keyboard</td><td>lamp</td></tr><tr><td>MLP-4m</td><td>12.45 ± 5.96</td><td>10.98 ± 3.73</td><td>5.44 ± 2.36</td><td>4.24 ± 1.74</td><td>24.83 ± 42.31</td></tr><tr><td>NoiseAppend-4m</td><td>9.38 ± 3.53</td><td>7.07 ± 1.79</td><td>2.19 ± 0.69</td><td>2.74 ± 0.62</td><td>9.36 ± 12.32</td></tr><tr><td>NoiseAdd-4m</td><td>10.15 ± 3.88</td><td>7.56 ± 2.10</td><td>2.11 ± 0.77</td><td>2.69 ± 0.63</td><td>8.15 ± 5.93</td></tr><tr><td>NoiseLearn-4m</td><td>7.87 ± 3.23</td><td>6.53 ± 1.61</td><td>1.45 ± 0.51</td><td>2.21 ± 0.48</td><td>6.28 ± 6.67</td></tr><tr><td/><td>laptop</td><td>mantel</td><td>monitor</td><td>night stand</td><td>person</td></tr><tr><td>MLP-4m</td><td>13.72 ± 5.49</td><td>13.97 ± 5.53</td><td>11.00 ± 6.65</td><td>12.64 ± 4.27</td><td>7.30 ± 3.16</td></tr><tr><td>NoiseAppend-4m</td><td>4.62 ± 0.96</td><td>6.64 ± 1.40</td><td>6.29 ± 2.00</td><td>7.36 ± 1.93</td><td>5.06 ± 1.50</td></tr><tr><td>NoiseAdd-4m</td><td>5.20 ± 1.26</td><td>6.83 ± 1.44</td><td>6.64 ± 2.12</td><td>7.65 ± 2.02</td><td>5.37 ± 1.96</td></tr><tr><td>NoiseLearn-4m</td><td>3.27 ± 0.53</td><td>5.57 ± 1.18</td><td>5.19 ± 1.72</td><td>6.24 ± 1.77</td><td>3.74 ± 1.31</td></tr><tr><td/><td>piano</td><td>plant</td><td>radio</td><td>range hood</td><td>sink</td></tr><tr><td>MLP-4m</td><td>14.19 ± 4.32</td><td>13.97 ± 7.85</td><td>10.28 ± 3.85</td><td>13.17 ± 4.07</td><td>14.79 ± 5.38</td></tr><tr><td>NoiseAppend-4m</td><td>8.49 ± 2.00</td><td>9.77 ± 4.76</td><td>5.85 ± 1.81</td><td>6.68 ± 1.43</td><td>6.70 ± 2.11</td></tr><tr><td>NoiseAdd-4m</td><td>8.93 ± 2.18</td><td>10.68 ± 5.26</td><td>6.09 ± 1.99</td><td>7.01 ± 1.57</td><td>7.03 ± 2.17</td></tr><tr><td>NoiseLearn-4m</td><td>7.09 ± 1.74</td><td>8.17 ± 4.09</td><td>5.06 ± 1.83</td><td>5.75 ± 1.23</td><td>5.67 ± 1.84</td></tr><tr><td/><td>sofa</td><td>stairs</td><td>stool</td><td>table</td><td>tent</td></tr><tr><td>MLP-4m</td><td>10.07 ± 3.23</td><td>19.81 ± 8.58</td><td>12.93 ± 4.80</td><td>10.41 ± 4.93</td><td>11.56 ± 3.88</td></tr><tr><td>NoiseAppend-4m</td><td>6.56 ± 1.59</td><td>8.11 ± 3.08</td><td>6.00 ± 2.33</td><td>4.22 ± 1.35</td><td>7.03 ± 1.74</td></tr><tr><td>NoiseAdd-4m</td><td>6.81 ± 1.72</td><td>8.60 ± 3.54</td><td>6.94 ± 3.35</td><td>4.71 ± 1.64</td><td>7.24 ± 1.99</td></tr><tr><td>NoiseLearn-4m</td><td>5.77 ± 1.37</td><td>6.20 ± 2.42</td><td>4.46 ± 1.37</td><td>3.16 ± 1.08</td><td>5.97 ± 1.58</td></tr><tr><td/><td>toilet</td><td>tv stand</td><td>vase</td><td>wardrobe</td><td>xbox</td></tr><tr><td>MLP-4m</td><td>16.68 ± 5.08</td><td>10.99 ± 4.61</td><td>10.03 ± 6.06</td><td>9.21 ± 4.15</td><td>9.51 ± 5.13</td></tr><tr><td>NoiseAppend-4m</td><td>7.97 ± 1.73</td><td>6.79 ± 1.74</td><td>7.28 ± 3.46</td><td>6.19 ± 1.21</td><td>6.35 ± 1.34</td></tr><tr><td>NoiseAdd-4m</td><td>8.62 ± 1.88</td><td>7.08 ± 1.94</td><td>8.33 ± 21.51</td><td>6.42 ± 1.37</td><td>6.58 ± 1.50</td></tr><tr><td>NoiseLearn-4m</td><td>6.68 ± 1.49</td><td>5.94 ± 1.63</td><td>6.15 ± 2.90</td><td>5.45 ± 1.16</td><td>5.64 ± 0.88</td></tr></table>

Peer review (iclr2020):
Summary This paper studies the problem of 3D point cloud autoencoding in a deep learning setup, and in particular, the choice of the architecture of a 3D point cloud decoder. The main claim of the paper is the adaptation of the PointNet architecture (a set of fully-connected layers applied per point) from encoding to decoding purposes. To achieve that, the authors propose three similar mechanisms to inject noise in the bottleneck shape representation, which are used to sample per point feature vectors suitable for a PointNet-like decoder. These include appending a low dimensional gaussian noise sample to the bottleneck shape representation, adding individual gaussian noise sample to each bottleneck shape representation dimension or using trainable variances per bottleneck shape representation dimension. Resulting networks are optimized by minimizing Chamfer distance between the ground truth and output point clouds. The main motivation of their choice of these architectures is their ability to output an arbitrary size point clouds, compared to naive multilayered fully-connected network applied to a bottleneck shape representation and outputting a *flattened* fixed-size point cloud. To verify their claims, the authors considered autoencoding task for ModelNet40 dataset. They compared a naive multilayered fully-connected decoder to PointNet-like decoders with all three proposed noise-injection mechanisms. This comparison was also performed for 5 variations of the proposed models differing is network size. To compare quantitatively, the authors present all results graphically, showing the dependency of resulting average Chamfer distance from the network size. This graph shows that using trainable per dimension variance for noise injection works consistently better, than other options; also, it shows inconsistent results for the baseline multilayered fully-connected decoder. In qualitative comparisons, the authors compare output point clouds visually and study the influence of sampling from various noise dimensions on the reconstructed points. Review Although the authors of this paper touch a very interesting topic in generative modeling of point clouds, I do not believe this paper should be published in its current state. Cons: 1. The main issue in this paper is the absence of comparisons to any external approaches. Even though comparison to their variation of the baseline model is provided, it is impossible to verify where their results stand compared to the related work. There is a whole stream of works in the same task using ShapeNet dataset, which the authors completely omit. Even if the authors do not want to perform experiments on ShapeNet, they, at least, could have provided comparison to AAE and PC-GAN results from PC-GAN paper [1] for generative modeling metrics on ModelNet40. As the authors stated, there are at least two works, PC-GAN and PointFlow, which allow sampling of the arbitrary size point clouds and comparison to these approaches is crucial (especially since the proposed approach is technically significantly simpler compared to hierarchical GANs from PC-GAN and continuous normalizing flows combined with VAEs from PointFlow papers). 2. The paper misses numerous citations to related work in generative 3D shape modeling: - voxel grid based, (since the authors mention them): [2, 3, 4, 5, 6, 7] - point cloud based: [8, 9, 10] - graph based: [11, 12] All these approaches propose various types of decoders for 3D shapes represented accordingly and some of them perform evaluations for pure generative, autoencoding task, which, again, can be used for comparison. 3. I believe, there is a factual mistake and arguable diminishing of the current state of the art in the related work section: The authors state, that both PC-GAN and PointFlow models *inhabit GAN settings*. However, PointFlow is, in fact, an approximate likelihood model and is considered as an instance of Variational Autoencoder with a shape prior and a point cloud decoder implemented as continuous normalizing flows (a special case of invertible neural networks). The authors also state, that it is *unclear... if they are capable of outperforming the baseline MLP approach* when they talk about these papers, while, in fact, both PC-GAN and PointFlow explicitly show that by comparing to [9], which use the multilayered fully-connected baseline decoder. It is also not unclear if these approaches need specific loss functions, contrary to what authors say. They were designed so they need their particular loss functions, which does not disqualify them from the comparison. These papers are the main competitors of this work and should be addressed properly instead of being diminished. Pros: 1. Besides the cons, the paper is written clearly and is easy to follow. 2. Rigorousness with respect to comparison of the proposed options (Table 1 and the fact that every model was retrained 15 times to obtain average result). 3. Qualitative experiments with control over noise dimensions are novel and interesting. Overall, this could have been a paper about a nice and simple alternative to PC-GAN and PointFlow for arbitrary size point cloud modeling, but, unfortunately, it missed the opportunity to properly address the current state-of-the-art in its current version. Small remarks and questions for authors: 1. Citation of Valsesia at al. in the related work should be reorganized to be consistent with other citations. Omitting full names there and in case of PC-GAN will also be more consistent. 2. All figures in the paper should be updated for better visibility of both graphics and text. Figures 1-4, 7 - text is barely visible, the scheme blocks and lines are too thin and overall lack readability. Figures 5-6, 8 - point clouds are almost invisible even with zooming, remove coordinate grid in the background, zoom in on the objects, so they take as much space in the images as possible, use bigger point sizes for better visibility. 3. Maybe I do not follow something, why have you ignored applications of your point cloud decoders, and particularly generative modeling task and its evaluation protocols, presented for example in [9]? [1] C.-L. Li, M. Zaheer, Y. Zhang, B. Póczos and R. Salakhutdinov. Point Cloud GAN. In ICLR Workshop on Deep Generative Models for Highly Structured Data, 2019 [2] C. Choy, D. Xu, J.-Y. Gwak, K. Chen, and S. Savarese. 3D-R2N2: A unified approach for single and multi-view 3d object reconstruction. In ECCV, 2016. [3] R. Girdhar, D. Fouhey, M. Rodriguez, and A. Gupta. Learning a predictable and gen- erative vector representation for objects. In ECCV, 2016. [4] J. Wu, C. Zhang, T. Xue, W. Freeman, and J. Tenenbaum. Learning a probabilistic latent space of object shapes via 3D generative-adversarial modeling. In NeurIPS, 2016. [5] M. Tatarchenko, A. Dosovitskiy, and T. Brox. Octree generating networks: Efficient convolutional architectures for high-resolution 3D outputs. In ICCV, 2017. [6] S. R. Richter and S. Roth. Matryoshka Networks: Predicting 3d geometry via nested shape layers. In CVPR, 2018. [7] R. Klokov, J. Verbeek, and E. Boyer. Probabilistic Reconstruction Networks for 3D Shape Inference from a Single Image. In BMVC, 2019 [8] T. Groueix, M. Fisher, V. Kim, B. Russell, and M. Aubry. AtlasNet: A papier-mâché approach to learning 3D surface generation. In CVPR, 2018. [9] P. Achlioptas, O. Diamanti, I. Mitliagkas, and L. Guibas. Learning Representations and Generative Models for 3D Point Clouds. In ICML, 2018. [10] P. Mandikal, K. Navaneet, M. Agarwal, and R. Babu. 3D-LMNet: Latent embedding matching for accurate and diverse 3D point cloud reconstruction from a single image. In BMVC, 2018. [11] N. Wang, Y. Zhang, Z. Li, Y. Fu, W. Liu, and Y. Jiang. Pixel2Mesh: Generating 3D mesh models from single RGB images. In ECCV, 2018. [12] C. Wen, Y. Zhang, Z. Li, and Y. Fu. Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformation. In ICCV, 2019.