Title: POINTGROW: AUTOREGRESSIVELY LEARNED POINT CLOUD GENERATION WITH SELF-ATTENTION

Abstract: A point cloud is an agile 3D representation, efficiently modeling an object's surface geometry. However, these surface-centric properties also pose challenges on designing tools to recognize and synthesize point clouds. This work presents a novel autoregressive model, PointGrow, which generates realistic point cloud samples from scratch or conditioned from given semantic contexts. Our model operates recurrently, with each point sampled according to a conditional distribution given its previously-generated points. Since point cloud object shapes are typically encoded by long-range interpoint dependencies, we augment our model with dedicated self-attention modules to capture these relations. Extensive evaluation demonstrates that PointGrow achieves satisfying performance on both unconditional and conditional point cloud generation tasks, with respect to fidelity, diversity and semantic preservation. Further, conditional PointGrow learns a smooth manifold of given images where 3D shape interpolation and arithmetic calculation can be performed inside.

1. INTRODUCTION:

3D visual understanding (Bogo et al. (2016) ; Zuffi et al. (2018) ) is at the core of next-generation vision systems. Specifically, point clouds, agile 3D representations, have emerged as indispensable sensory data in applications including indoor navigation (Díaz Vilariño et al. (2016) ), immersive technology (Stets et al. (2017) ; Sun et al. (2018a) ) and autonomous driving (Yue et al. (2018) ). There is growing interest in integrating deep learning into point cloud processing (Klokov & Lempitsky (2017) ; Lin et al. (2017a) ; Achlioptas et al. (2017b) ; Kurenkov et al. (2018) ; Yu et al. (2018) ; Xie et al. (2018) ). With the expressive power brought by modern deep models, unprecedented accuracy has been achieved on high-level point cloud related tasks including classification, detection and segmentation (Qi et al. (2017a; b) ; Wang et al. (2018) ; Li et al. (2018) ; You et al. (2018) ). Yet, existing point cloud research focuses primarily on developing effective discriminative models (Xie et al. (2018) ; Shen et al. (2018) ), rather than generative models. This paper investigates the synthesis and processing of point clouds, presenting a novel generative model called PointGrow. We propose an autoregressive architecture (Oord et al. (2016) ; Van Den Oord et al. (2016) ) to accommodate the surface-centric nature of point clouds, generating every single point recurrently. Within each step, PointGrow estimates a conditional distribution of the point under consideration given all its preceding points, as illustrated in Figure 1 . This approach easily handles the irregularity of point clouds, and encodes diverse local structures relative to point distance-based methods (Fan et al. (2017) ; Achlioptas et al. (2017b) ).

However, to generate realistic point cloud samples, we also need long-range part configurations to be plausible. We therefore introduce two self-attention modules (Lin et al. (2017b) ; Velickovic et al. (2017) ; Zhang et al. (2018) ) in the context of point cloud to capture these long-range relations. Each dedicated self-attention module learns to dynamically aggregate long-range information during the point generation process. In addition, our conditional PointGrow learns a smooth manifold of given images where interpolation and arithmetic calculation can be performed on image embeddings.

Compared to prior art, PointGrow has appealing properties::

• Unlike traditional 3D generative models that rely on local regularities on grids (Wu et al. (2016) and then samples a value (indicated as a red bar) according to it. The process is repeated to sample y i and x i with previously sampled coordinates as additional conditions. The i th point (red point in the last column) is obtained as

{x i , y i , z i }. When i = 1, p(z i |s ≤i-1 ) = p(z i ).

autoregressive architecture that is inherently suitable for modeling point clouds, which are irregular and surface-centric. • Our proposed self-attention module successfully captures the long-range dependencies between points, helping to generate plausible part configurations within 3D objects. • PointGrow, as a generative model, enables effective unsupervised feature learning, which is useful for recognition tasks, especially in the low-data regime.

Extensive evaluations demonstrate that PointGrow can generate realistic and diverse point cloud samples with high resolution, on both unconditional and conditional point cloud generation tasks.

2. POINTGROW:

In this section, we introduce the formulation and implementation of PointGrow, a new generative model for point cloud, which generates 3D shapes in a point-by-point manner.

Unconditional PointGrow. A point cloud, S, that consists of n points is defined as S = {s 1 , s 2 , ..., s n }, and the i th point is expressed as s i = {x i , y i , z i } in 3D space. Our goal is to assign a probability p(S) to each point cloud. We do so by factorizing the joint probability of S as a product of conditional probabilities over all its points:

EQUATION

The value p(s i |s ≤i-1 ) is the probability of the i th point s i given all its previous points, and can be computed as the joint probability over its coordinates:

EQUATION

, where each coordinate is conditioned on all the previously generated coordinates. To facilitate the point cloud generation process, we sort points in the order of z, y and x, which forces a shape to be generated in a "plane-sweep" manner along its primary axis (z axis). Following Oord et al. (2016) and Van Den Oord et al. (2016) , we model the conditional probability distribution of each coordinate using a deep neural network. Prior art shows that a softmax discrete distribution works 2018), a global feature is obtained by applying max pooling along each feature dimension, and then used to provide context information for solving semantic segmentation tasks. Similarly, we obtain "semi-global" features for all sets of available points in the point cloud generation process, as illustrated in Figure 2 (left). Each row of the resultant features aggregates the context information of all the previously generated points dynamically by fetching and averaging. This Context Awareness (CA) operation is implemented as a plug-in module in our model, and mean pooling is used in our experiments.

Self-Attention Context Awareness Operation. The CA operation summarizes point features in a fixed way via pooling. For the same purpose, we propose two alternative learning-based operations to determine the weights for aggregating point features. We define them as Self-Attention Context Awareness (SACA) operations, and the weights as self-attention weights.

The first SACA operation, SACA-A, is shown in the middle of Figure 2 . To generate self-attention weights, the SACA-A first associates local and "semi-global" information by concatenating input and "semi-global" features after CA operation, and then passes them to a Multi-Layer Perception (MLP). Formally, given a n × f point feature matrix, F, with its i th row, f i , representing the feature vector of the i th point for 1 ≤ i ≤ n, we compute the i th self-attention weight vector, w i , as below: , where M ean{•} is mean pooling, ⊕ is concatenation, and M LP (•) is a sequence of fully connected layers. The self-attention weight encodes information about the context change due to each newly generated point, and is unique to that point. We then conduct element-wise multiplication between input point features and self-attention weights to obtain weighted features, which are accumulated sequentially to generate corresponding context features. The process to calculate the i th context feature, c i , can be expressed as:

EQUATION

EQUATION

, where ⊗ is element-wise multiplication. Finally, we shift context features downward by one row, because when estimating the coordinate distribution for point, s i , only its previous points, s ≤i-1 , are available. A zero vector of the same size is attached to the beginning as the initial context feature, since no previous point exists when computing features for the first point.

Figure 2 (right) shows the other SACA operation, SACA-B. SACA-B is similar to SACA-A, except the way to compute and apply self-attention weights. In SACA-B, the i th "semi-global" feature after CA operation is shared by the first i point features to obtain self-attention weights, which are then used to compute c i . This process can be described mathematically as:

EQUATION

Compared to SACA-A, SACA-B self-attention weights encode the importance of each point feature under a common context, as highlighted in Eq. ( 4) and ( 5).

Learning happens only in MLP for both operations. In Figure 3 , we plot the attention maps, which visualize Euclidean distances between the context feature of a selected point and the point features of its accessible points before SACA operation.

Model Architecture. Figure 4 shows the proposed network model to output conditional coordinate distributions. The top, middle and bottom branches model p(z i |s ≤i-1 ), p(y i |s ≤i-1 , z i ) and p(x i |s ≤i-1 , z i , y i ), respectively, for i = 1, ..., n. The point coordinates are sampled according to the estimated softmax probability distributions. Note that the input points in the latter two cases are masked accordingly so that the network cannot see information that has not been generated. During the training phase, points are available to compute all the context features, thus coordinate distributions can be estimated in parallel. However, the point cloud generation is a sequential procedure, since each sampled coordinate needs to be fed as input back into the network, as demonstrated in Figure 1 .

Conditional PointGrow. Given a condition or embedding vector, h, we hope to generate a shape satisfying the latent meaning of h. To achieve this, Eq. ( 1) and ( 2) are adapted to Eq. ( 6) and ( 7), respectively, as below: From left to right for each set: table, car, airplane, chair and lamp.

EQUATION

EQUATION

The additional condition, h, affects the coordinate distributions by adding biases in the generative process. We implement this by changing the operation between adjacent fully-connected layers from

x i+1 = f (Wx i ) to x i+1 = f (Wx i + Hh)

, where x i+1 and x i are feature vectors in the i + 1 th and i th layer, respectively, W is a weight matrix, H is a matrix that transforms h into a vector with the same dimension as Wx i , and f (•) is a nonlinear activation function. In this paper, we experimented with h as an one-hot categorical vector which adds class dependent bias, and an high-dimensional embedding vector of a 2D image which adds geometric constraint.

3. EXPERIMENTS:

Datasets. We evaluated the proposed framework on ShapeNet dataset (Chang et al. ( 2015)), which is a collection of CAD models. We used a subset consisting of 17,687 models across 7 categories.

To generate corresponding point clouds, we first sample 10,000 points uniformly from each mesh, and then use farthest point sampling to select 1,024 points among them representing the shape. Each category follows a split ratio of 0.9/0.1 to separate training and testing sets. ModelNet40 (Wu et al. (2015) ) and PASCAL3D+ (Xiang et al. (2014) ), are also used for further analysis. ModelNet40 contains CAD models from 40 categories, and we obtain their point clouds from Qi et al. (2017a). PASCAL3D+ is composed of PASCAL 2012 detection images augmented with 3D CAD model alignment, and used to demonstrate the generalization ability of conditional PointGrow.

3.1. UNCONDITIONAL POINT CLOUD GENERATION:

Figure 5 shows point clouds generated by unconditional PointGrow. Since an unconditional model lacks knowledge about the shape category to generate, we train separate models for each category. Figure 1 demonstrates point cloud generation for an airplane category. Note that no semantic information of discrete coordinates (i.e.scattered points in point cloud) is provided during training, but the predicted distribution turns out to be categorically representative. (e.g.in the second row, the network model outputs a roughly symmetric distribution along X axis, which describes the wings' shape of an airplane.) The autoregressive architecture in PointGrow is capable of abstracting highlevel semantics even from unaligned point cloud samples.

Evaluation on Fidelity and Diversity. 68.2 LFD (Chen et al. (2003)) 75.2 T-L Network (Girdhar et al. (2016)) 74.4 VConv-DAE (Sharma et al. (2016)) 75.5 3D-GAN (Wu et al. (2016)) 83.3 Latent-GAN-EMD (Achlioptas et al. (2017a)) 84.0 Latent-GAN-CD (Achlioptas et al. (2017a)) 84.5 Ours (SACA-A)

85.8 Ours (SACA-B) 84.4

To faithfully evaluate the generation quality, we conduct user study w.r.t.two aspects, fidelity and diversity, among CA-Max, CA-Mean and PointGrow (implemented with SACA-A). We randomly select 10 generated airplane and car point clouds from each method. To calculate the fidelity score, we ask the user to score 0, 0.5 or 1.0 for each shape, and take the average of them. The diversity score is obtained by asking the user to scale from 0.1 to 1.0 with an interval of 0.1 about the generated shape diversity within each method. 8 subjects without computer vision background participated in this test. We observe that (1) CA-Mean is more favored than CA-Max, and (2) our PointGrow receives the highest preference on both fidelity and diversity. 1 . Note that the CA-Mean baseline achieves comparable performance against both GAN-based competing methods. In the first classification task, our SACA-A model outperforms existing models by a relatively large margin, while in the second task, SACA-A and SACA-B models show similar performance.

Unsupervised Feature Learning. We next evaluate the learned point feature representations of the proposed framework, using them as features for classification. We obtain the feature representation of a shape by applying different types of "symmetric" functions as illustrated in Qi et al. (2017a) (i.e. min, max and mean pooling) on features of each layer before the SACA operation, and concatenate them all. Following Wu et al. (2016) , we first pre-train our model on 7 categories from the ShapeNet dataset, and then use the model to extract feature vectors for both training and testing shapes of ModelNet40 dataset. A linear SVM is used for feature classification. We report our best results in Table 2 . SACA-A model achieves the best performance, and SACA-B model performs slightly worse than Latent-GAN-CD (Achlioptas et al. (2017a) uses 57,000 models from 55 categories of ShapeNet dataset for pre-training, while we use 17,687 models across 7 categories).

Shape Completion. Our model can also perform shape completion. Given an initial set of points, our model is capable of completing shapes multiple ways. Figure 7 visualizes example predictions. The input points are sampled from ShapeNet testing sets, which are not seen during the training process. The shapes generated by our model are different from the original ground truth point clouds, but look plausible. A current limitation of our model is that it works only when the input point set is given as the beginning part of a shape along its primary axis, since our model is designed and trained to generate point clouds along that direction. More investigation is required to complete shapes when partial point clouds are given from other directions.

3.2. CONDITIONAL POINT CLOUD GENERATION:

SACA-A is used to demonstrate conditional PointGrow here owing to its high performance.

Conditioned on Category Label. We first experiment with class-conditional modelling of point clouds, given an one-hot vector h with its nonzero element h i indicating the i th shape category.

The one-hot condition provides categorical knowledge to guide the shape generation process. We train the proposed model across multiple categories, and plausible point clouds for desired shape categories can be sampled (as shown in Figure 8 ). Failure cases are also observed: generated shapes present interwoven geometric properties from other shape types. For example, the airplane misses wings and generates a car-like body; the lamp and the car develop chair leg structures. We further quantitatively evaluate the conditional generation results by calculating mean Intersection-over-Union (mIoU) with ground truth volumes. Here we only consider 3D volumes containing more than 500 occupied voxels, and using furthest point sampling method to select 500 out of them to describe the shape. To compensate for the sampling randomness of PointGrow, we slightly align generated points to their nearest voxels within a neighborhood of 2-voxel radius. As shown in Table 3 , PointGrow achieves above-par performance on conditional 3D shape generation. 

4. CONCLUSION:

In this work, we propose PointGrow, a new generative model that can synthesize realistic and diverse point cloud with high resolution. Unlike previous works that rely on local regularities to synthesize 3D shapes, our PointGrow builds upon autoregressive architecture to encode the diverse surface information of point cloud. To further capture the long-range dependencies between points, two dedicated self-attention modules are designed and carefully integrated into our framework. Point-Grow as a generative model also enables effective unsupervised feature learning, which is extremely useful for low-data recognition tasks. Finally, we show that PointGrow learns a smooth image condition manifold where 3D shape interpolation and arithmetic calculation can be performed inside.

:



Figures and Tables:

Figure1: The point cloud generation process in PointGrow (best viewed in color). Given i -1 generated points, our model first estimates a conditional distribution of z i , indicated as p(z i |s ≤i-1 ), and then samples a value (indicated as a red bar) according to it. The process is repeated to sample y i and x i with previously sampled coordinates as additional conditions. The i th point (red point in the last column) is obtained as {x i , y i , z i }. When i = 1, p(z i |s ≤i-1 ) = p(z i ).

Figure 2: Context related operations. ⊗: element-wise production, ⊕: concatenation.

Figure 3: The self-attention fields in PointGrow for query locations (red points), visualized as Euclidean distances between the context feature of a query point and the features of its accessible points. The distances of inaccessible points (generated after the query point) are considered to be infinitely far. Left: SACA-A; Right: SACA-B.

Figure 4: The proposed model architecture to estimate conditional distributions of point coordinates.

Figure 5: Generated point clouds for different categories from scratch by unconditional PointGrow.From left to right for each set: table, car, airplane, chair and lamp.

Figure 6: Left: Negative log-likelihood for CA-Max and CA-Mean baselines on ShapeNet airplane and car testing sets. Right: User study results on fidelity and diversity of the generated point clouds.

Figure 7: Shape completion results generated by PointGrow.

Figure 8: Examples of successfully generated point clouds by PointGrow conditioned on one-hot categorical vectors. Bottom right shows failure cases.

Figure 9: Upper left: Generated point clouds conditioned on synthetic testing images from ShapeNet (first 4 columns) and real images from PASCAL3D+ (last 2 columns). Upper right: Examples of image condition arithmetic for chairs. Bottom: Examples of image condition linear interpolation for cars. Condition vectors from leftmost and rightmost images are used as endpoints for the shape interpolation.

Classification accuracy using PointNet (Qi et al. (2017a)). SG: Train on ShapeNet training sets, and test on generated shapes; GS: Train on generated shapes, and test on ShapeNet testing sets.

Table content: <table><tr><td>The negative log-likelihood is commonly used to evaluate</td></tr><tr><td>autoregressive models for image and audio generation (Oord et al. (2016); Van Den Oord et al.</td></tr><tr><td>(2016)). However, we observed inconsistency between its value and the visual quality of point</td></tr></table>

The comparison on classification accuracy between our models and other unsupervised methods on ModelNet40 dataset. All the methods train a linear SVM on the high-dimensional representations obtained from unsupervised training.

Table content: <table><tr><td>Methods</td><td>Accuracy</td></tr><tr><td>SPH (Kazhdan et al. (2003))</td><td/></tr></table>

Conditional generation evaluation by per-category IoU on 13 ShapeNet categories. We compare our results with 3D-R2N2 (Choy et al. (2016)) and PointOutNet(Fan et al. (2017)).Image Condition Interpolation. Linearly interpolated embedding vectors of image pairs can be conditioned upon to generate linearly interpolated geometrical shapes. As shown on Figure9bottom, walking over the embedded image condition space gives smooth transitions between different geometrical properties of different types of cars.Image Condition Arithmetic. Another interesting way to impose the image conditions is to perform arithmetic on them. We demonstrate this by combining embedded image condition vectors with different weights. Examples of this kind are shown on Figure9upper right. Note that the generated final shapes contain geometrical features shown in generated shapes of their operands.

Table content: <table><tr><td/><td colspan="2">airplane bench cabinet car chair monitor lamp speaker firearm couch table cellphone watercraft Avg.</td></tr><tr><td colspan="2">3D-R2N2 (1 view) 0.513 0.421 0.716 0.798 0.466 0.468 0.381 0.662 0.544 0.628 0.513 0.661</td><td>0.513 0.560</td></tr><tr><td colspan="2">3D-R2N2 (5 views) 0.561 0.527 0.772 0.836 0.550 0.565 0.421 0.717 0.600 0.706 0.580 0.754</td><td>0.610 0.631</td></tr><tr><td>PointOutNet</td><td>0.601 0.550 0.771 0.831 0.544 0.552 0.462 0.737 0.604 0.708 0.606 0.749</td><td>0.611 0.640</td></tr><tr><td>Ours</td><td>0.742 0.629 0.675 0.839 0.537 0.567 0.560 0.569 0.674 0.676 0.590 0.729</td><td>0.737 0.656</td></tr><tr><td colspan="2">3.3 LEARNED IMAGE CONDITION MANIFOLD</td><td/></tr></table>

Peer review (iclr2019):
 Paper summary: This paper studies the problem of point cloud generation using deep autoregressive models. More specifically, it proposed a novel architecture called PointGrow network that is able to iteratively generate 3D point cloud unconditionally or conditioned on semantic context (e.g., 2D image, class label, or partial 3D point cloud). Key design is a context-aware operation that aggregates all the previously generated points dynamically by fetching and averaging. For experimental evaluations, this paper used a subset of ShapeNet (objects from 7 categories), ModelNet40 and PASCAL 3D+ dataset. == Novelty & Significance: This is a very interesting paper that applies the pixelCNN (Oord et al. 2016) framework to 3D point cloud generation. Despite the high-quality generation results shown in the paper, reviewer feels the novelty of this paper is quite limited: (1) The iterative generation has been explored to some extent in image domain; (2) Similar self-attention operators have been proposed before on point cloud recognition and generation (e.g., Qi et al 2017, Wang et al 2018). == Quality & Presentation: Overall, this paper provides solid comparisons with previous work in terms of generation quality and representation power. However, reviewers have a few concerns in terms of experimental design. Q1: For the unconditional version, please comment on the performance when training the proposed PointGrow network on all categories. Does it work or not? Q2: In Figure 3, the visualization of self-attention field looks a bit noisy. For example, most of the activated context points are not very relevant to the point on the table (on 3rd row, 7th column of Figure 3). Please clarify this in the rebuttal. Q3: In Table 1 and Table 2, it makes more sense to provide information such as model complexity as well. Q4: For unsupervised feature learning, the results will be much more convincing if the proposed model is also trained on 55 categories. It is an important ablation study. Q5: The shape completion experiment is a bit artificial. As the points are generated by some pre-defined order, the PointGrow framework may not be able to perform shape completion task in general (e.g., to complete the shape given two wings of an airplane). Q6: For image condition interpolation (see Figure 9), reviewer would like to know the performance with other categories. Please comment on the performance in the rebuttal.