3DILG: Irregular Latent Grids for 3D Generative Modeling
Authors: Biao Zhang
Matthias Nießner
Peter Wonka
Venue: NeurIPS 2022 Accept (2022)
Keywords: shape representation
generative modeling
shape generation
probabilistic shape reconstruction from a single image
shape reconstruction

Abstract: We propose a new representation for encoding 3D shapes as neural fields. The representation is designed to be compatible with the transformer architecture and to benefit both shape reconstruction and shape generation. Existing works on neural fields are grid-based representations with latents being defined on a regular grid. In contrast, we define latents on irregular grids which facilitates our representation to be sparse and adaptive. In the context of shape reconstruction from point clouds, our shape representation built on irregular grids improves upon grid-based methods in terms of reconstruction accuracy. For shape generation, our representation promotes high-quality shape generation using auto-regressive probabilistic models. We show different applications that improve over the current state of the art. First, we show results of probabilistic shape reconstruction from a single higher resolution image. Second, we train a probabilistic model conditioned on very low resolution images. Third, we apply our model to category-conditioned generation. All probabilistic experiments confirm that we are able to generate detailed and high quality shapes to yield the new state of the art in generative 3D shape modeling.

3DILG: Irregular Latent Grids for 3D Generative Modeling
Biao Zhang KAUST
Matthias Nießner Technical University of Munich niessner @ tum.de
Peter Wonka KAUST
biao.zhang @ edu.sa
pwonka @oonka@ggainn gmail.com
Abstract
We propose new representation for encoding 3D shapes as neural fields. The representation is designed to be compatible with the transformer architecture and to benefit both shape reconstruction and shape generation. Existing works on neural fields are grid-based representations with latents defined regular grid. contrast, we define latents on irregular grids, enabling our representation sparse and adaptive. In the context of shape reconstruction from point clouds, our shape representation built on irregular grids improves upon grid- grid-based methods terms of reconstruction accuracy. For shape generation, our representation promotes high-quality shape generation using auto-regressive probabilistic models. show different applications that improve results for probabilistic shape reconstruction from a single higher resolution image. Second, we train a probabilistic model conditioned very low resolution images. Third, we apply our model to calegory-conditioned generation. All probabilistic experiments confirm that we able generate detailed and high quality shapes to yield the new state of the art in generative 3D shape modeling.
be
the current state of the art. First, show
Introduction
(1) SR
(2) HRICG
(3) LRICG
(4) PCCG
CCG
Figure 1: Irregular Latent Grids enable many applications: (1) shape reconstruction, (2) high- resolution-image- aange-conditioned generation, (3) resooluton-imaag-conniii cloud-conding connddoodiionned generation, and (5) category-condition pecially suited for auto-regressive modeling (applications 2-5). For each of these applications probabilistic approach enables sampling of many plausible models for single query.
generation, (4) point- generation. The data structure is es-
Neural fields for 3D shapes are popular in machine learning because they generally easier to process with neural networks than other alternative representations, e,g., triangle meshes or spline surfaces. Earlier works represent a shape with single global latent [29, 13, 7, 36, 66]. This already gives promising results in shape autoencoding and shape reconstruction. However, shape details often missing and hard to recover from global latent. Later, local latent grids for shapes were proposed [39, 23, 3, 48]. A local latent mainly influences the shape (surface) a local 3D neighborhood, thus perceiving shape details. However, in contrast global latents, local latents
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Global
Maliscale
Imreglar
Figure 2: Latent grids. From left to right, we show single global latent (e.g., OccNet [29]), latent grid (e.g. ConvONet [39]), multiscale latent grids (e.g. II-Net [9]), and our irregular latent grid.
require positional information about their location, e.g., as implicitly defined by regular grid. Furthermore, multiscule latent pyramid grids [9, 6] provide some performance improvement over basic grids. We illustrate the three current ways of modeling neural fields in Fig. 2.
In this paper. we set out to study irregular grids as 3D shape representation for neural fields. This extends previous grid- grid-based representations, but allows each latent have an arbitrary position in space, rather than position that is rre-determined on a regular grid. We do not want to give up the advantages of a fixed length representation, and therefore propose to encode shapes as fixed length sequence of tuples (x;, Zi)EE,, where X; are the 3D positions and 2, are the latents. The overall advantages of this representation are that it is fixed length, works well with transformer- architectures using established building blocks, and that it can adapt to the 3D shape it encodes by placing latents at the most important positions. The representation also scales better than grid-based (and especially pyramid-based) representations to a larger number of latents. The number of latents representation is factor that greatly influences the training time of transformer architectures, as the computation time is quadratic in the number of latents.
While our proposed representation brings some improvement for 3D shape reconstruction from point clouds, the improvement is very significant for 3D generative models that have been less explored. We believe that the most promising avenue for 3D shape generation to follow recent image generative methods based on vector quantization and auto-regressive models using transformer, e.g. VQGAN [16]. These models operate on discrete latent vectors represented by indices [53]. During training, an autoregressive probabilistic model is trained to predict discrete indices. When doing inference (sampling), discrete indices are sampled from the autoregressive model and are decoded to images with a learned decoder. These models work well in the image domain, because a single image can comfortably be represented by medium size, e.g. 16 x 16 latent grids of 256 latents [16]. This does not directly scale 3D shapes, 16 X 16 x 16 grid is too large to be comfortably trained on 4-8 GPUs due to the quadratic complexity of the transformer architecture [55]. As result, generative models based low-resolution regular grids lead to artifacts in the details, whereas our representation yields a nice and clean surface (e.g. Fig. 9).
We show the following applications of our proposed representations (see Fig. 1). For 3D shape reconstruction, we show results for reconstruction from point cloud. For generative modeling, we show results for image-conditioned 3D shape generation, object category-conditioned 3D shape generation, and cont-cloud-conditoonn 3D shape generation. The generative model probabilistic and can generate multiple different 3D shapes for the same conditioning information.
We summarize the contributions of our work follows: We propose irregular latent grids as 3D shape representation for neural fields. Our shape representation thereby extends existing works using a global latent, local latent grid or a multi-scale grid. 2) We improve upon grid-based SOTA methods for 3D shape reconstruction for point-clouds. 3) We improve state-of-the art generative models for 3D shape generation, including and point cloud-conditioned generation.
generation, object category conditione generation
2 Related Work
2.1 Neural shape representations
Shape analysis with neural networks processes shapes in different representations. Common rep- resentations include voxels [27, 10, 12, 44] and point clouds [40, 41, 57, 58, 51]. More recently, researchers study shapes represented with neural fields [63], egg., signed distance functions (SDFs) or occupancy (indicator) functions of shapes modeled by neural networks. Subsequently, meshes can be extracted by contouring methods such as marching cubes [28]. The methods have been called

---

neural implicit representations [29, 30, 36, 18, 13, 7, 66] or coordinate norddinate-baeed representations [47]. We decided use the term neural fields in this paper [63].
Point cloud processing. Earlier works for point cloud processing rely on multilayer perceptrons (MLPs), e.g., PointNet [40]. PointNet++ [41] and DGCNN [57] extended the idea by employing a hierarchical structure to capture local information. Inspired by VIT [15], which treats images as set of patches, PCT [20] and PT [67] propose a transformer backbone for point cloud processing. Both works introduce extra modules which are not standard transformers [55] anymore. Since our main goal not to develop a general backbone for point clouds, we simply work with standard transformer for shape autoencoding (first-stage training). Similarly, both PointBERT[65]] and PointMAR [34] use standard transformers for point cloud self- seff-super learning.
Neural fields for 3D shapes. One possible approach is to represent shape with single global latent, egg., OccNet [29], CvxNet [13] and Zhang el al. [66]. While this method very simple, is not suitable to encode surface details or shapes with complex topology. Later works studied the use of local latents. Compared to global latents, these representations represent shapes with multiple local latents. Each latent is responsible for reconstruction in local region. ConvONet [39], follow-up work of OceNet, learns latents on a grid (2d or 3d). IF-Net [9], trains a model which outputs 3d latent grids of several different resolutions. The grids are then concatenated into a final representation. Some more recent works [17, 26, 49] learn to put latents on 3D positions.
2.2 Generative models
Generative models include generative adversarial networks (GANs) [19], variational autoencoders
(VAFs) [24], energy-based models [25, 59, 61, 62, 60), normalizing flows (NFs) [43, 14] and auto-regressive models (ARs) [54]. Our shape representation is designed to be compatible with auto-regressive generative models. Thus we will mainly discuss related work in this area.
In the image domain, earlier AR works generate pixels one-by-one, e.g., PixeIRNN [54] and its follow-up works [52, 45]. Combining this idea with autoregressive transformers, similar approaches are applied to 3D data, e.g., PointGrow [46] for point cloud generation and PolyGen [33] for polygonal mesh generation. Other use cases include floor plan generation [35] and indoor scene generation [56, 38].
An important ingredient of many auto-regressive models is vector quantization (VQ) that was originally integrated into VAFS. VQVAE [53] adopts a two-stage training strategy. The first stage is an autoencoder with vector quantization, where bottleneck latents quantized to discrete indices. In the second stage, an autoregressive model is trained to predict the discrete indices. This idea is further developed in VQGAN [16] which builds the autoregressive model with a transformer [55]. Vector quantization, has been shown to be an efficient way for generating high resolution images. We study how VQ can be used in the 3D domain, specifically, neural fields for 3D shapes.
There are some works adopting VQVAE to the 3D domain. Canonical Mapping [8] proposes point cloud generation models with VQVAE. In concurrent work, AutoSDF [32] generalizes VQVAE to 3D voxels. However, the bottleneck latent grid resolution is low and it is difficult to capture details as in ConvONet and IF-Net. Increasing the resolution is the key for detailed 3D shape reconstruction. On the other hand, higher resolution leads to more difficult training of autoregressive transformers. Another concurrent work, ShapeFormer [64], shows a solution by sparsifying latent grids. However, ShapeFormer still only accepts voxel grids as input (point clouds need to be voxelized). Additionally, the sequence length varies for different shapes, requiring the method to go through whole dataset to find a maximum sequence length. This relies on dataset- -ettasttdeepndeett maximum sequence lengths. In contrast to existing works, our method processes point clouds directly, learns fixed-size latents and outputs neural fields. A comparison summary can be found in Table 1.
3 Shape Reconstruction
Here we build shape autoencoding method. Given shape surface, we want to output the same shape surface. We preprocess shape surface via uniform sampling to point cloud [x;: x; E 3 Jew (also in matrix form X E Nx3, our goal is to predict the indicator function O(-) 3 - [0,
corresponding to the volume represented by the point cloud. Our representation is being generated in
3

---

Table 1: Method comparison. All methods here take point clouds input. If the column Requiring Voxelization says yes, it means the method takes voxels as input. The numbers shown in parenthesis are alternative choices of hyperparameters.
Local
Requiring
Latent Resolution
Vector Quantization
Sequence Length
Output
Comments
Latents Voxelization
OccNet [29]
X
Fields
X
1
ConvONet [39]
V
Fields
3613)
x
32*
IF-Net [9]
V
Fields
128
x
128
Multiscale
AutoSDF [32]
V
Voxels
8
8
CanMap[8]
X
Point Cloud
128
128
ShapeFormer [64]
/
Fields
(322)
variable
Sparsification
3DILG (Ours)
X
Fields
512
512
FPS
KNN
PointNet
Transformer
Reconstruction
Transformer Blocks
Transformer Blocks
2
g
VQ
R RMXKS
MMxx
gaxe
[o.
Figure 3: Shape reconstruction from point clouds. Left: the main pipeline. The framework can be used with (Bottom Right) without (Top Right) vector quantization.
three steps (See Fig. 3): patch construction (Sec. 3.1), patch information processing (Sec. 3.2), and reconstruction (Sec. 3.3).
3.1 Patch Construction
We sub-sample the input point cloud via Farthest Point Sampling (FPS),
FPS
((x:
{x, (xiJiem= = X,M E R M3
(I)
where M C A and |M
M. Next, for each point the sub-sampled point set {x)ee. we apply the K-nearest neighbor (KNN) algorithm to find K 1 points to form a point patch of size K,
Vi E M,
KNN (x:) {xx jxm),,
where Mf is the neighbor index set for point x, and [MG] = K 1. Thus we have a collection of point patches,
(xx, (xyJJem)) Wi K 1}EE XM.K E M
We project each patch with mini- Moni-PointNettlike [40] module embedding vector
Vi e M,
PointNet (x,, (xy)ee))) = e RCC
(4)
where the embedding dimension for patches.
3.2 Transformer
Furthermore, we build transformer to learn local latents {z, he The point coordinates {x; JeM converted to positional embeddings (PEs) [31], p. PE( E R.C.
PE(x) = [sin (29x, cos(2" x), (2n((2xx))) x), cos(22x),
sin .ii.......iin" cos (2 x²))
(5)
Our transformer takes as input the sequence of patch-PE pairs,
Transformer (((e,, p:))imm {z; 2; E RC JieM-
(6)
The transformer is composed of blocks. We denote the intermediate outputs of all blocks as (0) (1)
z; 22 ... z where z, is the input e; and z, is the output z;.
(1) (L)

---

T0
10,2
20,3
Trnnsoomee Blocks
Transfommer Blocks
Tonnsormer Blocks
Transformet Blocks
P0.1
20.2
20,3
20
F1.2
11,33
Transformer
112
11.
F2,1
F2.2
P2.3
Figure 4: Autoregressive Generative Models with Unidirectional Transformer. Left: sequence
prediction. Right: detailed visualizations with sequence element components.
Steps
Figure 5: Autoregressive Generative Models with Bidirectional Transformer. Cubes (0) predicted tokens. From left to right, we show 8 decoding steps.
Vector Quantization. This model can be used with vector quantization [53]. We simply replace the intermediate output at block with its closest vector in a dictionary D,
2 (1)
Vi E M,
arg min 2 ED
The dictionary D contains (D] D vectors. As in VQVAE [53], the gradient with respect to (1) is approximated with the straight-through estimator [1].
3.3 Reconstruction
We expect that each latent z; is responsible for shape reconstruction near the point x;. However, our goal is to estimate (-) for an arbitrary x E RR.. We interpolate latent zx for point x with the Nadaraya-Watson Watson estimator,
ExpMexp( -
x2
Zx ΣIEM exp(-311 x
(8)
where 3 controls the smoothness of interpolation and can be fixed or learned. The final indicator an MLP with a sigmoid activation,
(xx) Sigmoid(MLP(x,.2) 2x)).
(9)
Loss Function. We optimize the estimated Ō(-) by comparing to ground-truth O(-) via binary cross entropy loss (BCE) and an additional commitment loss term [53],
Lrecon Xecmmmit EXERS BCF((((), +XBXERS FREM (2g( (2)
(10)
where sg(-) is the stop-gradient operation. When À 0, means we train the model without vector quantization.
Autoregressive Generative Modeling
With Vector Quantization (Eq. 7), we compress the bit size of intermediate latents (2() hem to logz D where D the size of the dictionary D. We denote the compressed index as 2; {0. 1......,, D- -1). We also quantize point coordinates x; to ( X1.3) where each entry is a 8-bit integer (0, 1....... ,55), As a result, we obtain a discrete representation of the 3D shape
{ 22,, 21,3, 24) Jem.

---

4.1 Unidirectional Transformer
Autoregressive generation with unidirectional transformer more classical approach. We often need to sequentialize an unordered set if the order is not defined yet. Specifically, we re-order the representations in ascending order by the first component Ti,1, then by the second component 21,2- and finally by the third component 34,3+
S { To,2, F0,3, 2o), (#1,1, 21,2 21,3 21)...., (0M- TM- - (M-1) (11)
Our goal to predict sequence elements one-by-one. A common way to flatten the sequence by concatenating the quadruplets, for examples, PointGrow [46] and PolyGen [33]. However, here we consider an approach generating coailleet----uuuuupleeee We write og (Zi,1, 21,2, 21,3 21). The
likelihood of generating o, autoregressively is
C=M-1
P(S
II k
p(o: Oci.C),
(12)
where C is conditioned context. Similar to ATISS (38), we predict components of o, = (24,1, T4, 24,3, 24) autoregressively:
p(o; loci C) =p(r;, 1os, C)- 21,1
(13)
24,1 p(zi|11,,, 24,2, Ti,1, Oci, C).
The model is shown in Fig. 4. Different from ATISS [38] which uses MLPs to decode different components, instead we continue to apply transformer blocks for component decoding.
4.2 Bidirectional Transformer
Bidirectional transformer for autoregressive generation is recently proposed by MaskGIT [5]. Here show model can also be combined with bidirectional transformer. However, here we consider a different task. We generate {zi]iem conditioned on {(Ti,), Zt,2, vi.3) Fie.)).eM. In training phase, we sample subset {=: JEVCM of izi as the input of the bidirectional transformer. We aim to predict (ee The coordinates {(Fi,1 Fi,2, x1,a)) VEM are converted to positional embeddings (either learned or fixed) as condition. The likelihood of generating M \\ V is as follows,
p((ziliem (eihev, ((Fi,1, Ti,2 F1,,, (133) VCM
II
(14)
In practice, the bidirectional transformer takes as input all tokens except that {=:)eemy are replaced by a special masked token [m]. When decoding (inference), we iteratively predict multiple tokens at the same time. Tokens are sampled based on their probabilities (transformer output). See [5] for detailed explanation. We show visualization of decoding steps in Fig. 5.
Reconstruction Experiments
We the size of the input point cloud N 2048 The number and the size of point patches are M = 512 and K = 32, respectively. In the case of Vector Quantization, there are D = 1024 vectors in the dictionary D. Other details of the implementation can be found in the Appendix. We use the datset ShapeNet-v2 [4] for shape reconstruction. We split samples into train/val/test (48597/1 000999//1///// 1283/2592) /252) set. Following the evaluation protocol of [13, 66]. we include three metries, volumetric IoU, the Chamfer-LI distance, and F-Score [50]. We also show reconstruction results on another leject-level dataset ABO [11], a real-world dataset D-FAUST [2], and a synthetic scene-level dataset in the appendix.
Results We compare our method with three existing works, OccNet [29], ConvONet [39] and IF-Net [9]. The results can be found in Table 2. We select 7 categories among 55 categories with largest training set (table, car, chair, airplane, sofa, rifle, lamp). Detailed results can found the Appendix. We show different choices of M. As we can see in this table, increasing M from 64 to 512 gives performance boost. Even with the simplest model M = 64, our results outperform ConvONet. The best results are achieved when setting M 512. In this case, results lead in most categories

---

Table 2: Shape reconstruction. We train all models on ShapeNet-v2 (55 categories). All baseline methods are trained with the corresponding officially released code. For our model, we N 2048 and K = 32. We select 7 categories to show. These categories have largest training set among 55 categories. We also show averaged metrics over all categories. The numbers shown in parenthesis are results of vector quantization.
3DILG (Ours)
OccNet ConvONet
IF-Net
M 64
128
M 256
512
mean (selected)
0.822
0.881
0.929
0.922(0.904)
0.936(0.929)
0.945(0.943)
(.952(0.950)
IoU f
mean (all)
0.825
0.888
0.934
0.937(0.229) 0.923)))907)
0.946(0.943)
(.953)0
mean (sclected)
0.058
0.040
0.034
0.038(0.038)
0.035(0.036)
0.034(0.034)
0.032
Chamfer
mean (all)
0.072
0,052
0,041
0.048(0.052)
0.044(0044)
(.041(0.022)
0.040(0000)
mean (selected)
0.898
0.951
0.975
0.959(09948)
0.972 0.968(09964)
0.976(0.775)
F-Score f
mean (all)
0.858
0.933
0.967
0.942(0.926) 0.955(039488) (.953)))958) (.966(0.965)
when comparing to IF-Net. The results of IF-Net are better than ours in terms of metric F-Score in some categories. However, we argue that this case the metric F-score is saturated (values are close 1), which making it hard to compare. We also show results after introducing vector quantization to our model in the same table. We can see that vector quantization harms the performance slightly.
Qualitative Comparison Qualitative results are shown in Fig. 6. From the visualization, it can be seen that the reconstruction quality increases M increases, partieularly for shapes with complex topology. OceNet, as global latent method, often fails to recover complex structures. ConvONet can recover better structures than OceNet, due its localized latents. By learning multiscale latents, IF-Net improves further upon ConvONet. Our method with M = 64 outperforms ConvONet, and with M 128, the results are comparable with IF-Net.
6 Generative Experiments
introduce three experiments to show how our model can be combined with auto-regressive trans- formers for generative modeling. In Sec. 6.1, we show image-conditioned generation probabilistic shape reconstruction from single image. Sec. 6.2, we show how we generate samples given a category label (using the 55 ShapeNet categories). In Sec. 6.3, further show generation condi- tioned on downsampled point clouds. In contrast to the first two tasks, the generation is based on a bidirectional transformer described in Sec 4.2. More generative results on additional datasets ABO and D-FAUST can be found in the appendix.
6.1 Probabilistic Shape Reconstruction from Single Image
We train uni-directional transformer for this task (see Sec. 4.1). The context is image. To train this model, we render 40 images (224 x 224) of different views for each shape in ShapeNet. The implementation of the uni-directional transformer is based GPT [42]. It contains 24 blocks, where each block has an attention layer with 16 heads and 1024 embedding dimension. When sampling, nucleus sampling [22] with top-p (0.85) and top-k (100) are applied predicted token probabilities.
High resolution images. We show some generated samples when C are high resolution images in Fig. 7. We compare our results with deterministic method, OceNet [29]. As we can see in the results, the deterministic method (OccNet) tends create blurred meshes. However, our probabilistic reconstruction is able to output detailed meshes.
Low resolution images. We also consider another more challenging task. The input images are downsampled to low resolution (16 16). In this case, the generative model has more freedom to find multiple plausible interpretations, including variations with different topology (see Fig. 8).
7

---

ADIL & (Qurs)
Topus
GT
OceNet
ComoNNer
Figure 6: Shape reconstruction. The column Input shows input point clouds of size 2048. The column GT shows ground-truth meshes. We compare our results with different M to OceNet, ConvONet and IF-Net. We also show {x)Tee obtained via Farthest Point Sampling.
6.2 Category-Conditionel Generation
To generate shapes given a category label, we use the context C encode the category label. We employ the uni-directional transformer based on GPT as in Sec. 6.1. 1. We compare to our proposed baseline model that encodes shapes as a latent grid of resolution 83, To make fair comparison, we also extend VIT [15] to the 3D voxel domain in the first stage of training. In the second stage, we the same uni-directional transformer for sequence prediction. Here the sequence length is g = 512, which is the same as in proposed model. The baseline model is named Grid-83. The comparison to the baseline model is in Fig. We can see that the baseline unable to generate high quality shapes. We argue that this is because the representation is not expressive enough to capture surface details. Furthermore, we show more generated samples of our model in Fig. 10. The three selected categories are bookshelf, bench and chair. For both the baseline and model, render 10 images of predicted shapes, and calculate the Fréchet Inception Distance (FID) [21, 37] between predictions and test sets. The metrics can be found in Table 3. A perceptual study the quality of generated samples can be found in the appendix.
6.3 Point Cloud Conditioned Generation
We train a bidirectional transformer described in See 4.2. The model takes as input (xiJee- The results can be found in Fig. 11.

---

Input
(Oum)
Input
SDILG
OceNet
Figure 7: Image-conditioned generation (224 224). We sample shapes for each input image, and compare them with OccNet.
Probabilistic Reconstruction with 3DIGG (Ours)
Figure 8: Image-conditioned generation (16 x 16). We sample shapes for each input image.
7 Conclusion
We have studied neural fields for shapes and presented new representation. In contrast to common approaches which define latents a on a regular grid or multiple regular grids, we position latents on an irregular grid. Comparing to existing works, the representation better scales to larger models, because the irregular grid is sparse and adapts to the underlying 3D shape the object. In our results, we demonstrated improvement in 3D shape reconstruction from point clouds and generative modeling conditioned on images, object category, or point-clouds over alternative grid-based methods. In future work, suggest explore other applications of our proposed representation, e.g., shape completion, and extensions to textured 3D shapes and 31) scenes.
Broader impact
We introduce a new 3d shape representation for generative modeling and shape analysis. This shape representation is designed to be compatible with the transformer architecture. We demonstrate some example applications in the paper including shape generation conditioned on images, point clouds, and shape class, and 3d shape reconstruction. However, envision our proposed shape representation to be general and could be employed all shape processing tasks.
Potential societal impacts of generative modeling in general exist. Future iterations of our work could possibly used to generate high fidelity 3D virtual humans. However, we do not see an important negative societal impact that is specific to work and that would constitute immediate concern.
(Ours)
3
Figure 9: Comparison of category-conditioned generation. We compare our results (Left) with an 8 latent grid baseline (Right).

---

Table 3: FID J. for category-conditione generation. We compare our results with the baseline
Grid-8 The 7 categories are the largest categories in ShapeNet.
Categories
mean
table
car
chair
airplane
sofa
rifle
lamp
Grid-8a
72.396 68.016
95.566
58.649 45.333
42.009 30.957
58.092 53.244
59,456 40.500
87.319 72.672
67.641
3DILG (Ours)
92.597
57.617
Figure 10: Category-conditione generation. We choose categories to show (bookshelf, bench and chair). We show 100 samples for each category.
Figure 11: Point cloud conditioned generation. We show decoding steps.
Acknowledgements
We would like acknowledge support from the SDAIA-KAUST Center of Excellence in Data Science and Artificial Intelligence.
10

---

References
[1] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 1308.3432 2013. (Cited on 5)
[2] Federica Bogo, Javier Romero, Gerard Pons-Moll, and Michael J Black. Dynamic faust: Registering human bodies in motion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6233-6242. 2017. (Cited on 6)
[3] Rohan Chabra, Jan E. Lenssen, Eddy 1lg, Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe. Deep local shapes: Learning local sdf priors for detailed 3d reconstruction. In European Conference on Computer Vision, pages 605- Springer, 2020. (Cited
[4] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint 1512 030112, 2015. (Cited on 6)
[5] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. arXiv preprint arXiv:2202. 04200, 2022. (Cited on 6)
[6] Zhang Chen, Yinda Zhang, Kyle Genova, Sean Fanello, Sofien Bouaziz, Christian Häne, Ruofei Du, Cem Keskin, Thomas Funkhouser, and Danhang Tang. Multiresolution deep implicit functions for 3d shape representation. Proceedings of the IEEF/CVF International Conference Computer Vision, pages 13087-13096, 2021. (Cited 2)
[7] Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang. Bsp-net: Generating compact meshes via binary space partitioning. arXiv preprint 19Xiv:991100997 2019,, (Cited on 3)
[8] An-Chieh Cheng, Xueting Li, Sifei Liu, Min Sun, and Ming-Hsuan Yang. Autoregressive 3d shape generation via canonical mapping. arXiv preprini arXiv:2204.01955, 2022. (Cited on 3,
[9] Julian Chibane, Thiemo Alldieck, and Gerard Pons-Moll. Implicit functions in feature space for 3d shape reconstruction and completion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6970-6981, 2020. (Cited on 2, 3, 4, 6)
[10] Christopher B Choy, Danfei Xu, Jun Young Gwak, Kevin Chen, and Silvio Savarese, 3d-r2n2: unified approach for single and multi view 3d object reconstruction. In European conference on computer vision, pages
Springer, 2016. (Cited on 2)
[11] Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F Yago Vicente, Thomas Dideriksen, Himanshu Arora, et al. Abo: Dataset and benchmarks for real-world 3d object understanding. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 21 21126-211211 21,,, 2022. (Cited on
[12] Angela Dai, Charles Ruizhongtai Qi, and Matthias Nießner. Shape completion using 3d-encoder- predictor enns and shape synthesis. In Proceedings of the IEEE conference computer vision and pattern recognition, pages
2017. (Cited on 2)
[13] Boyang Deng, Kyle Genova, Soroosh Yazdani, Sofien Bouaziz, Geoffrey Hinton, and Andrea Tagliasacchi. Cvxnet: Learnable convex decomposition. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition, pages 31-44, 2020. (Cited on 1, 3, 6)
[14] Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: non-linear independent components estimation. In Yoshua Bengio and Yann LeCun, editors, International Conference on Learning Representations (ICLR), 2015. (Cited on 3)
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021. (Cited on 8)

---

[16] Patrick Esser, Robin Rombach, and Bjor Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12873-12883, 2021. (Cited on 2)
[17] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sama,, and Thomas Funkhouser. Local deep implicit functions for 3d shape. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 4857-4866, 2020. (Cited on 3)
[18] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In Proceedings of the IEEE International Conference on Computer Vision, pages 7154-7164, 2019. (Cited on 3)
[19] Ian Goodfellow, Jean Pouget-Abadis Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Information Processing Systems, 27:2672-200 26800, 2014. (Cited on 3)
[20] Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pet: Point cloud transformer. Computational Visual Media, 7(2):187-199, 2021. (Cited on 3)
[21] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017. (Cited on 8)
[22] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint 1904 09751, 2019. (Cited on 7)
[23] Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, Thomas Funkhouser, et Local implicit grid representations for 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6001-6010, 2020. (Cited on 1)
[24] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun, editors, International Conference on Learning Representations (ICLR), 2014. (Cited on 3)
[25] Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial on energy- learning. Predicting structured data, 1(0), 2006. (Cited on 3)
[26] Tianyang Li, Xin Wen, Yu-Shen Liu, Hua Su, and Zhizhong Han. Learning deep implicit functions for 3d shapes with dynamic code clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12840-12850, 2022. (Cited on 3)
[27] Yangyan Li, Soeren Pirk, Hao Su, Charles R Qi, and Leonidas J Guibas. Fpnn: Field probing neural networks for 3d data. Advances in neural information processing systems, 29, 2016. (Cited on 2)
[28] William E Lorensen and Harvey Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM siggruph computer graphics. 2164):163-169, 1987. (Cited 2)
[29] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 4460-4470, 2019, (Cited on 1,
2, 3, 4, 6)
[30] Mateusz Michalkiewicz. Jhony K Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders Eriksson. Deep level sets: Implicit surface representations for 3d shape inference. arXiv preprint arXiv: 100:: 06802, 2019. (Cited on 3)
[31] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference computer vision, pages 405-421. Springer, 2020. (Cited on 4)
12

---

[32] Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, and Shubham Tulsiani. Autosdf: Shape priors for 3d completion, reconstruction and generation. arXiv preprint arXiv:2203.095 2022.
(Cited on 3,4) 4)
[33] Charlie Nash, Yaroslav Ganin, SM Ali Eslami, and Peter Battaglia. Polygen: An autoregressive generative model of 3d meshes. In International Conference on Machine Learning, pages 7220-7229. PMLR, 2020. (Cited on 3, 6)
[34] Yatian Pang. Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud self-supervised learning. arXiv preprint 3xXiv:2003.06604, 2022. (Cited on 3)
[35] Wamiq Para, Paul Guerrero, Tom Kelly, Leonidas J Guibas, and Peter Wonka. Generative layout modeling using constraint graphs. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6690-6700, 2021. (Cited on 3)
[36] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 165-174, 2019. (Cited on 3)
[37] Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. In CVPR, 2022. (Cited on 8)
[38] Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis, Andreas Geiger, and Sanja Fidler. Atiss: Autoregressive transformers for indoor scene synthesis. Advances in Neural Information Processing Systems, 34, 2021. (Cited on 3, 6)
[39] Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In European Conference on Computer Vision, pages 523 123-540.36 Springer, 2020. (Cited on 1, 2, 3,
[40] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017. (Cited on 3,
[41] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Advances in neural information processing systems, pages 5099-519900 2017. (Cited on 2, 3)
[42] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. (Cited on 7)
[43] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning, pages 1530-1538, 2015. (Cited on 3)
[44] Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. Octnet: Learning deep 3d represen- tations at high resolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3577-3586, 2017. (Cited on 2)
[45] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelenn++: Improving the pixelemn with discretized logistic mixture likelihood and other modifications. arXiv preprint 1701v:1701.05517. 2017. (Cited on 3)
[46] Yongbin Sun, Yue Wang, Ziwei Liu, Joshua Siegel, and Sanjay Sarma. Pointgrow: Autore- gressively learned point cloud generation with self-attention. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 61-70, 2020. (Cited 3, 6)
[47] Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P. Srinivasan, Jonathan T. Barron, and Ren Ng. Learned initializations for optimizing coordinate- based neural representa- tions. In CVPR, 2021. (Cited on 3)
[48] Jiapeng Tang, Jiabao Lei, Dan Xu, Feiying Ma, Kui Jia, and Lei Zhang. Sa-convonet: Sign- agnostic optimization of convolutional occupancy networks. Proceedings of the IEEF/CVF International Conference Computer Vision, pages 3504-6513, 2021. (Cited on 1)
13

---

[49] Jiapeng Tang, Markhasin Lev, Wang Bi. Thies Justus, and Matthias Nießner. Neural shape deformation priors. In Advances in Neural Information Processing Systems, 2022. (Cited on 3)
[50] Maxim Tatarchenko, Stephan R Richter, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox. What do single-view 3d reconstruction networks learn? In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3405-3414, 2019, (Cited on 6)
[51] Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE International Conference on Computer Vision, pages (411-6420, 2019. (Cited on 2)
[52] Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Con- ditional image generation with pixelenn decoders. Advances in Neural Information Processing Systems, 29:4790-4798, 2016. (Cited on 3)
[53] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017. (Cited on 2, 3,
[54] Aaron Van Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning, pages 1747-1756, 2016. (Cited on 3)
[55] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. (Cited on 2, 3)
[56] Xinpeng Wang, Chandan Yeshwanth, and Matthias Nießner. Sceneformer: Indoor scene generation with transformers. In 2021 International Conference on 3D Vision (3DV), pages 105- IEEE, 2021. (Cited on 3)
[57] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph enn for learning on point clouds. Acm Transactions On Graphics (tog), 1-- 2019. (Cited on 3)
[58] Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pages 9621-9638 2019. (Cited on 2)
[59] Jianwen Xie, Yang Lu, Song- Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference on Machine Learning, pages 2635-2644. PMLR, 2016. (Cited on 3)
[60] Jianwen Xie, Yifei Xu, Zilong Zheng. Mhng-Chun2uu Zhu, and Ying Nian Wu. Generative pointnet: Deep energy-based based learning on unordered point sets for 3d generation, reconstruction and classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14975- 2021. (Cited on 3)
[61] Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang, Song-Chun Zhu, and Ying Nian Wu. Learning descriptor networks for 3d shape synthesis and analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8629-8638, 2018. (Cited on 3)
[62] Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang, Song- Chun Zhu, and Ying Nian Wu. Generative voxelnet: learning maaey--based models for 3d shape synthesis and analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020. (Cited on 3)
[63] Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual computing and beyond. Computer Graphics Forum, 2022. (Cited on 2,
[64] Xingguang Yan, Liqiang Lin, Niloy J Mitra, Dani Lischinski, Danny Cohen-Or, and Hui Huang. Shapeformer: Transformer-basee shape completion via sparse representation. arXiv preprint arXiv:2201.10:3 2022. (Cited on 3, 4)
[65] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point- bert: Pre-training 3d point cloud transformers with masked point modeling. arXiv preprint 2rX.v:2111.14819, 2021. (Cited on 3)
14

---

[66] Biao Zhang and Peter Wonka. Training data generating networks: Shape reconstruction via bi-level optimization. International Conference on Learning Representations, 2022. (Cited on 1, 6)
[67] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16259-
16268, 2021. (Cited on
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Appendix.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] See Appendix. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix.
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [N/A] We specify the random seed. The results are reprodueible.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [Yes] See Appendix.
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [Yes]
15


Reviews:
[{"review_id": "xMJOIOzK9uU", "reviewer": "Reviewer_uFdG", "summary": "This paper proposes a geometry representation based on irregular grids for generative shape modeling.", "questions": "My suggestions for improvement are:\n- Having examples with non-shape-net data to illustrate robustness and utility in practice.\n- Having comparisons with surface reconstruction techniques other than only neural grids. The point clouds look clean and seem to sample the shapes well.\n- If no good results with real data for generative tasks, I would drop that claim and focus on the new representation's ability for reconstruction, efficiency, spatial adaptivity, etc.", "limitations": "Please see above.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- It is an interesting idea, especially when combined with transformers and vector quantization.\n- Some good results on reconstructed and generated shapes, although I am not convinced with the comparisons.\n\nWeaknesses:\n- The advantage over point cloud-based generative modeling is not well established, both conceptually and experimentally.\n- It is a bit of a stretch to call this a grid-based algorithm and only consider grid-based alternatives.\n- The reconstructions are only compared against other networks, there are no comparisons to surface reconstruction methods e.g. Poisson or edge-preserving MLS.\n- The proposed generative tasks are only on shape-net, no real images or point clouds.\n- Memory and runtimes are not clear.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "rY6lB1VEEQe", "reviewer": "Reviewer_jKmw", "summary": "The paper proposes a method for detailed shape reconstruction from point clouds and images, as well as shape generation. The method utilizes FPS and KNN together with a PointNet to create patch embeddings of train shapes, and then it encodes a shape indicator function using a transformer architecture. The paper demonstrates impressive shape reconstruction and generation results for all of the aforementioned tasks.", "questions": "Please see and address the questions raised in the previous section.", "limitations": "The authors need to address further a potential limitation of the method, namely, its sensitivity to changes in sampling of the input point clouds.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper presents a novel method for encoding local shape information using irregular latent spaces and using transformer-based autoregressive models for shape reconstruction from point clouds and images.\n- The paper presents impressive - accurate and detailed, shape generation results from high-res and blurred images, using compact shape latent spaces.\n- The presentation is very clear.\n- The paper presents a comprehensive set of experimental results illustrating the advantages of the proposed method and comparing it to current SOTA methods.\n\nWeaknesses:\n- A possible weakness of the proposed approach may be its sensitivity to shape sampling. E.g., how would the method work if the input point clouds were not all sampled at N = 2048 points? How were these points obtained? Would irregular sampling density affect the results of the proposed method? This possible limitation needs to be addressed in the paper, with some experimental results and a discussion.\nParts of the method's description which require further classification.\n- In Figure 4, what’s the output of the model? How the point cloud, or another shape representation, is decoded from it?\n- Figure 5 is inconsistent with Figure 4. Show a block diagram of the proposed architecture, as you do in Figure 4.\n- In Section 4, how are x_{i,k} obtained for different i-s? What is the difference between x_{i,k} and x_{j,k} for different i, j? Please explain.\n- In Section 5, further comparison with IF-NET is needed: the proposed method slightly improves over IF-NET. How does it compare to IF-NET (and other methods) in other respects - resource consumption, train and inference time, etc.?\n- Explain the purpose of Eq.(8) and Eq.(9).\nComments regarding supplementary material.\n- In A.4., how do the methods compare in terms of memory and runtime?\n- In A.6., ll.62-63 - “Bad” in what sense? Non-realistic generations by the proposed method should be highlighted as well.\nOther comments.\n- In l.130, explicitly state which positional encodings are used, for completeness of the presentation.\n- Introduction section needs to be revised to more clearly describe the main aspects of the proposed approach.\n- A related work which should be cited: Y. Li, S. Pirk, H. Su, C. R. Qi, L. J. Guibas, FPNN: Field Probing Neural Networks for 3D Data, Neural Information Processing Systems (NIPS 2016).\n- In Section 6.1, l.214, provide reference for GPT.\n- Nit: In Table 1, consider writing all numbers in the same manner such that it will be clear which parameters are identical for different methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HMyyucKlkJk", "reviewer": "Reviewer_tLeQ", "summary": "This paper presents a new 3D shape representation for 3D generative modeling. The proposed shape representation is formed as irregular grids of latent vectors in the 3D space that roughly forms the shape, which can be encoded with Transformer-based networks and decoded via autoregressive generation, similar to VQ-GAN. Experiments on 3D shape generation conditioned on different context is performed on ShapeNet and shows promising results.\n", "questions": "It would be great if the authors could address all the concerns listed above.", "limitations": "The limitations and potential societal impact have been adequately addressed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The authors proposed a very interesting idea of bringing recent advances on generative image modeling into 3D shapes, specifically VQ-GAN. It nicely brings in the sparse nature of 3D shapes, and bridges together commonly accepted differentiable point cloud processing operations with autoregressive generation.\n- The experimental results are pretty promising. I especially like the fact that, conditioned on a single input image, the model is capable of generating different shape modalities, which is something previous deterministic shape reconstruction methods cannot achieve.\n- The paper is nicely written and the presentation is clear.\n\nWeaknesses:\n- Maybe I missed it, but it was not clear how the autoregressive generation ordering was determined (L156-158). I think the authors should also provide some discussions on whether (or how) the model is / is not invariant to the permutation of such ordering; for completeness, results comparing different input permutations may also give a better understanding.\n- For the \"image-conditioned generation\" experiments comparing single-image shape reconstruction, I would like to understand more where the actual performance gain comes from. Besides the new irregular latent grid representation, the entire backbone architecture (GPT) is also much more powerful than OccNet (ResNet-18); even the pretraining data is entirely different. Therefore, I am not totally convinced that all the benefit comes from the new representation. I think a critical missing experiment is to ablate this factor; specifically, a baseline that uses GPT but predicting occupancy like OccNet should be included to verify that the irregular latent grids do have its merits.\n- For the image-conditioned experiments (following the above concern), it was not clear whether there are still input point clouds being conditioned. If yes, then the comparison against OccNet becomes yet more unfair; if not, it is not very clear what the input pipeline would look like. My guess is that a full shape reconstruction network (Fig 3) is trained first, then only the Transformer part is taken for the experiment (with the PointNet discarded); it would be good if the authors could clarify.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4Tqf7Ah7MRr", "reviewer": "Reviewer_uMXb", "summary": "This submission proposes to irregularly distribute local SDF latent codes in 3D, to adapt to shapes and not be restricted to a 3D grid with fixed resolution.\nIn addition, authors present ways to quantize their representation, and use it in the generative setting with uni- and bi-directional transformers.", "questions": "l. 33 : Why is fixed length representation a benefit? Wouldn't an adaptive representation be more useful, with a tiny representation for simple shapes, and long representation for complex ones?\n\nl.141 What is the motivation behind the non linear interpolation between local latent codes? Could it be ablated?", "limitations": "Compared to autodecoders, this probabilistic pipeline can no longer represent a shape with a latent code.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- excellent technical presentation. The use of colors in the text and equations to refer to different pipeline parts is a great idea.\n- generating multiple plausible outputs given 1 single conditioning signal (eg. Fig. 10) is a very good use of probabilistic models\n- results look qualitatively good, and the proposed pipeline is properly compared against recent models (IF-Net, ConvOccNet) that distribute latent codes on fixed grids.\n\n\nWeaknesses:\n- Quantizing 3D positions of local latents sounds very similar to using a grid. What is the difference here?\n- Some very related papers such as \"Local Deep Implicit Functions for 3D Shape\" [CVPR 2020] are not mentioned. In this one, \"the global implicit function is decomposed into the sum of N local implicit functions.\", not on a grid...\n- The intro+related Sections mention shape representation and generation, while the method section starts with an input point cloud.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept