Title: CONVOLUTIONAL NEURAL NETWORKS ON NON-UNIFORM GEOMETRICAL SIGNALS USING EUCLIDEAN SPECTRAL TRANSFORMATION

Abstract: Convolutional Neural Networks (CNN) have been successful in processing data signals that are uniformly sampled in the spatial domain (e.g., images). However, most data signals do not natively exist on a grid, and in the process of being sampled onto a uniform physical grid suffer significant aliasing error and information loss. Moreover, signals can exist in different topological structures as, for example, points, lines, surfaces and volumes. It has been challenging to analyze signals with mixed topologies (for example, point cloud with surface mesh). To this end, we develop mathematical formulations for Non-Uniform Fourier Transforms (NUFT) to directly, and optimally, sample nonuniform data signals of different topologies defined on a simplex mesh into the spectral domain with no spatial sampling error. The spectral transform is performed in the Euclidean space, which removes the translation ambiguity from works on the graph spectrum. Our representation has four distinct advantages: (1) the process causes no spatial sampling error during the initial sampling, (2) the generality of this approach provides a unified framework for using CNNs to analyze signals of mixed topologies, (3) it allows us to leverage state-of-the-art backbone CNN architectures for effective learning without having to design a particular architecture for a particular data structure in an ad-hoc fashion, and (4) the representation allows weighted meshes where each element has a different weight (i.e., texture) indicating local properties. We achieve results on par with the state-of-the-art for the 3D shape retrieval task, and a new state-of-the-art for the point cloud to surface reconstruction task.

1. INTRODUCTION:

We present a unifying and novel geometry representation for utilizing Convolutional Neural Networks (CNNs) on geometries represented on weighted simplex meshes (including textured point clouds, line meshes, polygonal meshes, and tetrahedral meshes) which preserve maximal shape information based on the Fourier transformation. Most methods that leverage CNNs for shape learning preprocess these shapes into uniform-grid based 2D images (rendered multiview images) or 3D images (binary voxel or Signed Distance Function (SDF)). However, rendered 2D images do not preserve the 3D topologies of the original shapes due to occlusions and the loss of the third spatial dimension. Binary voxels and SDF representations under low resolution suffer big aliasing errors and under high resolution become memory inefficient. Loss of information in the input bottlenecks the effectiveness of the downstream learning process. Moreover, it is not clear how a weighted mesh where each element is weighted by a different scalar or vector (i.e., texture) can be represented by binary voxels and SDF. Mesh and graph based CNNs perform learning on the manifold physical space or graph spectrum, but generality across topologies remains challenging.

In contrast to methods that operate on uniform sampling based representations such as voxel-based and view-based models, which suffer significant representational errors, we use analytical integration to precisely sample in the spectral domain to avoid sample aliasing errors. Unlike graph spectrum based methods, our method naturally generalize across input data structures of varied topologies. Using our representation, CNNs can be directly applied in the corresponding physical domain obtainable by inverse Fast Fourier Transform (FFT) due to the equivalence of the spectral and physical domains. This allows for the use of powerful uniform Cartesian grid based CNN backbone architectures (such as DLA (Yu et al., 2018) , ResNet (He et al., 2016) ) for the learning task on arbitrary geometrical signals.

Although the signal is defined on a simplex mesh, it is treated as a signal in the Euclidean space instead of on a graph, differentiating our framework from graph-based spectral learning techniques which have significant difficulties generalizing across topologies and unable to utilize state-of-the-art Cartesian CNNs.

We evaluate the effectiveness of our shape representation for deep learning tasks with three experiments: a controlled MNIST toy example, the 3D shape retrieval task, and a more challenging 3D point cloud to surface reconstruction task. In a series of evaluations on different tasks, we show the unique advantages of this representation, and good potential for its application in a wider range of shape learning problems. We achieve state-of-the-art performance among non-pre-trained models for the shape retrieval task, and beat state-of-the-art models for the surface reconstruction task.

The key contributions of our work are as follows:

• We develop mathematical formulations for performing Fourier Transforms of signals defined on a simplex mesh, which generalizes and extends to all geometries in all dimensions. (Sec.

3)

• We analytically show that our approach computes the frequency domain representation precisely, leading to much lower overall representational errors. (Sec. 3)

• We empirically show that our representation preserves maximal shape information compared to commonly used binary voxel and SDF representations. (Sec. 4.1)

• We show that deep learning models using CNNs in conjunction with our shape representation achieves state-of-the-art performance across a range of shape-learning tasks including shape retrieval (Sec. 4.2) and point to surface reconstruction (Sec. 4.3) (Su et al., 2015a; Shi et al., 2015; Kar et al., 2017) , volumetric voxels (Wu et al., 2015; Maturana & Scherer, 2015; Wu et al., 2016; Brock et al., 2016 ), point clouds (Qi et al., 2017a; b; Wang et al., 2018b) , polygonal meshes (Kato et al., 2018; Wang et al., 2018a; Monti et al., 2017; Maron et al., 2017) , shape primitives (Zou et al., 2017; Li et al., 2017) , and hybrid representations (Dai & Nießner, 2018) .

Our proposed representation is closest to volumetric voxel representation, since the inverse Fourier Transform of the spectral signal in physical domain is a uniform grid implicit representation of the shape. However, binary voxel representation suffers from significant aliasing errors during the uniform sampling step in the Cartesian space (Pantaleoni, 2011) . Using boolean values for de facto floating point numbers during CNN training is a waste of information processing power. Also, the primitive-in-cell test for binarization requires arbitrary grouping in cases such as having multiple points or planes in the same cell (Thrun, 2003) . Signed Distance Function (SDF) or Truncated Signed Distance Function (TSDF) (Liu et al., 2017; Canelhas, 2017) provides localization for the shape boundary, but is still constrained to linear surface localization due to the linear interpolation process for recovering surfaces from grids. Our proposed representation under Fourier basis can find nonlinear surface boundaries, achieving subgrid-scale accuracy (See Figure 2 ).

2.2. LEARNING ARCHITECTURES:

Cartesian CNNs are the most ubiquitous and mature type of learning architecture in Computer Vision. It has been thoroughly studied in a range of problems, including image recognition (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; He et al., 2016) , object detection (Girshick, 2015; Ren et al., 2015) , and image segmentation (Long et al., 2015; He et al., 2017) . In the spirit of 2D image-based deep learning, Cartesian CNNs have been widely used in shape learning models that adopted multiview shape representation (Su et al., 2015a; Shi et al., 2015; Kar et al., 2017; Su et al., 2015b; Pavlakos et al., 2017; Tulsiani et al., 2018) . Also, due to its straightforward and analogous extension to 3D by swapping 2D convolutional kernels with 3D counterparts, Cartesian CNNs have also been widely adopted in shape learning models using volumetric representations (Wu et al., 2015; Maturana & Scherer, 2015; Wu et al., 2016; Brock et al., 2016) . However, the dense nature of the operations makes it inefficient for sparse 3D shape signals. To this end, improvements to Cartesian CNNs have been made using space partitioning tree structures, such as Quadtree in 2D and Octree in 3D (Wang et al., 2017; Häne et al., 2017; Tatarchenko et al., 2017) . These Cartesian CNNs can leverage backbone CNN architectures being developed in related computer vision problems and thus achieve good performance. Since the physical domain representation in this study is based on Cartesian uniform grids, we directly use Cartesian CNNs.

Graph CNNs utilize input graph structure for performing graph convolutions. They have been developed to engage with general graph structured data (Bruna et al., 2013; Henaff et al., 2015; Defferrard et al., 2016) . Yi et al. (2017) used spectral CNNs with the eigenfunctions of the graph Laplacian as a basis. However, the generality of this approach across topologies and geometris is still challenging since consistency in eigenfunction basis is implied.

Specially Designed Neural Networks have been used to perform learning on unconventional data structures. For example, Qi et al. (2017a) designed a Neural Network architecture for points that achieves invariances using global pooling, with follow-up work (Qi et al., 2017b) using CNNsinspired hiearchical structures for more efficient learning. Masci et al. (2015) performed convolution directly on the shape manifold and Cohen et al. (2017) designed CNNs for the spherical domain and used it for 3D shapes by projecting the shapes onto the bounding sphere.

2.3. FOURIER TRANSFORM OF SHAPE FUNCTIONS:

The original work on analytical expressions for Fourier transforms of 2D polygonal shape functions is given by Lee & Mittra (1983) . Improved and simpler calculation methods have been suggested in Chu & Huang (1989) . A 3D formulation is proposed by Zhang & Chen (2001) . Theoretical analyses have been performed for the Fourier analysis of simplex domains (Sun, 2006; Li & Xu, 2009) and Sammis & Strain (2009) designed approximation methods for Fast Fourier Transform of polynomial functions defined on simplices. Prisacariu & Reid (2011) describe shape with elliptic Fourier descriptors for level set-based segmentation and tracking. There has also been a substantial literature on fast non-uniform Fourier transform methods for discretely sampled signal (Greengard & Lee, 2004 ). However we are the first to provide a simple general expression for a j-simplex mesh, an algorithm to perform the transformation, and illustrations of their applicability to deep learning problems.

3. REPRESENTATION OF SHAPE FUNCTIONS:



3.1. MATHEMATICAL PRELIMINARIES:

Almost all discrete geometric signals can be abstracted into weighted simplicial complexes. A simplicial complex is a set composed of points, line segments, triangles, and their d-dimensional counterparts. We call a simplicial complex consisting solely of j-simplices as a homogeneous simplicial j-complex, or a j-simplex mesh. Most popular geometric representations that the research community is familiar with are simplex meshes. For example, the point cloud is a 0-simplex mesh, the triangular mesh is a 2-simplex mesh, and the tetrahedral mesh is a 3-simplex mesh. A j-simplex mesh consists of a set of individual elements, each being a j-simplex. If signal is non-uniformly distributed over the simplex, we can define a piecewise constant j-simplex function over the j-simplex mesh.

We call this a weighted simplex mesh. Each element has a distinct signal density.

J-simplex function For the n-th j-simplex with domain Ω j n , we define a density function f j n (x). For example, for some Computer Vision and Graphics applications, a three-component density value can be defined on each element of a triangular mesh for its RGB color content. For scientific applications, signal density can be viewed as mass or charge or other physical quantity.

EQUATION

The piecewise-constant j-simplex function consisting of N simplices is therefore defined as the superposition of the element-wise simplex function. Using the linearity of the integral in the Fourier transform, we can decompose the Fourier transform of the density function on the j-simplex mesh to be a weighted sum of the Fourier transform on individual j-simplices.

F j (k) = • • • ∞ -∞ f j (x)e -ik•x dx = N n=1 ρ n • • • Ω j n e -ik•x dx :=F j n (k) = N n=1 ρ n F j n (k) (2) 3.2 SIMPLEX MESH FOURIER TRANSFORM

We present a general formula for performing the Fourier transform of signal over a single j-simplex.

We provide detailed derivation and proof for j = 0, 1, 2, 3 in the supplemental material.

F j n (k) = i j γ j n j+1 t=1 e -iσt j+1 l=1,l =t (σ t -σ l ) , σ t := k • x t (3)

We define γ j n to be the content distortion factor, which is the ratio of content between the simplex over the domain Ω j n and the unit orthogonal j-simplex. Content is the j-dimensional analogy of the 3-dimensional volume. The unit orthogonal j-simplex is defined as a j-simplex with one vertex at the Cartesian origin and all edges adjacent to the origin vertex to be pairwise orthogonal and to have unit length. Therefore from Equation (2) the final general expression for computing the Fourier transform of a signal defined on a weighted simplex mesh is:

F j (k) = N n ρ n i j γ j n j+1 t=1 e -iσt j+1 l=1,l =t (σ t -σ l ) (4)

For computing the simplex content, we use the Cayley-Menger Determinant for a general expression:

EQUATION

For the matrix Bj n , each entry d 2 mn represents the squared distance between nodes m and n. The matrix is of size (j + 2) × (j + 2) and is symmetrical. Since the unit orthogonal simplex has content of C j I = 1/j!, the content distortion factor γ j n can be calculated by:

EQUATION

Auxiliary Node Method: Equation (3) provides a mean of computing the Fourier transform of a simplex with uniform signal density. However, how do we compute the Fourier transform of polytopes (i.e., polygons in 2D, polyhedra in 3D) with uniform signal density efficiently? Here, we introduce the auxiliary node method (AuxNode) that utilizes signed content for efficient computing. We show that for a solid j-polytope represented by a watertight (j -1)-simplex mesh, we can compute the Fourier transform of the entire polytope by traversing each of the elements in its boundary (j -1)-simplex mesh exactly once (Zhang & Chen, 2001 ).

The auxiliary node method performs Fourier transform over the the signed content bounded by an auxilliary node (a convenient choice being the origin of the Cartesian coordinate system) and each (j -1)-simplex on the boundary mesh. This forms an auxiliary j-simplex:

Ω j n , n ∈ [0, N ],

where N is the number of (j -1)-simplices in the boundary mesh. However due to the overlapping of these auxiliary j-simplices, we need a means of computing the sign of the transform for the overlapping regions to cancel out. Equation (3) provides a general expression for computing the unsigned transform for a single j-simplex. It is trivial to show that since the ordering of the nodes does not affect the determinant in Equation ( 5), it gives the unsigned content value.

Therefore, to compute the Fourier transform of uniform signals in j-polytopes represented by its watertight (j -1)-simplex mesh using the auxiliary node method, we modify Equation (3): s n γ j n is the signed content distortion factor for the n th auxiliary j-simplex where s n ∈ {-1, 1}. For practical purposes, assume that the auxiliary j-simplex is in R d where d = j. We can compute the signed content distortion factor using the determinant of the Jacobian matrix for parameterizing the auxiliary simplex to a unit orthogonal simplex:

EQUATION

EQUATION

Since this method requires the boundary simplices to be oriented, the right-hand rule can be used to infer the correct orientation of the boundary element. For 2D polygons, it requires that the watertight boundary line mesh be oriented in a counter-clockwise fashion. For 3D polytopes, it requires that the face normals of boundary triangles resulting from the right-hand rule be consistently outward-facing.

Algorithmic implementation: Several efficiencies can be exploited to achieve fast runtime and high robustness. First, since the general expression Equation (4) involves division and is vulnerable to division-by-zero errors (that is not a singularity since it can be eliminated by taking the limit), add a minor random noise to vertex coordinates as well as to the k = 0 frequency mode for robustness. Second, to avoid repeated computation, the value should be cached in memory and reused, but caching all σ and e -iσ values for all nodes and frequencies is infeasible for large mesh and/or high resolution output, thus the Breadth-First-Search (BFS) algorithm should be used to traverse the vertices for efficient memory management.

4. EXPERIMENTS:

In this section, we will discuss the experiment setup, and we defer the details of our model architecture and training process to the supplementary material since it is not the focus of this paper.

4.1. MNIST WITH POLYGONS:

We use the MNIST experiment as a first example to show that shape information in the input significantly affects the efficacy of the downstream learning process. Since the scope of this research is on efficiently learning from nonuniform mesh-based representations, we compare our method with the state of the art in a slightly different scenario by treating MNIST characters as polygons. We choose this experiment as a first toy example since it is easy to control the learning architecture and input resolution to highlight the effects of shape representation on deep learning performance.

Experiment setup: We pre-process the original MNIST raw pixel images into polygons, which are represented by watertight line meshes on their boundaries. The polygonized digits are converted into (n × n) binary pixel images and into distance functions by uniformly sampling the polygon at the (n × n) sample locations. For NUFT, we first compute the lowest (n × n) Fourier modes for the polygonal shape function and then use an inverse Fourier transform to acquire the physical domain image. We also compare the results with the raw pixel image downsampled to different resolutions, which serves as an oracle for information ceiling. Then we perform the standard MNIST classification experiment on these representations with varying resolution n and with the same network architecture.

Results::

The experiment results are presented in Figure (3b) . It is evident that binary pixel representation suffers the most information loss, especially at low resolutions, which leads to rapidly declining performance. Using the distance function representation preserves more information, but underperforms our NUFT representation. Due to its efficient information compression in the Spectral domain, NUFT even outperforms the downsampled raw pixel image at low resolutions.

4.2. 3D SHAPE RETRIEVAL:

Shape retrieval is a classic task in 3D shape learning. SHREC17 (Savva et al., 2016) which is based on the ShapeNet55 Core dataset serves as a compelling benchmark for 3D shape retrieval performance. We compare the retrieval performance of our model utilizing the NUFT-surface (NUFT-S) and NUFT-volume (NUFT-V) at various resolutions against state-of-the-art shape retrieval algorithms to illustrate its potential in a range of 3D learning problems. We performed the experiments on the normalized dataset in SHREC17. Our model utilizes 3D DLA (Yu et al., 2018) as a backbone architecture.

Results: Results from the experiment are tabulated in Table 2 . For the shape retrieval task, most state-of-the-art methods are based on multi-view representation that utilize a 2D CNN pretrained on additional 2D datasets such as ImageNet. We have achieved results on par with, though not better than, state-of-the-art pretrained 2D models. We outperform other models in this benchmark that have not been pre-trained on additional data. We also compared NUFT-volume and NUFT-surface representations in Figure 4 . Interestingly NUFT-volume and NUFT-surface representations lead to similar performances under the same resolution. We achieve better results than the current stateof-the-art method by a sizable margin, and our results are robust to noise in the input. we can frame the task as a 3D image-to-image translation problem, which is easy to address using analogous 2D techniques in 3D. We use the U-Net (Ronneberger et al., 2015) architecture and train it with a single L 2 loss between output and ground truth NUFT-surface representation.

Experiment Setup: We train and test our model using shapes from three categories of ShapeNet55 Core dataset (car, sofa, bottle) . We trained our model individually for these three categories. As a pre-processing step we removed faces not visible from the exterior and simplified the mesh for faster conversion to NUFT representation. For the input point cloud, we performed uniform point sampling of 3000 points from each mesh and converted the points into the NUFT-point representation (128 3 ). Then, we converted the triangular mesh into NUFT-surface representation (128 3 ). At test time, we post-process the output NUFT-surface implicit function by using the marching cubes algorithm to extract 0.5 contours. Since the extracted mesh has thickness, we further shrink the mesh by moving vertices to positions with higher density values while preserving the rigidity of each face. Last but not least, we qualitatively and quantitatively compare the performance by showing the reconstructed mesh against results from the traditional Poisson Surface Reconstruction (PSR) method (Kazhdan & Hoppe, 2013) at various tree depths (5 and 8) and the Deep Marching Cubes (DMC) algorithm (Liao et al., 2018) . For quantitative comparison, we follow the literature (Seitz et al., 2006) and use Chamfer distance, Accuracy and Completeness as the metrics for comparison. For comparison with Liao et al. (2018) , we also test the model with noisy inputs (Gaussian of sigma 0.15 voxel-length under 32 resolution), computed distance metrics after normalizing the models to the range of (0, 32).

Results: Refer to Table 3 for quantitative comparisons with competing algorithms on the same task, and Figures 5 and 6 for visual comparisons. GT stands for Ground Truth. We achieve new state-of-the-art in the point to surface reconstruction task, due to the good localization properties of the NUFT representations and its flexibility across geometry topologies.

5. CONCLUSION:

We present a general representation for multidimensional signals defined on simplicial complexes that is versatile across geometrical deep learning tasks and maximizes the preservation of shape information. We develop a set of mathematical formulations and algorithmic tools to perform the transformations efficiently. Last but not least, we illustrate the effectiveness of the NUFT representation with a well-controlled example (MNIST polygon), a classic 3D task (shape retrieval) and a difficult and mostly unexplored task by deep learning (point to surface reconstruction), achieving new state-of-the-art performance in the last task. In conclusion, we offer an alternative representation for performing CNN based learning on geometrical signals that shows great potential in various 3D tasks, especially tasks involving mixed-topology signals.

y z x x 2 x 3 x 1 (a) Original j-simplex in R d space, d ≥ j. p q 1 1 (b) Unit orthogonal j-simplex in R j

Figure 7: Schematic of example for 2-simplex. Original j-simplex is parameterized to a unit orthogonal j-simplex in R j space for performing the integration. Parameterization incurs a content distortion factor γ j n which is the ratio between the original simplex and the unit-orthogonal simplex in parametric space.

A.3 TRIANGLE: 2-SIMPLEX:

For a triangle with vertices x 1 , x 2 , x 3 ∈ R d , parameterization onto a unit orthogonal triangle gives:

EQUATION

A.4 TETRAHEDRON: 3-SIMPLEX For a tetrahedron with vertices x 1 , x 2 , x 3 , x 4 ∈ R d , parameterization onto a unit orthogonal tetrahedron gives:

F 3 n (ω) = γ 3 n 1 0 dp 1-p 0 dq 1-p-q 0 dr e -iω•x(p) (20) = γ 3 n e -iω•x4 1 0 dp e -ipω•(x1-x4) 1-p 0 dq e -iqω•(x2-x4) 1-p-q 0 dre -irω•(x3-x4) (21) = -iγ 3 n e -iω•x1 ω 3 (x 1 -x 2 )(x 1 -x 3 )(x 1 -x 3 ) + e -iω•x2 ω 3 (x 2 -x 1 )(x 2 -x 3 )(x 2 -x 4 ) + e -iω•x3 ω 3 (x 3 -x 1 )(x 3 -x 2 )(x 3 -x 4 ) + e -iω•x4 ω 3 (x 4 -x 1 )(x 4 -x 2 )(x 4 -x 3 ) (22)

B COMPARISON OF GEOMETRICAL SHAPE INFORMATION:

Besides evaluating and comparing the shape representation schemes in the context of machine learning problems, we evaluate the different representation schemes in terms of its geometrical shape information. To evaluate geometrical shape information, we first convert the original polytopal shapes into the corresponding representations and then reconstruct the shape from these representations using interpolation based upsampling followed by contouring methods. Finally, we use the mesh Intersection over Union (mesh-IoU) metric between the original and constructed mesh to quantify geometrical shape information. Mesh boolean and volume computation for 2D polygons and 3D triangular mesh can be efficiently performed with standard computational geometry methods. In three dimensions contouring and mesh extraction can be performed using marching cubes algorithm for mesh reconstruction. For binarized representation, we perform bilinear upsampling (which does not affect the final result) followed by 0.5-contouring. For our NUFT representation, we perform spectral domain upsampling (which corresponds to zero-padding of higher modes in spectral domain), followed by 0.5-contouring. Qualitative side-by-side comparisons are presented for visual inspection, and qualitative empirical evaluation is performed for the Bunny Mesh (1K faces) model. Refer to Figures 8, 9 [32, 64, 256, 512] filter numbers. We keep the architecture constant while varying the input resolution.

Training Details: We train the model with batch size of 64, learning rate of 0.01 and learning rate step size of 10. We use the SGD optimizer with momentum of 0.9, weight decay of 1 × 10 -4 for 15 epochs.

C.2 3D SHAPE RETRIEVAL:

Model Architecture: We use DLA34 architecture with all 2D convolutions modified to be 3D convolutions. It consists of [1,1,1,2,2,1] levels, with [16, 32, 64, 128, 256, 512] filter numers.

Training Details We train the model with batch size of 64, learning rate of 1 × 10 -3 , learning rate step size of 30. We use the Adam optimizer with momentum of 0.9, and weight decay of 1 × 10 -4 for 40 epochs.

C.3 3D SURFACE RECONSTRUCTION FROM POINT CLOUD:

Model Architecture We use a modified 3D version of the U-Net architecture consisting of 4 down convolutions and 4 up-convolutions with skip layers. Number of filters for down convolutions are [32, 64, 128, 256] and double of that for up convolutions.

Training Details We train the model using Adam optimizer with learning rate 3 × 10 -4 for 200 epochs. We use NUFT point representation as input and a single L 2 loss between output and ground truth (NUFT surface) to train the network. We train and evaluate the model at 128 3 resolution.

:



Figures and Tables:

Figure 1: Top: Schematic of the NUFT transformations of the Stanford Bunny model. Bottom: Schematic for shape retrieval and surface reconstruction experiments.

Figure 2: Surface localization in (a) binary pixel/voxel representation, where the boundary can only be in one of 2 4 (2D) or 2 8 (3D) discrete locations (b) Signed Distance Function representation, where boundary is linear (c) proposed representation, with nonlinear localization of boundary, achieving subgrid accuracy

Figure 3: MNIST experiment. (a) schematic for experiment setup. The original MNIST pixel image is up-sampled using interpolation and contoured to get a polygonal representation of the digit. For the polygon, it is transformed into binary pixels, distance functions, and the NUFT physical domain. (b) classification accuracy versus input resolution under various representation schemes. NUFT representation is more optimal, irrespective of resolution.

Figure 5: Zoom-in comparison.Method Chamfer Accuracy Complete DMC 0.218 0.182 0.254 PSR-5 0.352 0.405 0.298 PSR-8 0.198 0.196 0.200 Ours(w/ Noise) 0.144 0.150 0.137 Ours(w/o Noise) 0.145 0.125 0.165

Figure 6: Qualitative side-by-side comparison of surface reconstruction results.

, and 10. C MODEL ARCHITECTURE AND TRAINING DETAILS C.1 MNIST WITH POLYGONS Model Architecture: We use the state-of-the-art Deep Layer Aggregation (DLA) backbone architecture with [1,1,2,1] levels and



Table content: <table><tr><td>: Notation list</td></tr></table>

Quantitative comparison of surface reconstruction methods. The metrics above are distances, hence lower value represents better performance. Best result is highlighted in bold.

Table content: <table/>

Peer review (iclr2019):
 Overall Thoughts: I think this paper addresses an interesting topic and that the community as a whole are interested on the application of learning algorithms to non-Euclidean domains. It is nice to see the application of Fourier sampling to geometric primitives in a sensible manner and I am positive about that part of the paper. However, in its current form, I have quite a few questions about the approach and the empirical studies - I would need to here more information from the authors on the points below. Specific Comments/Questions: Sec1: The authors make a number of assertions about the representational errors that occur in other approaches - I feel that these claims should be supported by specific references. Contributions: It is not clear to me that the experiments show that the method “preserves maximal information content” - in my understanding, information content has a formal definition and I don’t see where this is presented in the results? Sec2: Before CNNs there has been a substantial analysis of Fourier methods applied to shape models, e.g. elliptical Fourier series for shape contours (and signed distance representations) by Prisacariu et al. Sec2: It is also worth noting that there is substantial literature on non-uniform Fourier methods including the non-uniform Fourier transform and a number of accelerations (e.g. NUFFT) as well as consideration of the implications of band-limiting the sampled spectra. Sec3: The mathematical derivation all makes sense to me and makes use of results for piecewise uniform signals. Please could the authors provide more details on how the spectra are represented? These discontinuous signals (esp. delta functions) will have infinite bandwidth in the spectral domain so how they are stored would seem very important to me. Are the signals band limited at some point? If so, how does this affect the approximation and should filtering/windowing be used? Otherwise is there not a difficult storage problem? The final paragraph suggests all the analytic signals might be stored but this has a big impact on how efficient the algorithm is and really is far too important a point to just have a single sentence - please can the authors expand on how this is actually implemented and what the computational considerations are (and resulting impacts on performance)? Sec4: Please can the authors add error bars (at the least) to all the tables/plots in the results. It is entirely unreasonable to make any statements about how significant the results may be without even the most basic of analysis. Ideally we should see histograms of the results for the retrieval and shape reconstruction results. Sec4: How is the downsampling performed in the MNIST experiment? It would seem very important to take care with this for the purpose of comparison. A significant disparity between the signed distance function and the NUFT would seem slightly surprising to me? Again, without error bars we really cannot say very much about the results on the right of Fig3(b). Sec4: Could the Fig4 results not be provided as histograms? We also need many more details about how the results were obtained and procedures to ensure that the results are meaningful and robust (e.g. repeated tests and partitions of the data, etc..) Arch and Training Details: How are we to know that these choices provide fair comparisons to previous approaches? Sec3/4: All the results seem to require the input to be reconstructed back into a dense sampling domain (via inverse FT) - is this the case? Would it not be more efficient to perform the convolutions in the spectral domain where the signal is sparse? Sec4: It seems pretty unfair to train and test on a single category of shapes in the shape test since the data is known not to be very diverse? Particularly when, unless I’ve misunderstood, the baselines on the shape recovery test to not involve learning and so (while helpful to have) they are not really fair baselines compared to other learning approaches? Also, please can the authors provide much more information about the extra processing applied (e.g. the part starting with the “extra mesh thickness”) since there seem to be some extra steps that are nothing to do with the rest of the method and may impact the results significantly. Sec4: It is interesting that Table 3 (again difficult to say without error bars) indicates that there are times when the method performs better with the additional of noise - this seems counter-intuitive - please could the authors comment on this? Other Points: I’m afraid that there are quite a few grammatical errors in the text (too many to list here) so I would recommend another round of proof-reading.