Title: VOLUMETRIC CONVOLUTION: AUTOMATIC REPRE-SENTATION LEARNING IN UNIT BALL

Abstract: Convolution is an efficient technique to obtain abstract feature representations using hierarchical layers in deep networks. Although performing convolution in Euclidean geometries is fairly straightforward, its extension to other topological spaces-such as a sphere (S 2 ) or a unit ball (B 3 )-entails unique challenges. In this work, we propose a novel 'volumetric convolution' operation that can effectively convolve arbitrary functions in B 3 . We develop a theoretical framework for volumetric convolution based on Zernike polynomials and efficiently implement it as a differentiable and an easily pluggable layer for deep networks. Furthermore, our formulation leads to derivation of a novel formula to measure the symmetry of a function in B 3 around an arbitrary axis, that is useful in 3D shape analysis tasks. We demonstrate the efficacy of proposed volumetric convolution operation on a possible use-case i.e., 3D object recognition task.

1. INTRODUCTION:

Convolution-based deep neural networks have performed exceedingly well on 2D representation learning tasks (Krizhevsky et al., 2012; He et al., 2016) . The convolution layers perform parameter sharing to learn repetitive features across the spatial domain while having lower computational cost by using local neuron connectivity. However, most state-of-the-art convolutional networks can only work on Euclidean geometries and their extension to other topological spaces e.g., spheres, is an open research problem. Remarkably, the adaptation of convolutional networks to spherical domain can advance key application areas such as robotics, geoscience and medical imaging. Some recent efforts have been reported in the literature that aim to extend convolutional networks to spherical signals. Initial progress was made by Boomsma & Frellsen (2017) , who performed conventional planar convolution with a careful padding on a spherical-polar representation and its cube-sphere transformation (Ronchi et al., 1996) . A recent pioneering contribution by Cohen et al. (2018) used harmonic analysis to perform efficient convolution on the surface of the sphere (S 2 ) to achieve rotational equivariance. These works, however, do not systematically consider radial information in a 3D shape and the feature representations are learned at specified radii. Specifically, Cohen et al. (2018) estimated similarity between spherical surface and convolutional filter in S 2 , where the kernel can be translated in S 2 . Furthermore, Weiler et al. (2018) recently solved the more general problem of SE(3) equivariance by modeling 3D data as dense vector fields in 3D Euclidean space. In this work however, we focus on B 3 to achieve the equivariance to SO(3).

In this paper, we propose a novel approach to perform volumetric convolutions inside unit ball (B 3 ) that explicitly learns representations across the radial axis. Although we derive generic formulas to convolve functions in B 3 , we experiment on one possible use case in this work, i.e., 3D shape recognition. In comparison to closely related spherical convolution approaches, modeling and convolving 3D shapes in B 3 entails two key advantages: 'volumetric convolution' can capture both 2D texture and 3D shape features and can handle non-polar 3D shapes. We develop the theory of volumetric convolution using orthogonal Zernike polynomials (Canterakis, 1999) , and use careful approximations to efficiently implement it using low computational-cost matrix multiplications. Our experimental results demonstrate significant boost over spherical convolution and that confirm the high discriminative ability of features learned through volumetric convolution.

Furthermore, we derive an explicit formula based on Zernike Polynomials to measure the axial symmetry of a function in B 3 , around an arbitrary axis. While this formula can be useful in many function analysis tasks, here we demonstrate one particular use-case with relevance to 3D shape recognition. Specifically, we use the the derived formula to propose a hand-crafted descriptor that accurately encodes the axial symmetry of a 3D shape. Moreover, we decompose the implementation of both volumetric convolution and axial symmetry measurement into differentiable steps, which enables them to be integrated to any end-to-end architecture.

Finally, we propose an experimental architecture to demonstrate the practical usefulness of proposed operations. We use a capsule network after the convolution layer as it allows us to directly compare feature discriminability of spherical convolution and volumetric convolution without any bias. In other words, the optimum deep architecture for spherical convolution may not be the same for volumetric convolution. Capsules, however, do not deteriorate extracted features and the final accuracy only depends on the richness of input shape features. Therefore, a fair comparison between spherical and volumetric convolutions can be done by simply replacing the convolution layer.

It is worth pointing out that the proposed experimental architecture is only a one possible example out of many possible architectures, and is primarily focused on three factors: 1) Capture useful features with a relatively shallow network compared to state-of-the-art. 2) Show richness of computed features through clear improvements over spherical convolution. 3) Demonstrate the usefulness of the volumetric convolution and axial symmetry feature layers as fully differentiable and easily pluggable layers, which can be used as building blocks for end-to-end deep architectures.

The main contributions of this work include:

• Development of the theory for volumetric convolution that can efficiently model functions in B 3 .

• Implementation of the proposed volumetric convolution as a fully differentiable module that can be plugged into any end-to-end deep learning framework. • The first approach to perform volumetric convolution on 3D objects that can simultaneously model 2D (appearance) and 3D (shape) features. • A novel formula to measure the axial symmetry of a function defined in B 3 , around an arbitrary axis using Zernike polynomials. • An experimental end-to-end trainable framework that combines hand-crafted feature representation with automatically learned representations to obtain rich 3D shape descriptors.

The rest of the paper is structured as follows. In Sec. 2 we introduce the overall problem and our proposed solution. Sec. 3 presents an overview of 3D Zernike polynomials. Then, in Sec. 4 and Sec. 5 we derive the proposed volumetric convolution and axial symmetry measurement formula respectively. Sec. 6.2 presents our experimental architecture, and in Sec. 7 we show the effectiveness of the derived operators through extensive experiments. Finally, we conclude the paper in Sec. 8.

2. PROBLEM DEFINITION:

Convolution is an effective method to capture useful features from uniformly spaced grids in R n , within each dimension of n, such as gray scale images (R 2 ), RGB images (R 3 ), spatio-temporal data (R 3 ) and stacked planar feature maps (R n ). In such cases, uniformity of the grid within each dimension ensures the translation equivariance of the convolution. However, for topological spaces such as S 2 and B 3 , it is not possible to construct such a grid due to non-linearity. A naive approach to perform convolution in B 3 would be to create a uniformly spaced three dimensional grid in (r, θ, φ) coordinates (with necessary padding) and perform 3D convolution. However, the spaces between adjacent points in each axis are dependant on their absolute position and hence, modeling such a space as a uniformly spaced grid is not accurate.

To overcome these limitations, we propose a novel volumetric convolution operation which can effectively perform convolution on functions in B 3 . It is important to note that ideally, the convolution in B 3 should be a signal on both 3D rotation group and 3D translation. However, since Zernike polynomials do not have the necessary properties to automatically achieve translation equivariance, we stick to 3D rotation group in this work and refer to this operation as convolution from here onwards. Fig. 1 shows the analogy between planar convolution and volumetric convolution. In Sec. 3, we present an overview of 3D Zernike polynomials that will be later used in Sec. 4 to develop volumetric convolution operator.

3. 3D ZERNIKE POLYNOMIALS:

3D Zernike polynomials are a complete and orthogonal set of basis functions in B 3 , that exhibits a 'form invariance' property under 3D rotation (Canterakis, 1999) . A (n, l, m) th order 3D Zernike basis function is defined as,

EQUATION

where R n,l is the Zernike radial polynomial (Appendix D.3), Y l,m (θ, φ) is the spherical harmonics function (Appendix D.1), n ∈ Z + , l ∈ [0, n], m ∈ [-l, l] and n -l is even. Since 3D Zernike polynomials are orthogonal and complete in B 3 , an arbitrary function f (r, θ, φ) in B 3 can be approximated using Zernike polynomials as follows.

EQUATION

where Ω n,l,m (f ) could be obtained using,

Ω n,l,m (f ) = 1 0 2π 0 π 0 f (θ, φ, r)Z † n,l,m r 2 sinφdrdφdθ (3)

where † denotes the complex conjugate. In Sec. 4, we will derive the proposed volumetric convolution.

4. VOLUMETRIC CONVOLUTION OF FUNCTIONS IN B 3:

When performing convolution in B 3 , a critical problem which arises is that several rotation operations exist for mapping a point p to a particular point p . For example, using Euler angles, we can decompose a rotation into three rotation operations R(θ, φ) = R(θ) y R(φ) z R(θ) y , and the first rotation R(θ) y can differ while mapping p to p (if y is the north pole). However, if we enforce the kernel function to be symmetric around y, the function of the kernel after rotation would only depend on p and p . This observation is important for our next derivations because we can then uniquely define a 3D rotation on kernel in terms of azimuth and polar angles.

Let the kernel be symmetric around y and f (θ, φ, r), g(θ, φ, r) be the functions of object and kernel respectively. Then we define volumetric convolution as,

f * g(α, β) := f (θ, φ, r), τ (α,β) (g(θ, φ, r)) = 1 0 2π 0 π 0 f (θ, φ, r), τ (α,β) (g(θ, φ, r)) sin φdφdθdr

(4) where τ (α,β) is an arbitrary rotation, that aligns the north pole with the axis towards (α, β) direction (α and β are azimuth and polar angles respectively). Eq. 4 is able to capture more complex patterns compared to spherical convolution due to two reasons: 1) the inner product integrates along the radius and 2) the projection onto spherical harmonics forces the function into a polar function, that can result in information loss.

In Sec. 4.1 we derive differentiable relations to compute 3D Zernike moments for functions in B 3 . 4.1 SHAPE MODELING OF FUNCTIONS IN B 3 USING 3D ZERNIKE POLYNOMIALS Instead of using Eq. 3, we derive an alternative method to obtain the set {Ω n,l,m }. The motivations are two fold: 1) ease of computation and 2) the completeness property of 3D Zernike Polynomials ensures that lim n→∞ fn l m Ω n,l,m Z n,l,m = 0 for any arbitrary function f . However, since n should be finite in the implementation, aforementioned property may not hold, leading to increased distance between the Zernike representation and the original shape. Therefore, minimizing the reconstruction error

(θ,φ,r)∈S 3 f (θ, φ, r) -f (θ, φ, r) where f (θ, φ, r) = N n l

m Ω n,l,m Z n,l,m , pushes the set {Ω n,l,m } inside frequency space, where {Ω n,l,m } has a closer resemblance to the corresponding shape. Following this conclusion, we derive the following method to obtain {Ω n,l,m }. In volumetric convolution a 3D rotation is applied to the kernel and the inner product is computed between 3D function and 3D kernel over

B 3 . Since Y l,m (θ, φ) = (-1) m 2l+1 4π (l-m)! (l+m)! P m l (cosφ)e imθ , where P m l (•) is the associated Legendre function (Appendix D.2), it can be deduced that, Y l,-m (θ, φ) = (-1) m Y † l,m (θ, φ). Using this relationship we obtain Z n,l,-m (θ, φ) = (-1) m Z †

n,l,m (θ, φ) and hence approximate Eq. 2 as,

EQUATION

where Re{Z n,l,m } and Img{Z n,l,m } are real and imaginary components of Z n,l,m respectively. In matrix form, this can be rewritten as,

f (θ, φ, r) = U a + V b = f (θ, φ, r) = Xc (6

) where c is the set of 3D Zernike moments Ω n,l,m . Eq. 6 can be interpreted as an overdetermined linear system, with the set Ω n,l,m as the solution. To find the least squared error solution to the Eq. 6 we use the pseudo inverse of X. Since this operation has to be differentiable to train the model end-to-end, a common approach like singular value decomposition cannot be used here. Instead, we use an iterative method to calculate the pseudo inverse of a matrix (Li et al., 2011) . It has been shown that V n converges to A + where A + is the Moore-Penrose pseudo inverse of A if,

EQUATION

7) for a suitable initial approximation V 0 . They also showed that a suitable initial approximation would be V 0 = αA T with 0 < α < 2/ρ(AA T ), where ρ(•) denotes the spectral radius. Empirically, we choose α = 0.001 in our experiments. Next, we derive the theory of volumetric convolution within the unit ball.

4.2. CONVOLUTION IN B 3 USING 3D ZERNIKE POLYNOMIALS:

We formally present our derivation of volumetric convolution using the following theorem. A short version of the proof is then provided. Please see Appendix A for the complete derivation.

Theorem 1: Suppose f, g : X -→ R 3 are square integrable complex functions defined in B 3 so that f, f < ∞ and g, g < ∞. Further, suppose g is symmetric around north pole and

τ (α, β) = R y (α)R z (β) where R ∈ SO(3). Then, 1 0 2π 0 π 0 f (θ, φ, r), τ (α,β) (g(θ, φ, r)) sin φdφdθdr ≡ 4π 3 ∞ n=0 n l=0 l m=-l Ω n,l,m (f )Ω n,l,0 (g)Y l,m (θ, φ) (8)

where Ω n,l,m (f ), Ω n,l,0 (g) and Y l,m (θ, φ) are (n, l, m) th 3D Zernike moment of f , (n, l, 0) th 3D Zernike moment of g, and spherical harmonics function respectively.

Proof: Completeness property of 3D Zernike Polynomials ensures that it can approximate an arbitrary function in B 3 , as shown in Eq. 2. Leveraging this property, Eq. 4 can be rewritten as,

f * g(θ, φ) = ∞ n=0 n l=0 l m=-l Ω n,l,m (f )Z n,l,m , τ (θ,φ) ( ∞ n =0 n l =0 l m =-l Ω n ,l ,m (g)Z n ,l ,m ) (9)

However, since g(θ, φ, r) is symmetric around y, the rotation around y should not change the function. This ensures, g(r, θ, φ) = g(r, θ -α, φ) (10) and hence,

∞ n =0 n l =0 l m =-l Ω n ,l ,m (g)R n ,l (r)Y l ,m = ∞ n =0 n l =0 l m =-l Ω n ,l ,m (g)R n ,l (r)Y l ,m e -im α

(11) This is true, if and only if m = 0. Therefore, a symmetric function around y, defined inside the unit sphere can be rewritten as,

EQUATION

which simplifies Eq. 9 to,

EQUATION

Using the properties of inner product, Eq. 13 can be rearranged as,

EQUATION

Using the rotational properties of Zernike polynomials, we obtain (see Appendix A for our full derivation),

EQUATION

Since we can calculate Ω n,l,m (f ) and Ω n,l,0 (g) easily using Eq. 6, f * g(θ, φ) can be found using a simple matrix multiplication. It is interesting to note that, since the convolution kernel does not translate, the convolution produces a polar shape, which can be further convolved-if needed-using the

relationship f * g(θ, φ) = 4π 2l+1 l l m=-l f (l, m)ĝ(l, m)Y (l,m) (θ, φ)

where, f (l, m) and ĝ(l, m)

are the (l, m) th frequency components of f and g in spherical harmonics space. Next, we present a theorem to show the equivariance of volumetric convolution with respect to 3D rotation group.

4.3. EQUIVARIANCE TO 3D ROTATION GROUP:

One key property of the proposed volumetric convolution is its equivariance to 3D rotation group. To demonstrate this, we present the following theorem.

Theorem 1: Suppose f, g : X -→ R 3 are square integrable complex functions defined in B 3 so that f, f < ∞ and g, g < ∞. Also, let η α,β,γ be a 3D rotation operator that can be decomposed into three Eular rotations R y (α)R z (β)R y (γ) and τ α,β another rotation operator that can be decomposed

into R y (α)R z (β). Suppose η α,β,γ (g) = τ α,β (g). Then, η (α,β,γ) (f ) * g(θ, φ) = τ (α,β) (f * g)(θ, φ),

where * is the volumetric convolution operator.

The proof to our theorem can be found in Appendix B. The intuition behind the theorem is that if a 3D rotation is applied to a function defined in B 3 Hilbert space, the output feature map after volumetric convolution exhibits the same rotation. The output feature map however, is symmetric around north pole, hence the rotation can be uniquely defined in terms of azimuth and polar angles.

5. AXIAL SYMMETRY OF FUNCTIONS IN B 3:

In this section we present the following proposition to obtain the axial symmetry measure of a function in B 3 , around an arbitrary axis using 3D Zernike polynomials. In contrast, spherical convolution is performed in S 2 that can handle only polar 3D shapes with uniform texture.

Proposition: Suppose g : X -→ R 3 is a square integrable complex function defined in B 3 such that g, g < ∞. Then, the power of projection of g in to S = {Z i } where S is the set of Zernike basis functions that are symmetric around an axis towards (α, β) direction is given by,

||sym (α,β) ||= n n l=0 || l m=-l Ω n,l,m Y m,l (α, β)|| 2 (16)

where α and β are azimuth and polar angles respectively.

The proof to our proposition is given in Appendix C.

6 A CASE STUDY: 3D OBJECT RECOGNITION

6.1. 3D OBJECTS AS FUNCTIONS IN B 3:

A 2D image is a function on Cartesian plane, where a unique value exists for any (x, y) coordinate.

Similarly, a polar 3D object can be expressed as a function on the surface of the sphere, where any direction vector (θ, φ) has a unique value. To be precise, a 3D polar object has a boundary function in the form of f :

S 2 → [0, ∞].

Translation of the convolution kernel on (x, y) plane in 2D case, extends to movements on the surface of the sphere in S 2 . If both the object and the kernel have polar shapes, this task can be tackled by projecting both the kernel and the object onto spherical harmonic functions (Appendix E). However, this technique suffers from two drawbacks. 1) Since spherical harmonics are defined on the surface of the unit sphere, projection of a 3D shape function into spherical harmonics approximates the object to a polar shape, which can cause critical loss of information for non-polar 3D shapes. This is frequently the case in realistic scenarios.

2) The integration happens over the surface of the sphere, which is unable to capture patterns across radius.

These limitations can be addressed by representing and convolving the shape function inside the unit ball (B 3 ). Representing the object function inside B 3 allows the function to keep its complex shape information without any deterioration since each point is mapped to unique coordinates (r, θ, φ), where r is the radial distance, θ and φ are azimuth and polar angles respectively. Additionally, it allows encoding of 2D texture information simultaneously. Figure 2 compares volumetric convolution and spherical convolution. Since we conduct experiments only on 3D objects with uniform surface values, in this work we use the following transformation to apply a simple surface function f (θ, φ, r) to the 3D objects:

EQUATION

6.2. AN EXPERIMENTAL ARCHITECTURE:

We implement an experimental architecture to demonstrate the usefulness of the proposed operations. While these operations can be used as building-tools to construct any deep network, we focus on three key factors while developing the presented experimental architecture: 1) Shallowness: Volumetric convolution should be able to capture useful features compared to other methodologies with less number of layers. 2) Modularity: The architecture should have a modular nature so that a fair comparison can be made between volumetric and spherical convolution. We use a capsule network after the convolution layer for this purpose. For each angle, axial symmetry and volumetric convolution features are generated for P + and P -. These two features are then separately combined using compact bilinear pooling. Finally, the features are fed to two individual capsule networks, and the decisions are max-pooled. of axial symmetry features as a hand-crafted and fully differentiable layer. The motivation is to demonstrate one possible use case of axial symmetry measurements in 3D shape analysis.

The proposed architecture consists of four components. First, we obtain three view angles, and later generate features for each view angle separately. We optimize the view angles to capture complimentary shape details such that the total information content is maximized. For each viewing angle 'k', we obtain two point sets P + k and P - k consisting of tuples denoted as:

EQUATION

such that y denotes the horizontal axis. Second, the six point sets are volumetrically convolved with kernels to capture local patterns of the object. The generated features for each point set are then combined using compact bilinear pooling. Third, we use axial symmetry measurements to generate additional features. The features that represent each point set are then combined using compact bilinear pooling. Fourth, we feed features from second and third components of the overall architecture to two independent capsule networks and combine the outputs at decision level to obtain the final prediction. The overall architecture of the proposed scheme is shown in Fig. 3 .

6.3. OPTIMUM VIEW ANGLES:

We use three view angles to generate features for better representation of the object. First, we translate the center of mass of the set of (x, y, z) points to the origin. The goal of this step is to achieve a general translational invariance, which allows us to free the convolution operation from the burden of detecting translated local patterns. Subsequently, the point set is rearranged as an ordered set on x and z and a 1D convolution net is applied on y values of the points. Here, the objective is to capture local variations of points along the y axis, since later we analyze point sets P + and P -independently. The trained filters can be assumed to capture properties similar to ∂ n y/∂x n and ∂ n y/∂z n , where n is the order of derivative. The output of the 1D convolution net is rotation parameters represented by a 1 × 9 vector r = {r 1 , r 2 , • • • , r 9 }. Then, we compute

R 1 = R x (r 1 )R y (r 2 )R z (r 3 ), R 2 = R x (r 4 )R y (r 5 )R z (r 6 ) and R 3 = R x (r 7 )R y (r 8 )R z (r 9 )

where R 1 , R 2 and R 3 are the rotations that map the points to three different view angles.

After mapping the original point set to three view angles, we extract the P + k and P - k point sets from each angle k that gives us six point sets. These sets are then fed to the volumetric convolution layer to obtain feature maps for each point set. We then measure the symmetry around four equi-angular axes using Eq. 16, and concatenate these measurement values to form a feature vector for the same point sets.

6.4. FEATURE FUSION USING COMPACT BILINEAR POOLING:

Compact bilinear pooling (CBP) provides a compact representation of the full bilinear representation, but has the same discriminative power. The key advantage of compact bilinear pooling is the significantly reduced dimensionality of the pooled feature vector.

We first concatenate the obtained volumetric convolution features of the three angles, for P + and P - separately to establish two feature vectors. These two features are then fused using compact bilinear pooling (Gao et al., 2016) . The same approach is used to combine the axial symmetry features. These fused vectors are fed to two independent capsule nets.

Furthermore, we experiment with several other feature fusion techniques and present results in Sec. 7.2.

6.5. CAPSULE NETWORK:

Capsule Network (CapsNet) (Sabour et al., 2017) brings a new paradigm to deep learning by modeling input domain variations through vector based representations. CapsNets are inspired by so-called inverse graphics, i.e., the opposite operation of image rendering. Given a feature representation, CapsNets attempt to generate the corresponding geometrical representation. The motivation for using CapsNets in the network are twofold: 1) CapsNet promotes a dynamic 'routing-by-agreement' approach where only the features that are in agreement with high-level detectors are routed forward. This property of CapsNets does not deteriorate extracted features and the final accuracy only depends on the richness of original shape features. It allows us to directly compare feature discriminability of spherical and volumetric convolution without any bias. For example, using multiple layers of volumetric or spherical convolution hampers a fair comparison since it can be argued that the optimum architecture may vary for two different operations. 2) CapsNet provides an ideal mechanism for disentangling 3D shape features through pose and view equivariance while maintaining an intrinsic co-ordinate frame where mutual relationships between object parts are preserved.

Inspired by these intuitions, we employ two independent CapsNets in our network for volumetric convolution features and axial symmetry features. In this layer, we rearrange the input feature vectors as two sets of primary capsules-for each capsule net-and use the dynamic routing technique proposed by Sabour et al. (2017) to predict the classification results. The outputs are then combined using max-pooling, to obtain the final classification result. For volumetric convolution features, our architecture uses 1000 primary capsules with 10 dimensions each. For axial symmetry features, we use 2500 capsules, each with 10 dimensions. In both networks, decision layer consist of 12 dimensional capsules.

6.6. HYPERPARAMETERS:

We use n = 5 to implement Eq. 15 and three iterations to calculate the Moore-Penrose pseudo inverse using Eq. 7. We use a decaying learning rate lr = 0.1 × 0.9 g step 3000 , where g step is incremented by one per each iteration. For training, we use the Adam optimizer with β 1 = 0.9, β 2 = 0.999, = 1 × 10 -8 where parameters refer to the usual notation. All these values are chosen empirically. Since we have decomposed the theoretical derivations into sets of low-cost matrix multiplications, specifically aiming to reduce the computational complexity, the GPU implementation is highly efficient. For example, the model takes less than 15 minutes for an epoch during the training phase for ModelNet10, with a batchsize 2, on a single GTX 1080Ti GPU.

7. EXPERIMENTS:

In this section, we discuss and evaluate the performance of the proposed approach. We first compare the accuracy of our model with relevant state-of-the-art work, and then present a thorough ablation study of our model, that highlights the importance of several architectural aspects. We use Model-Net10 and ModelNet40 datasets in our experiments. Next, we evaluate the robustness of our approach against loss of information and finally show that the proposed approach for computing 3D Zernike moments produce richer representations of 3D shapes, compared to the conventional approach.

7.1. COMPARISON WITH THE STATE-OF-THE-ART:

Table 1 illustrates the performance comparison of our model with state-of-the-art. The model attains an overall accuracy of 92.17% on ModelNet10 and 86.5% accuracy on ModelNet40, which is on par with state-of-the-art. We do not compare with other recent work, such as Kanezaki et al. (2016) (Simonovsky & Komodakis, 2017) 4Conv + 1FC --83.2% DeepPano (Shi et al., 2015) 4Conv + 3FC -85.45% 77,63% 3DShapeNets (Wu et al., 2015) 4-3DConv + 2FC 38M 83.5% 77% PointNet (Garcia-Garcia et al., 2016) 2Conv + 2FC 80M 77.6% - ModelNet10 and ModelNet40. These are not comparable with our proposed approach, as we propose a shallow, single model without any data augmentation, with a relatively low number of parameters. Furthermore, our model reports these results by using only a single volumetric convolution layer for learning features. Fig. 4 demonstrates effectiveness of our architecture by comparing accuracy against the number of trainable parameters in state-of-the-art models. 

7.2. ABLATION STUDY:

Table 3 depicts the performance comparison between several variants of our model. To highlight the effectiveness of the learned optimum view points, we replace the optimum view point layer with three fixed orthogonal view points. This modification causes an accuracy drop of 6.57%, emphasizing that the optimum view points indeed depends on the shape. Another interestingperhaps the most important-aspect to study is the performance of the proposed volumetric convolution against spherical convolution. To this end, we replace the volumetric convolution layer of our model with spherical convolution and compare the results. It can be seen that our volumetric convolution scheme outperforms spherical convolution by a significant margin of 12.56%, indicating that volumetric convolution captures shape properties more effectively. Furthermore, using mean-pooling instead of maxpooling, at the decision layer drops the accuracy to 87.27%. We also evaluate performance of using a single capsule net. In this scenario, we combine axial symmetry features with volumetric convolution features using compact bilinear pooling (CBP), and feed it a single capsule network. This variant achieves an overall accuracy of 86.53%, is a 5.64% reduction in accuracy compared to the model with two capsule networks.

Moreover, we compare the performance of two feature categories-volumetric convolution features and axial symmetry features-individually. Axial symmetry features alone are able to obtain an accuracy of 66.73%, while volumetric convolution features reach a significant 85.3% accuracy. On the contrary, spherical convolution attains an accuracy of 71.6%, which again highlights the effectiveness of volumetric convolution.

Then we compare between different choices that can be applied to the experimental architecture. We first replace the capsule network with a fully connected layer and achieve an accuracy of 87.3%. This is perhaps because capsules are superior to a simple fully connected layer in modeling view- Our Zernike frequencies computation approach has far less error than the conventional approach. point invariant representations. Then we try different substitutions for compact bilinear pooling and achieve 90.7%, 90.3% and 85.3% accuracies respectively for feature concatenation, max-pooling and average-pooling. This justifies the choice of compact bilinear pooling as a feature fusion tool. However, it should be noted that these choices may differ depending on the architecture.

7.3. ROBUSTNESS AGAINST INFORMATION LOSS:

One critical requirement of a 3D object classification task is to be robust against various information loss. To demonstrate the effectiveness of our proposed features in this aspect, we randomly remove data points from the objects in validation set, and evaluate model performance. The results are illustrated in Fig. 5 . The model shows no performance loss until 20% of the data is lost, and only gradually drops to an accuracy level of 66.5 at a 50% data loss, which implies strong robustness against random information loss.

7.4. EFFECTIVENESS OF THE PROPOSED METHOD FOR CALCULATING 3D ZERNIKE MOMENTS:

In Sec. 4.1, we proposed an alternative method to calculate 3D Zernike moments (Eq. 5, 6), instead of the conventional approach (Eq. 3). We hypothesized that moments obtained using the former has a closer resemblance to the original shape, due to the impact of finite number of frequency terms. In this section, we demonstrate the validity of our hypothesis through experiments. To this end, we compute moments for the shapes in the validation set of ModelNet10 using both approaches, and compare the mean reconstruction error defined as:

1 T T t f (t) -n l m Ω n,l,m Z n,l,m (t) ,

where T is the total number of points and t ∈ S 3 . Fig. 6 shows the results. In both approaches, the mean reconstruction error decreases as n increases. However, our approach shows a significantly low mean reconstruction error of 0.0467% at n = 5 compared to the conventional approach, which has a mean reconstruction error of 0.56% at same n. This result also justifies the utility of Zernike moments for modeling complex 3D shapes.

8. CONCLUSION:

In this work, we derive a novel 'volumetric convolution' using 3D Zernike polynomials, which can learn feature representations in B 3 . We develop the underlying theoretical foundations for volumetric convolution and demonstrate how it can be efficiently computed and implemented using low-cost matrix multiplications. Furthermore, we propose a novel, fully differentiable method to measure the axial symmetry of a function in B 3 around an arbitrary axis, using 3D Zernike polynomials. Finally, using these operations as building tools, we propose an experimental architecture, that gives competitive results to state-of-the-art with a relatively shallow network, in 3D object recognition task. An immediate extension to this work would be to explore weight sharing along the radius of the sphere.

Under review as a conference paper at ICLR 2019

EQUATION

Using the properties of inner product, Eq. 26 can be rearranged as,

EQUATION

Consider the term τ (θ,φ) (Z n ,l ,0 ). Then,

EQUATION

where D l m,m is the Wigner-D matrix. But we know that D l m ,0 (θ, φ) = Y l ,m (θ, φ). Then Eq. 27 becomes,

EQUATION

B EQUIVARIANCE OF VOLUMETRIC CONVOLUTION TO 3D ROTATION GROUP:

Theorem 1: Suppose f, g : X -→ R 3 are square integrable complex functions defined in B 3 so that f, f < ∞ and g, g < ∞. Also, let η α,β,γ be a 3D rotation operator that can be decomposed into three Eular rotations R y (α)R z (β)R y (γ) and τ α,β another rotation operator that can be decomposed into R y (α)R z (β). Suppose η α,β,γ (g) = τ α,β (g). Then,

EQUATION

)

where * is the volumetric convolution operator.

Proof: Since η (α,β,γ) ∈ SO(3), we know that η (α,β,γ) (f (x)) = f (η -1 (α,β,γ) (x)). Also we know that η (α,β,γ) : R 3 → R 3 is an isometry. Define, η (α,β,γ) f, η (α,β,γ) g = S 3 f (η -1 (α,β,γ) (x))g(η -1 (α,β,γ) (x))dx (32)

Consider the Lebesgue measure λ(S 3 ) = S 3 dx. It can be proven that a lebesgue measure invariant under the isometries, which gives us dx = dη (α,β,γ) (x) = dη -1 (α,β,γ) (x), ∀x ∈ S 3 . Therefore, < η (α,β,γ) f, η (α,β,γ) g >= S 3

f (η -1 (α,β,γ) (x))g(η -1 (α,β,γ) (x))d(η -1 (α,β,γ) x) =< f, g > (33)

Let f (θ, φ, r) and g(θ, φ, r) be the object function and kernel function (symmetric around north pole) respectively. Then volumetric convolution is defined as, f * g(θ, φ) =< f, τ (θ,φ) g > (34)

Applying the rotation η (α,β,γ) to f , we get, η (α,β,γ) (f ) * g(θ, φ) =< η (α,β,γ) (f ), τ (θ,φ) g > (35)

By the result 33, we have, η (α,β,γ) (f ) * g(θ, φ) =< f, η -1 (α,β,γ) (τ (θ,φ) g) > (36)

However, since η α,β,γ (g) = τ α,β (g) we get, η (α,β,γ) (f ) * g(θ, φ) =< f, τ (θ-α,φ-β,) g > (37)

We know that, f * g(θ, φ) =< f, τ (θ,φ) g >= Proposition: Suppose g : X -→ R 3 is a square integrable complex function defined in B 3 such that g, g < ∞. Then, the power of projection of g in to S = {Z i } where S is the set of Zernike basis functions that are symmetric around an axis towards (α, β) direction is given by, 

:



Figures and Tables:

Figure 1: Analogy between planar and volumetric convolutions. Top (left to right): image, kernel and planar convolution. Bottom (left to right): 3D object, 3D kernel and volumetric convolution. In planar convolution the kernel translates and inner product between the image and the kernel is computed in (x, y) plane.In volumetric convolution a 3D rotation is applied to the kernel and the inner product is computed between 3D function and 3D kernel over B 3 .

Figure 2: Kernel representations of spherical convolution (left) vs Volumetric convolution (right).In volumetric convolution, the shape is modeled and convolved in B 3 which allows encoding non-polar 3D shapes with texture. In contrast, spherical convolution is performed in S 2 that can handle only polar 3D shapes with uniform texture.

Figure3: Experimental architecture: An object is first mapped to three view angles. For each angle, axial symmetry and volumetric convolution features are generated for P + and P -. These two features are then separately combined using compact bilinear pooling. Finally, the features are fed to two individual capsule networks, and the decisions are max-pooled.

Figure 4: Accuracy vs number of trainable params (in millions) trend (ModelNet40)

Figure6: The mean reconstruction error Vs 'n'. Our Zernike frequencies computation approach has far less error than the conventional approach.

l,m (f )Ω n,l,0 (g)Y l,m (θ, φ) β,γ) (f ) * g(θ, φ) =< f, τ (θ-α,φ-β) l,m (f )Ω n,l,0 (g)Y l,m (θ -α, φ -β) (39) η (α,β,γ) (f ) * g(θ, φ) = (f * g)(θ -α, φ -β) (40) η (α,β,γ) (f ) * g(θ, φ) = τ (α,β) (f * g)(41)Hence, we achieve equivariance over 3D rotations.C AXIAL SYMMETRY MEASURE OF A FUNCTION IN B 3 AROUND AN ARBITRARY AXIS.

l,m Y m,l (α, β)|| 2 (42) where α and β are azimuth and polar angles respectively. Proof: The subset of complex functions which are symmetric around north pole is S = {Z n,l,0 }. Therefore, projection of a function into S gives, sym y (θ, φ) = n n l=0 f, Z n,l,0 z n,l,0 (θ, φ) (43)

; Qi et al. (2016); Sedaghat et al. (2016); Wu et al. (2016); Qi et al. (2016); Bai et al. (2016); Maturana & Scherer (2015) that show impressive performance on

Table content: <table><tr><td>Method</td><td>Trainable layers</td><td># Params</td><td>M10</td><td>M40</td></tr><tr><td>SO-Net (Li et al., 2018)</td><td>11FC</td><td>60M</td><td>95.7%</td><td>93.4%</td></tr><tr><td>Kd-Networks (Klokov &amp; Lempitsky, 2017)</td><td>15KD</td><td>-</td><td>94.0%</td><td>91.8%</td></tr><tr><td>VRN (Brock et al., 2016)</td><td>45Conv</td><td>90M</td><td>93.11%</td><td>90.8%</td></tr><tr><td>Pairwise (Johns et al., 2016)</td><td>23Conv</td><td>143M</td><td>92.8%</td><td>90.7%</td></tr><tr><td>MVCNN (Su et al., 2015)</td><td>60Conv + 36FC</td><td>200M</td><td>-</td><td>90.1%</td></tr><tr><td>Ours</td><td>3Conv + 2Caps</td><td>4.4M</td><td>92.17%</td><td>86.5%</td></tr><tr><td>PointNet (Qi et al., 2017)</td><td>2ST + 5Conv</td><td>80M</td><td>-</td><td>86.2%</td></tr><tr><td>ECC</td><td/><td/><td/><td/></tr></table>



Table content: <table><tr><td/><td colspan="2">: Comparison</td></tr><tr><td>with</td><td colspan="2">state-of-the-</td></tr><tr><td colspan="3">art methods on</td></tr><tr><td colspan="2">ModelNet10</td><td>and</td></tr><tr><td colspan="3">ModelNet40 datasets</td></tr><tr><td colspan="3">(ranked according to</td></tr><tr><td colspan="3">performance). Ours</td></tr><tr><td colspan="3">achieve a competitive</td></tr><tr><td colspan="3">performance with the</td></tr><tr><td colspan="3">least network depth.</td></tr></table>

Ablation study of the proposed architecture on ModelNet10 dataset

Table content: <table><tr><td>Method</td><td>Accuracy</td></tr><tr><td>Final Architecture (FA)</td><td>92.17%</td></tr><tr><td>FA + Orthogonal Rotation</td><td>85.60%</td></tr><tr><td>FA -VolCNN + SphCNN</td><td>79.53%</td></tr><tr><td>FA -MaxPool + MeanPool</td><td>87.27%</td></tr><tr><td>FA + Feature Fusion (Axial + Conv)</td><td>86.53%</td></tr><tr><td>Axial Symmetry Features</td><td>66.73%</td></tr><tr><td>VolConv Features</td><td>85.3%</td></tr><tr><td>SphConv Features</td><td>71.6%</td></tr><tr><td>FA -CapsNet + FC layers</td><td>87.3 %</td></tr><tr><td>FA -CBP + Feature concat</td><td>90.7%</td></tr><tr><td>FA -CBP + MaxPool</td><td>90.3%</td></tr><tr><td>FA -CBP + Average-pooling</td><td>85.3 %</td></tr></table>

Average rotation parameter values across classes of ModelNet10. The values are reformatted to be positive angles between 0 and 360.

Table content: <table><tr><td>Class</td><td>r 1</td><td>r 2</td><td>r 3</td><td>r 4</td><td>r 5</td><td>r 6</td><td>r 7</td><td>r 8</td><td>r 9</td></tr><tr><td colspan="10">Bathtub Bathtub 319.2 100.5 57.8 185.2 223.4 98.3 350.6 167.4 14.2</td></tr><tr><td>Bed</td><td colspan="9">264.3 196.3 103.7 208.5 186.2 194.4 267.9 246.3 81.2</td></tr><tr><td>Chair</td><td colspan="9">198.6 91.2 243.7 47.4 161.2 87.9 240.5 47.3 203.4</td></tr><tr><td>Desk</td><td>88.4</td><td colspan="8">80.2 130.9 206.6 86.5 112.8 291.7 233.2 351.4</td></tr><tr><td>Dresser</td><td colspan="7">58.0 145.7 353.1 148.4 346.4 125.3 47.0</td><td>2.2</td><td>35.4</td></tr><tr><td>Monitor</td><td colspan="3">218.9 279.0 58.1</td><td>10.4</td><td colspan="5">30.3 331.4 90.7 285.6 346.1</td></tr><tr><td>Night stand</td><td colspan="9">85.3 336.1 175.9 246.4 169.4 278.7 317.0 137.6 302.9</td></tr><tr><td>Sofa</td><td colspan="7">306.1 86.9 109.2 311.1 22.5 321.4 96.9</td><td>47.0</td><td>76.2</td></tr><tr><td>Table</td><td colspan="9">299.8 85.2 126.5 215.1 221.9 245.5 237.1 50.6 128.4</td></tr><tr><td>Toilet</td><td colspan="9">277.0 325.3 215.5 255.6 192.2 19.8 278.4 193.4 348.2</td></tr><tr><td>Average</td><td colspan="9">211.6 172.6 157.4 183.4 164.1 182.4 221.8 141.1 189.7</td></tr></table>

Peer review (iclr2019):
 There is a great amount of interest in extending the notion of equivariance in neural networks from just translations to other groups. In particular, in the past year a sequence of papers have appeared starting with (Cohen, Geiger et al.) on "spherical CNNs" that achieve equivariance to rotations for images painted on the surface of the unit sphere. The present paper extends these ideas to volumetric data in the unit ball (rather than just the sphere) by the use of Zernike polynomials. Since Zernike polynomials can be expressed as the product of spherical harmonics with a radial function, this is essentially the same as adding a radial component to a spherical CNN. The main result of the paper appears to be Theorem 1, which shows that what the authors define as volumetric convolution is equivariant to rotations. This is split across Sections 4.2 and 4.3.. However, apart from the radial component, this result is bascally the same as the SO3-equivariance of spherical CNNs, as discussed in three very recent spherical CNN papers: (Cohen, Geiger et al.) (Esteves Allen-Blanchette et al) and (Kondor, Lin and Trivedi). However, the somewhat more abstract, representation theoretic approach of some of these works allows a more compact derivation than the one in the present paper. The authors also fail to cite recent work on SE(2) and SE(3) equivariant neural networks. SE(3) comprises all rotations and translations of R^3, so the latter, in particular, encapuslates SO(3) equivariance as a special case. In particular, part of the construction in (Weiler, Hamprecht and Storath, CVPR 2018) is to add Gaussian radial functions to SO(2) equivariant filters, which is just the 2D analog of what is happening in the present paper. Then in (Weiler, Geiger, et al., 2018) the same is done in 3D, except of course they go further by also adding translation equivariance. Admittedly, these papers are very new, so the authors might not have known about them. I also find some of the mathematical details a little puzzling: 1. As explained in the spherical CNN papers, taking the cross correlation of two functions on the sphere (by extension, in the unit ball) naturally results in a function that lives on the rotation group SO(3), i.e. the cross-correlation (or convolution) is parametrized by three Euler angles. I don*t understand why the authors restrict themselves to considering just two angles, forcing their filters to be polar, as derived in Section 5. This seems like an artificial restriction that will limit the power of their approach. 2. The paper mentions that Zernike polynomials are "orthogonal and complete in B^3". I think what they mean is that they are an ortogonal and complete basis for an appropriate space of functions on B^3, and that space of functions is L_2(B^3). However, this is still not enough. For (3) to hold, one also needs the basis to be orthonormal. Please be more precise. 3. In the same vein, at one point in the proof, the authors mention that "rotations are unitary operators in a Hilbert space". This is not true of Hilbert spaces in general. It requires the above orthonormality etc.. 4. Exactly as a consequence of orthonorlamity, Equation 5 is essentially just a generalized Fourier transform on B^3, hence, in principle, it can be inverted analytically. I understand that the fact that the input image is rasterized complicates things and in a practical implementation it might be expedient to invert (5) by using the pseudo-inverse. However, this is a one-time operations and is really just a hack. It seems strange that the authors use a special iterative method just to invert a close to unitary matrix. The mathematical shortcomings of the paper could be compensated by amazingly good experimental results. The actual results are good, but still not best-in-class, possibly because at the end of the day the network is still only rotationally equivariant and does not take into account translations. The spherical CNN and SE(n) equivariance papers generally apply the group equivariant operations consistently across multiple layers. In contrast, the present paper only applies it in the first layer, and then uses a combination of tricks like multiple viewpoints and bilinear pooling to boost performance. Unfortunately, the benefits of this additional conceptual complexity are not entirely borne out in the experimental results.