Adaptive Wavelet Transformer Network for 3D Shape Representation Learning
Authors: Hao Huang
Yi Fang
Venue: ICLR 2022 Poster (2022)

Abstract: We present a novel method for 3D shape representation learning using multi-scale wavelet decomposition. Previous works often decompose 3D shapes into complementary components in spatial domain at a single scale. In this work, we study to decompose 3D shapes into sub-bands components in frequency domain at multiple scales, resulting in a hierarchical decomposition tree in a principled manner rooted in multi-resolution wavelet analysis. Specifically, we propose Adaptive Wavelet Transformer Network (AWT-Net) that firstly generates approximation or detail wavelet coefficients per point, classifying each point into high or low sub-bands components, using lifting scheme at multiple scales recursively and hierarchically. Then, AWT-Net exploits Transformer to enhance the original shape features by querying and fusing features from different but integrated sub-bands. The wavelet coefficients can be learned without direct supervision on coefficients, and AWT-Net is fully differentiable and can be learned in an end-to-end fashion. Extensive experiments demonstrate that AWT-Net achieves competitive performance on 3D shape classification and segmentation benchmarks.

Published as a conference paper at ICLR 2022
ADAPTIVE WAVELET TRANSFORMER NETWORK FOR 3D SHAPE REPRESENTATION LEARNING
Hao Huang, Yi Fang
NYU Multimedia and Visual Computing Lab, USA
NYUAD Center for Artificial Intelligence and Roboties (CAIR), Abu Dhabi, UAE
NYU Tandon School of Engineering, New York University, USA
New York University Abu Dhabi, UAE
[hh1811, yfang) @nyu. edu
ABSTRACT
We present novel method for 3D shape representation learning using multi-scale wavelet decomposition. Previous works often decompose 3D shapes into comple- mentary components in spatial domain at single scale. In this work, we study decompose 3D shapes into sub-band components frequency domain at multiple scales, resulting in a hierarchical decomposition tree in principled manner rooted in mulli-resolution wavelel analysis. Specifically, we propose Adaptive Wavelet detail
Transformer Network (AWT-Net) that firstly generates approximation wavelet coefficients per point, classifying each point into high or low sub-bands components, using lifting scheme at multiple scales recursively and hierarchically. Then, AWT-Net exploits Transformer to enhance the original shape features by querying and fusing features from different but integrated sub-bands. The wavelet coefficients can be learned without direct supervision on coefficients, and AWT- Net fully differentiable and can be learned in an end-to-end fashion. Extensive experiments demonstrate that AWT-Net achieves competitive performance 3D shape classification and segmentation benchmarks.
INTRODUCTION
Analysis of 3D point cloud is crucial topic in computer vision and graphics, and has wide appli- cations in robotics (Chen et al. 2019) autonomous driving (Qi al. 2018), and visual SLAM (Hitchcox Forbes 2020), etc. To better under- stand the 3D shapes, effective point cloud analysis approaches and meth- ods are in great demand. With the thriving of deep learning, numerous task-driven neural networks are cre- ated to generate high-level 3D shape representations with learnable rons, rather than designing low-level shape descriptors manually. Point- Net (Qi et al., 2017a) is firstly pro- posed use deep neural network to directly consumes take as input raw point cloud and max-pools per-point features to obtain the global shape representation. Point- Net++ (Qi et al.) 2017b) introduces feature aggregation operations at multi-scales to group of the entire point clouds in both down- and up-sampling layers. The following works learn 3D shape representation from different perspectives: SO-Net (Li et al.) 2018a) utilizes self-organizing map to model the spatial distribution of points, SpiderCNN (Xu et al. 2018) exploits step function and Taylor polynomial captures local geodesic efficiently, KPConv (Thomas et al. 2019) designs de-
Figure I: The detail component of table, car, chair, and cup from the test set of ModelNet40 (Wu et al.) 2015). Colors represent the absolute values of the detail coefficients per point. The larger values (redder colors), the higher prob- ability that the points locate on high-frequency parts, e.g., edges or non-flat areas. Such geometric representations learned implicitly in 3D shape classification task.

---

Published as a conference paper at ICLR 2022
formable convolutions to localize kernel points for local geometry learning, DGCNN (Wang al. 2019) treats point clouds as graphs and applies graph convolution networks, to name a few.
An interesting yet challenging problem is how to decompose 3D shapes into interpretable represen- tations and exploit them for better shape analysis, e.g., decomposing shapes into areas containing flat planes or areas containing edges. Another challenging problem is how to analyze 3D shapes multiple scales/resolutions in an explainable way. Though some works (Qi et al. 2017b] Jingg Jiang et al.)
2019) extract point cloud features hierarchically, these works build the hierarchy in spatial/semantic spaces based on pairwise point distance, which is less interpretable w.rl. shape geometry.
As indicated in GDA-Net et al. 2021b), the geometric components with different frequencies in 3D shapes contain distinct geometric characteristics (e.g.. -eee-freyynncy components correspond to edges or sharp parts, while low-frequency components correspond to flat areas or smooth parts) and provide complementary geometric information for shape representation. To effectively learn desirable to separate these components and process them differently, rather than feed the whole shape into single processor. GDA-Net (Xu et al. 202 disentangles 3D shapes into sharp and gentle variation components using spectral graph analysis. Generally, GDA- Net starts from building a dynamic graph from given point cloud, then applies a 2-order polynomial approximation (Laplacian) as filter to the graph to disentangle the graph into high and low-frequency components, and finally fuses the two component features with the original shape features through two attention modules. GDA-Net has achieved excellent performance on 3D shape classification and segmentation. However, for given graph, the spectral analysis based approach adopted in GDA- Net provides a static and deterministic graph decomposition, posing a limitation that disconnects the graph decomposition process from downstream tasks of shape representation learning. There- fore, to address the limitation, we propose a novel wavelet analysis based method to decompose 3D shapes into sub-bands components for 3D shape representation learning. The decomposition process is realized by neural networks in a learnable fashion, thus connected to the downstream tasks of shape representation learning. Theoretical properties of wavelets signal processing, such as time-frequee fre-frruy localization and multi-resolution analysis, have been well studied, making it ap- propriate to fuse into deep neural networks with interpretability. Inspired by that, we propose to perform multi-resolution analysis within neural networks through lifting scheme (Sweldens 1998) to achieve a data-driven wavelet transform. The multi-resolution analysis generates decomposed visual representation at each scale, which contributes to the interpretability of the network. The generated approximation or detail coefficients per point at each scale, corresponding to high- or low-frequency components, capture essential geometry for downstream tasks.
3D shape representation,
Specifically, we first adapt wavelet transform that analyzes 1D temporal or 2D image signals to 3D shapes at different semantic levels and decompose the original point cloud into the approxima- tion/coarse and detail components, as explained in Section 4.1 and 4.2 Next, in Section 4.3 we utilize Transfomer (Vaswani et al., 2017) to pay different attention to features from approximation and detail variation components, and then fuse them with the original point features, respectively. As shown in Figure the approximation and detail components have distinct but complementary effects on reflecting the geometry of 3D shapes. Equipped with wavelet transform and Transformer, we propose Adaptive Wavelet Transformer Network (AWT-Net) that captures and refines the inte- grated and complementary geometry of 3D shapes to supplement neighboring local information. AWT-Net is trained using the loss functions proposed in Section 4..4 Experimental results on chal- lenging benchmarks demonstrate that our AWT-Net achieves competitive performance on 3D shape classification and segmentation tasks. Comprehensive analysis and visualizations verify that our model effectively extracts the complementary geometric features from two decomposed compo- nents at multiple scales. For instance, as shown in Figure [ we observe that our model can learn meaningfully samplings of point clouds in different categories and can detect key regions consis- tently within different categories, e.g., it separates chair legs and backs from chair seat, and separates a cup handle from body.
2 RELATED WORK
Geometric Point Cloud Models. PointNet (Qi et al. 2017a) and DeepSet (aheer et al. 2017) are pioneering neural network models that directly process point cloud to learn spatial encoding of each point and then aggregate all individual point features to an integrated representation. Point- Net++ (Qi et al. 2017b) partitions point cloud into overlapping local regions by the distance metric
2

---

Published as a conference paper at ICLR 2022
of spatial and/or semantic spaces and extracts local features capturing fine geometric structures from neighbors hierarchically. DGCNN (Wang et al. 2019) reconstructs k-NN graphs using nearest neighbors in feature space to capture local geometric structure while maintaining permutation invari- ance. KC-Net (Shen et al. 2018) proposes kernel correlation layers to extract the local geometric features. Point2Sequence (Liu al. (019a)) designs sequence model capture the correlations among points. FPConv (Lin et al. (020a) introduces a surface-style convolution operator to directly work on point cloud surface. Poi (Shi & Rajkumar 2020) learns local shape geometry by capturing interactions among points in local patches through node and edge feature aggregation. PAConv (Xu et al., (021a)) proposes a convolution kernel that adaptively assembles different weight matrices using coefficients learned from point positions. GDA-Net (Xu et al. 202 is built on graph spectral theory and uses a 2-order polynomial approximation (Laplacian) as a filter and ap- plies it to graph to disentangle the graph into high and low-frequency components, thus resulting in deterministic and static decomposition. Our model learns point cloud geometry from a novel perspective, i.e., multi-resolution wavelet analysis, which has been extensively applied in signal processing but seldom explored for 3D shapes.
Wavelet in Vision. Theoretical properties of musti-resolution analysis using wavelets have been well studied, making such approaches more interpretable than CNNs. Several prior works incorporate wavelet representations into CNNs. (Oyallon et al. 2017) proposes hybrid network that replaces the first layers of a ResNet with a wavelet scattering network. This modified ResNet resulted in comparable performance to the original ResNet but has fewer trainable parameters. The wavelet pooling layer proposed in (Williams & Li (018)) subsamples image features based on second-level wavelet decomposition. Wavelet-SRNet (Huang et al. 2017) reconstructs super-resolution images from multi-scale wavelet coefficients estimated from low-resolution images under coefficient super- vision. Similar scheme is adapted in (Liu et al., 2020p for image demoiréing and (Rong et al., 2020) for burst denoising. In (Ramamonjisoa et al. 2021b, they decompose single image using Haar Transform and then reconstruct depth images hierarchically through inverse discrete wavelet trans- form. All of the mentioned works use pre-defined Haar wavelet which is less flexible for complex 3D shapes. (Rustamov & Guibas| (013) adopts lifting scheme but still restricts to linear transforma- tions of Haar coefficients for graph signals. DAWN-Net (Rodriguez et al., 2020) uses lifting scheme to implicitly define wavelet transform functions for image recognition. Due to the irregularity of 3D shape point arrangement, applying lifting scheme to learn 3D shape representation is more chal- lenging than to 2D images proposed in DAWN-Net, and as far as we know, we are among the first to tackle this problem.
Transformer in Vision. The seminal work to adapt Transformer (Vaswani et al. 2017) to vision tasks commences with Vision Transformer (ViT) (Dosovitskiy et al. (2220) and Detection Trans- former (Carion et al. 2020). Since then, various applications of Transformer in vision are proposed, including but not limited to: semantic segmentation (Wang et al. 202T: Zhang et al., 2020 gener- ative modeling (Chen et al.) 2020; Hudson& & Zitnick (021), object detection (Zhu et al. [021 Dai let al. 202T), video analysis (Gabeur et al. 2020: Zeng et al. 2020), to name a few. PCT (Guo et al. 2021 applies Transformer in 3D point cloud classification and segmentation. Concurrent works include the other two types of Point Transfomer proposed by (Zhao et al. 202T) and (Engel et al. 2021). PoinTr (Yu et al. (021) converts point cloud to a sequence of point proxies and employs Transformer for point cloud completion. PPT-Net (Hui et al. (02I)) adopts Transformer for large- scale place recognition. MVTT (Fan et al. 2021) and Swin (Liu et al., (02T) connect multi-scale feature hierarchies with transformer models by applying different Transformers for features with distinct spatial resolutions. Since multi-scale discrete wavelet transform decomposes point clouds at each subsequent scales, it is intrinsically essential to combine with Transformer in a similar way as in MVIT and Swin. Due to quadratic computational complexity of self self-attention, Fast Autoregres- sive Transformer (Katharopoulos et al., (020)) is among the first to address the quadratic complexity by using the associativity property of matrix products. Some recent works (Kitaev et al.] [2020] Dai et al., 2020) Tay et al., 2020) Choromanski et al. (021) propose efficient Transformers for long sequences and these models can be also adapted for vision tasks.
3 BACKGROUND
We briefly review wavelet transform and lifting scheme as the building blocks of our model. We refer the reader to (Sweldens) 1996: 1998 Claypoole et al. 2003] Rodriguez et al., 2020) for more information.
3

---

Published as a conference paper at ICLR 2022
Multi- head attention
Add &
Feedforwa
Add nomn
①
c[n]
X=[nt]
Q: 2
x[n]
Split
K: Cj
Tarsfonner
V:C;
[o[n]
d[n]
Q: 2
K: 0e
Tranffrmerr
V: 0eldi
Figure 2: (Left) Lifting stages: split, update (U) and predict (P). (Right) The architecture of Trans- former in AWT-Net.
Wavelet Transform//Decompsiitie Let us consider continuous scalar function of single vari- able [()). Wavelet transform to find basis generated by mother wavelet, denoted by (xx)), to represent f(x) with few parameters. The basic rule for generating this basis is translate and di- late vxx)), f.e., (/22 - i), where j and i are integers. The basis is localized in both time (controlled by i) and scale/frequency (controlled by j) domains. The original function f() is decomposed linear combination of the basis:
f(x) = Σ (2'x0
where 0z are the wavelet coefficients of f(x) and can be computed as the inner product of (2/2x x i) with f*) For discretized signal x|n]] (n E 2), we sample points from (2'xz i) to form filter banks decompose z/n]. This is classical or first-generation wavelet (Santosa 201 Some commonly used mother wavelets include Haar wavelet (Haar 1910), Daubechies wavelet, Biorthogonal wavelet (Daubechies) 1992), to name few. Wavelet transform provides a nice frame- work for multi-resolution analysis of functions, ie., wavelet transform generates coarser and coarser versions the same function while keeping track of the details missed going from fine to coarse. However, the first-generation wavelet mainly works well for infinite or periodic signals, which are rare in many applications. It is still unclear how to design proper basis for any arbitrary signals sampled irregularly from unbounded domain. One solution is give up designing mother wavelet manually. Instead, the decomposition is entirely based on an approach called lifting scheme.
Lifting Scheme. Lifting scheme, a.k.a., second-generatin wavelets (Sweldens) 1998 Daubechies & Sweldens) 1998), is a simple yet effective approach to define wavelets sharing the same properties the first-generation wavelets. The lifting scheme takes as input signal and generates outputs the approximation and the detail d sub-bands of the wavelet transform. The lifting scheme consists of three stages (Claypoole et 2003 Rodriguez al. 2020 on the left panel of Figure 2
Split. This stage splits an input signal into two non-overlapping partitions, e.g., the input signal is divided into i. and odd x. components, which are defined as xr 2: 22]]
and [n] x x|2n= + 1].
Update (U). This stage separates the signal in frequency domain, generating the approx-
imation that has the same running average as the original input signal. Let 2 [n]
H. n Lu]. .n Lu + 1).... 2n[n Lu - 1],
Lu] denote the sequence of 2L + 1 neighboring odd polyphase samples around xe [n]. The even polyphase samples
are updated through an update operator U(-) on x-4 [n]] resulting the approximation e as:
c(n) = [r][ + (= [n])
(2)
Predict (P). As the partitions x. and J. are usually closely correlated, given one of them, it is feasible to build a predictor P(-) for the other, by tracking the difference (or detail)
among them. As the even component ze corresponds to the approximation c(n)), it is possi-
ble to define P(-) as function of c[n]. Let che [r] = cin - cln.c(n Lp + +]. 11......c. c(n + Lp 1), c/n + Lr] denote sequence of 2Lp approximation coefficients. The odd polyphase samples are predicted as P(c [n]). The resulting prediction residual, or detail d, are computed as:
d/ri] [o P(ct"(n))
(3)
4

---

Published as a conference paper at ICLR 2022
Note that the update and predict operators U and P iipllcitly and effectively define the wavelet for signal decomposition.
4 ADAPTIVE WAVELET TRANSFORMER NETWORK
Overview. this section, we propose an Adaptive Wavelet Transformer Network, dubbed as AWT- Net, for 3D shape representation learning, as shown on the right panel of Figure 3 Similar to the overall structure of GDA-Net (Xu et al., 202Tb), AWT-Net consists of three modules: k-NN graph construction module to build graphs from point clouds, 2) adaptive lifting module, reflecting our main contribution that decomposes 3D shapes into high- and wwwfrreuency components using wavelet analysis, and 3) transformer module to fuse decomposed component features. The AWT- Net generates shape representation hierarchically. and the decomposition tree shown on the left show three scales where each scale corresponds to separate lifting We omit the split, update and predict blocks for coarse version of the input signal while
panel of Figure 3 Here
process as illustrated on the left panel of Figure 2
conciseness. At each scale, the approximation c; the detail d, maintains the detailed difference between G and the input signal. The approximation c is further decomposed by another lifting process the next scale. The AWT--Net is built based on this decomposition tree but replaces the linear operators U and P in Eq. 2 and Eq. 3 with non-linear neural networks. Once trained and fixed, the non-linear update U' and predict /' operators can be regarded as two sub-band basis/filters to subdivide 3D shapes into high-frequenge and low- low-frequency components, in analogy to the basis in Eq.
clessification
Mr.Ps
segmentation
seale
scalle
d1
C1
Adaptive Lintng
Adappive Litting
Z2
C2
Adaptive Litting
Z3
d2
Z
P. GCN
Spltt
22
dg
C3
U: UCN
da
Figure 3: (Left) A multi-resolution wavelet decomposition tree (of three scales) with lifting scheme. (Right) An unrolled version of AWT-Net corresponding to the decomposition tree. Here omit update and predict operators for simplicity.
4.1 EVEN-ODD SPLIT
Graph Construction. Assume a point cloud containing N points associated with features 2; E Ri, E {1,... .N.. Features be coordinates, normals and/or other -igheniinnsional features. We construct a k-NN graph G (V, A"")", where each point I; corresponds to node ea e V. A and Á" R. NxN N are unweighted and weighted adjacency matrices encoding point similarity in feature space. The unweighted and weighted edges connecting two nodes Us and e are defined as:
[1 [. [,<((1==; 23)12)1, Iy N(xx) otherwise
À A
10
where s(-) is an non-negative function, e.g., Gaussian function, to ensure that Á" is a diagonally dominant matrix and N represents neighborhood. Before constructing k-NN graph, we compute point features through a spatial transform layer (Wang al. 2019), also termed local operator in (Xu et al.) 2021b). After graph construction, we normalize edge weights each row in AA to remove the impacts of varying neighborhood sizes and feature scales as in (Xu al. 2021b).
Graph Split. We partition all nodes in a graph into two sets, containing even and odd nodes, respec- tively. Since lifting scheme uses information from odd (resp. even) nodes to update (resp. predict) even (resp. odd) nodes as in Eq. 2 resp. Eq. 34) having all neighboring nodes with same parity means that local information cannot be used. Following (Narang & Ortega 2009), define the number of
5

---

Published as a conference paper at ICLR 2022
Algorithm 1: Even-Odd Node Split
Require: A graph 9 = (V, with unweighted adjacency, maximum number of iterations M Randomly assign initial parity (e.g., 1 for odd, 0 for even) to each node
for m 1:M do
Activate each node with a probability sampled from uniform distribution independently for each activated node do
L Choose parity that minimizes its conflict with neighboring nodes
/+ Keep the number of odd and even nodes roughly the same No. Ne f number of nodes with parity 1, parity 0
if N, > Ne then
Randomly flip No odd nodes' parity
else
Randomly flip No even nodes' parity
10
conflicts as the percentage of neighboring nodes that have the same parity after partition. The opti- mal splitting minimizes the number of conflicts. Theoretically, is an NP-hard bipartite sub-graph problem, but heuristic solution can be found. We adapt the conservative fixed probability colorer algorithm in (Fitzpatrick & Meertens, 2001), which is based on iteratively greedy local heuristies: at each iteration, few randomly chosen nodes are activated, and each activated node counts the number of conflictigg edges (i.e., connecting neighboring nodes with the same parity) and changes its parity based on the conflict. The algorithm is summarized in Algorithm [
4.2 ADAPTIVE LIFTING SCHEME
Based on nodes' parity after even-odd split, we re-arrange nodes V gather odd and even nodes, and also re-arrange the weighted adjacency matrix A" accordingly:
Vodd Veven :
FN.xNN. JN.XN.
V
A
KN.XN.
LN.NN.)
where Voda and V Veven represent the sets of odd and even nodes. The sub-matrix F is the adjacency matrix of a subgraph containing odd nodes only. The submatrix J contains edges that do not have conflicts, i.e., edges connecting nodes with the opposite parity. Similar definitions for and
Unlike the linear lifting scheme introduced in Section 3 we propose data-dependent non-linear update and predict operators, implemented as neural networks whose parameters are adaptively op- timized to capture complex shape geometry. Specifically, we apply two Graph Convolutional Net- works (GCNs) proposed in (Kipf & Welling 2017) as the update and predict operators, separately. The graph convolution allows us to compute the response of a node based its neighbors decided by the adjacency matrix. Performing graph convolution is equal to performing message gathering within neighborhood around each center node, sharing similarities with the definitions of a [r] and c (n] in Eq. 2 and Eq. 3 The non-linear update and predict stages at i-th scale are defined as:
C = Veven + GCN(V odd, F.K..
(5)
d, = ood GCN(c,, T)
(6)
where e and d, represent the approximation and detail components at i-th scale. We repeat this process at each scale recursively and form hierarchical decomposition tree, capturing shape geom- etry at multi-scales. The maximum number of scales can be decided analytically as log, N and the coarsest scale contains only single point.
4.3 TRANSFORMER FOR CROSS ATTENTION
The approximation and detail components depict 3D shape geometry from different but comple- mentary aspects, and thus can be exploited to enhance the original input shape features. Similar to GDA-Net (Xu et al. al, (021b) which adopts an attention module to fuse variation components with the original shape features, different sub- bands gained from lifting scheme a principled way. The architecture of Transformer is shown on the right panel of Figure and multi-head attention is defined as:
adopt Transformer (Vaswani et al. (017)) to query and fuse features from
Attention QQ K, V) softmax V

---

Published as a conference paper at ICLR 2022
where Q, K, V denote query, key and value, respectively.
For approximation c, as coarse component at i-th scale, we set query, key and value to:
Q 2,-1, K V..V= V
(8)
where the definition of 2; is shown in Figure 3 Note that for the first scale (i.e., i = 1), we set 1 x input. For detail d,, most of the values are close zeros (explained in Section 4.4 directly feeding them into Transformer yields sub-optimal results. Utilizing nice property wavelets, namely frequenyy localization we regard d, as a multiplier to strengthen or weaken the context features. Formally, the query, key and value for detail d: at i-th scale are set as:
Q = =-, K G-11 & exp(Id)), V = G-11 @ exp(1d))
(9)
where @ represents element-wei wise multiplication. For the first scale, we set both 2;-1 and c,-1 to æ.
4.4 LOss FUNCTION
End-to-end training is performed using task-specific cross-entropy loss, in combination with reg- ularization terms, to enforce a wavelet decomposition structure during training. The loss function takes the form of Eq. TO, where K in the first term denotes the number of classes, I and pt the ground-truth and the predicted probability of belonging to class i, respectively.
K
Loss = Σ y log(p:) + A1 Σ S - -4) + X2 Σ Smooth
(10)
In lifting scheme, a perfect output of Eq. 2 i.e., the series e(n]. would be a coarse approxima- tion of ar[n]] at half the resolution. In the second term, we enforce each lifted even node has the same local average as its corresponding node in the original graph. Due to the half resolution of oa compared with the input G-1, multiply the adjacent matrix A with C:-1 aggregate the node features within the neighborhood, and minimize the difference between the input graph features and the coarser version across all the decomposition scales. The rationale for the third term is: if the underlying signal æ(no is locally smooth, the prediction residuals d[n] should be small. Thus, the third term promotes the minimization of the magnitude of detail coefficients d[n] in Eq. indicating smaller energy is contained in d(r.).
5 EXPERIMENTS AND RESULTS
We evaluate our model on 3D shape classification and part segmentation tasks on three datasets following the settings (Qi et al.) 2017 7a b Wang et al., 20199; Xu et al. 2021b; Zhao et al.
2021)
ModelNet40 (Wu et al. 2015), ScanObjectNN (Uy et al. 2019) and ShapeNet Part (Yi et al. 2016
5.1 IMPLEMENTATION DETAILS
The number of the constructed neighborhoods in k-NN graphs at each stage is 64. The maximum number of iterations M used in the split Algorithm D is set to 10, which strikes a balance between
efficiency and accuracy. The update (-) and P(-) operators are implemented as two GCNs (Kipf & Welling, 2017) and the internal structure of each GCN is: Conv-BN- -eN-ReeU-Drrpoutt-C CorooputtCCoy---ie conut-Conv-BBu Manv-BN. We two-heads attention in Transformer. We implement We use SGD optimizer with momentum and weight decay set to 0.9 and 0.000T, respectively. For both 3D shape classification and 3D object part segmentation tasks, we train for 350 epochs. The
model in Py Torch (Paszke al. 2019).
initial learning rate is set to 0.2 for shape classification and 0.005 for object part segmentation, and clipped at 0.9e The values of X₁ and À2 are set to 0.1 for all tasks.
5.2 ОВJЕCТ CLASSIFICATION We evaluate AWT-Net on ModelNet40 (Wu et al. 2015) which contains 9,843 training shapes and 2,468 test shapes across 40 categories. Points are uniformly sampled from mesh models by (Qi et al. 2017a). Following (Wang et al.) 2019 Xu et al. shuffle points the training set. Similar to (Qi et al. 2017a b). we conduct several voting tests with random scaling and average the predictions during testing. The quantitative comparisons of the
2021b), we translate shapes and randomly
In analogy wave signals expanding time domain, for 3D shapes which expand in the spatial domain, this property can be rephrased fretial-frequency localization.
7

---

Published as a conference paper at ICLR 2022
overall classification aceuracy with the state-of-the-ar methods in Table AWT-Net with two scales achieves the best performance. Beyond quantitative evaluation, we exhibit that our method can decompose shapes into approximation and detail components as shown in Figure and Figure
We also validate the robustness of AWT-Net on a real-scanned dataset (Uy et al. 2019) following the experimental settings in GDA-Net (Xu et al.) 202Tb). We test the OBJ ONLY and noisy OBJ BG splits, and the results are summarized in Table 2 where our model achieves comparable accuracy and performance drop from OBJ JNLY OBJ BG.
Method (time onder)
Accuracy
PointNet Q al 2017a)
IK
89.2
PointNetes et
2077h)
IK
90.7
SCN Xie
1K
90.0
KCNaT Singn el s.. 2018
IK
91.0
Method
OBJ.ONLY OBLBB Acc. drop
PainCCN at.
PointNet @Q al
20
79.2
73.3
J.5.9
Point2Sequence [Tab ci
2019
IK
92.6
PointNet+s ct
201
7b
84.3
82.3
2.0
DGCNN Wang al 2019
2019
IK
IK
93.0
92.9
SpiderCNN
PoinCNNN
Au 3
2018
79.5
77.1
85.8
2.1
5,4
RS-CNN Liu al. T95
IK
93.6
3D-GCN
DGCNN
Wane cf
2019
86.2
82.8
3,4
FPCony Lin al
1K
92.5
GDA-Net
et
20216
88.5
87.0
1.5
GSNet
al.
(120)
AWT-Net (ours)
88.5
86.8
1.7
Poin(Transtomner
Zhao al.
2021
IK 93.7
PCT (Guo et 2021
IK
93.2
GDA-Net
2021b
IK
93.8
Table 2: Classification accuracy (%) W.E.f noise on
ScanObjectNN dataset.
AWT-Net (ours)
1K
93.9
Table 1: Classification accuracy (%) on ModelNet40 dataset.
5.3 PART SEGMENTATION
We evaluate AWT-Net with three scales for shape part segmentation ShapeNet Part (Yi et al. 2016) dataset, which contains 16,881 shapes in 16 categories and is labeled 50 parts where each shape has 25 parts. The same voting test in classification is adopted. The instance average, the class average, and each class mean Inter-over-Union (mloU) are summarized in Table 3 AWT- Net achieves the best performance with class mloU of 85.0% and instance mloU of 86.6%. Some qualitative segmentation results are visualized in Figure
Method (time order)
class instance
mLOU
acro
bag
car cap car chair phone
guitar knife
lamp
lap
top
motor mug pistol rocket
skate board table
PointNet (Qh ct al.
20
80.4
83.7
83.4
7 82.5 74,9 89.6
73,0
91.5
85.9
80.8
95.3
65.2
93.0
81.2
57.9
72.8
80.6
PointNate+ [Q)
82.4
79.0 87.7
90.8
71.8
83.7
95.3
71.6
94.1
81.3
58.7
76.4
82.6
SyncCNN (Yrel al 2017
82.0
84.7
81.6
81.7
81.9
75.2
90.2
74.9
93.0
86.1
84.7
95.6
66.7
92.7
81.6
60.6
82.9
82.1
SCN Xie al. 2013)
81.8
84.6
83,8
80.8 83.5 79.3
90.5
69.8
91.7
86.5
82.9 96.0
69.2
93.8
82.5
62.9
74.4
80.8
KCNGT (She
2018)
82.2
84.7
82.8
86.4
77.6
76.8
87.0
84.5 95.5
81.6
60.1
75.2
81.3
SpidecCNN (Xu cu 2018
82.4
85.3
83.5 81.0 87.2
77.5
90.7
76,8
91.1
87.3
K3.3 95.8
70,2
93.5
W2.7
59.7
75.8
82.8
Poini2Sequence [Lau
2019a
82.6
77.3
86.9
83,9 95.7
79.3
DGCNN (Wang è
2019
82,3
85.2
84.0 83.4 86.7
77.8
90.6
74.7
91.2
87.5
82.8 95.7
66.3
94.9
81.1
63.5
74.5
82.6
RS-CNN Liu al
2019b
86.2
83.5 84.8
79.6
91.2
81.1
91.6
88.4
86.0 96.0
73.7
94.1
83,4
60.5
77.7
83.6
3D-GCN Lin
202UB
82.1
85.1
83.1 84.0 86.6
77.5
90.3
74.1
90.0
86.4
83.8 95.6
66.8
94.8
81.3
59.6
75.7
82.8
GSNei (Xa eiu
2020)
83.5
85.3
82.9 84.3 88.6
78.4 89.7
78.3
91.7
86.7
81.2 95.6
72.8
94.7
83.1
62.3
81.5
83.8
PCT (Gllo
86.4
85.0
82.4
89.0
81.2 91.9
71.5
91.3
88.1
86.3
95.8
64.6
95.8
83.6
62.2
77.6
83.7
GDA-Nat Xu el ai
20/1b)
85.0
86.5
84.2
88.0
90.6
80.2
90.7
82.0
91.9
88.5
82.7
96.1
75.8
95.7
83.9
62.9
83.1
84.4
Poin/Translrrmer /hha
2021
AWT-Net (ours)
85.0
86.6
83,9
88.4
88.5 79,2 90.6 81.3
92.5
88.7
82,1
96.3 73.9
95.9 83,5 63,0
84,1 84.2
Table Segmentation results (%) on ShapeNet Part dataset.
ABLATION STUDY
We conduct ablative experiments ModelNet40 and no voting during testing (unless specified). In all ablative experiments, we follow the same experimental settings DGA-Net (Xu et al. 2021b). In addition, adopt the same performance measurement metrics that were used in DGA-Ne Net to report our results in Table T to Table g and Figures | in validating and analyzing the effectiveness of our 3D shape representation learning approach based on wavelet analysis for shape decomposition.
8

---

Published as a conference paper at ICLR 2022
Module Analysis. We ablate each module in AWT-Net, and the results are listed in Table 4 When the Transformer takes as input the original point cloud features fused with both approximation and detail components, AWT-Net achieves the best accuracy with 93,4%. Our method ultimately obtains an accuracy of 93.9% with voting. This experiment shows that the decomposed approximation and detail component complement mutually capture different but complementary geometric features to get complete shape representation.
Split Scheme. We validate the split of approximation and detail compo- nents against random and farthest point sampling (FPS). The results are listed in Table 5] where decompos- ing point clouds into approximation and detail components gets the high- est accuracy.
H
I
Adjacency Matrix Computation. The result of dynamic adjacency ma- trix computation in Section [.1 is shown in Table 6 We pre-compute the adjacency matrix on 3D coordi-
Figure 4: Qualitative results of part semantic segmentation.
nates split point clouds, which achieves the accuracy of 91.8%. Nevertheless, constructing the adjacency matrix in high dimensional semantic space at different scales gains 1.6% improvement. Figure and and Figureldsshow that our decomposition strategy successfully decomposes points into flat areas and areas containing edges. Note that we do not provide any supervision w.r.t. shape geometry.
Decomposition Levels. We investigate the impact of decomposition levels (i.e., resolution scales), and the results are summarized in Table 7 AWT-Net performs best on the overall accuracy when the decomposition level is to 2. We also observe that for shapes with simple geometric structures (e.g., radio and wardrobe), more scales have negative effect on the performance, as more scales make the network prone to over-fitting on these simple shapes. While for shapes with complex structures (e.g., stool and person), more scales are needed capture both coarse and fine-grained geometric features. That is, the accuracy increases with more scales applied to geometrically more complex objects. We find out that the ModelNet40 dataset contains more geometrically simple objects than complex ones, which leads to the overall accuracy drops with more scales adopted.
Sampling and Rotation Test. First, test AWT-Net the 3D shapes with different numbers of randomly sampled points and Figure lists Net against rotation and Table & summarizes the results. We note that AWT-Net outperforms the compared methods.
lists the results. Next, we evaluate the robustness of AWT-
CONCLUSION
Previous works such as GDA-Net have observed the effectiveness of learning shape representation via decomposing the 3D shapes into low- and high-frequency components. In this work, we further advance the shape decomposition with a novel wavelet analysis based method that decomposes 3D shapes into sub-bands components. Our work proposed use the deep neural networks realize the shape decomposition in learnable fashion that seamlessly connects to the downstream tasks of shape representation learning. Extensive experiments show that our AWT-Net achieves excellent performance on 3D shape classification and segmentation tasks which are par with the state-of- the-art approaches.
ACKNOWLEDGMENTS
We would like to thank the reviewers and the authors of GDA-Net (Xu et al., 2021b) for their com- ments and efforts towards improving manuscript. The authors appreciate the generous support provided by Inception Institute of Artificial Intelligence (IIAI) in the form of NYUAD Global Ph.D. Student Fellowship. This work was also partially supported by the NYUAD Center for Artificial Intelligence and Robotics (CAIR), funded by Tamkeen under the NYUAD Research Institute Award CG010.

---

Published as a conference paper at ICLR 2022
REFERENCES
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proceedings of the Eu- ropean Conference on Computer Vision, pp. 213-229. Springer, 2020.
Jingdao Chen, Yong Kwon Cho, and Zsolt Kira. Multi-view incremental segmentation 3-d point clouds for mobile robots. Robotics and Aulomation Letters, 4(2):1240-1246 2019.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and lly Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pp. 1691- 1703. PMLR, 2020.
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, and Adrian Weller. Rethinking attention with per- formers. In International Conference on Learning Representations, 2021. URL https:
//openreview.neeee htt.://www....ettooooomeeeeeeaae
Roger L Claypoole, Geoffrey M Davis, Wim Sweldens, and Richard G Baraniuk. Nonlinear wavelet transforms for image coding via lifting. Transactions on Image Processing, 12(12):1449-144 2003.
Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen. Up-detr: Unsupervised pre-training for object detection with transformers. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 1601-1610, 2021.
Zihang Dai, Guokun Lai, Yiming Yang, and Quoc Le. funnel-transformee Filtering sequential redundancy for efficient language processing. In Advances in Neural Information Processing Systems, 2020.
Ingrid Daubechies. Ten lectures wavelets. SIAM, 1992.
Ingrid Daubechies and Wim Sweldens. Factoring wavelet transforms into lifting steps. Journal of Fourier Analysis and Applications, 4(3):247-269, 1998.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom,, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. International Confer- ence on Learning Reprexentations, 2020.
Nico Engel, Vasileios Belagiannis, and Klaus Dietmayer. Point transformer. IEEE Access, 9: 134826-1333 134840, 2021.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the International Conference on Computer Vision, pp. 6824-6835, 2021.
Stephen Fitzpatrick and Lambert Meertens. An experimental assessment of a stochastic, anytime, decentralized, soft colourer for sparse graphs. In International Symposium on Stochastic Algo- rithms, pp. 64. Springer, 2001.
Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. Multi-modal transformer for video retrieval. In Proceedings of the European Conference on Computer Vision, pp. 214-229. Springer, 2020.
Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pet: Point cloud transformer. Computational Visual Media, 7(2):177-199 2021.
Alfred Haar. Zur theorie der orthogonalen funktionensystemme. Mathematische Annalen, 69(3): 331-371, 1910.
Thomas Hitchcox and James Richard Forbes. A point cloud registration pipeline using gaussian process regression for bathymetric slam. In International Conference on Intelligent Robots and Systems, pp. 4615-4622, IEFE, 2020.
10

---

Published as a conference paper at ICLR 2022
Huaibo Huang, Ran He, Zhenan Sun, and Tieniu Tan. Wavelet-smet: A wavelet-based emn for multi- scale face super resolution. In Proceedings of the International Conference on Computer Vision, pp. 1689-1697, 2017.
Drew A Hudson and Larry Zitnick. Generative adversarial transformers. In International Conference on Machine Learning. pp. 4487-4499, PMLR, 2021.
Le Hui, Hang Yang, Mingmei Cheng, Jin Xie, and Jian Yang. Pyramid point cloud transformer for large-scale place recognition. In Proceedings of the International Conference on Computer Vision, pp. 6098-6107, 2021.
Li Jiang, Hengshuang Zhao, Shu Liu, Xiaoyong Shen, Chi-Wing Fu, and Jiaya Jia. Hierarchical point-edge interaction network for point cloud semantic segmentation. In Proceedings of the International Conference on Computer Vision, pp. 10433-10441, 2019.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention. In International Conference Ma- chine Learning, pp. 5156-5165. PMLR, 2020.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In International Conference on Learning Representations, 2017.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference Learning Represeniations, 2020. htt.ttts::/ /ol.iittpp:://peeeevieeee
fest/forum?an wot.rrmmmmdrrkgkkktttt..
Jiaxin Li, Ben M Chen, and Gim Hee Lee. So-net: Self-organizing network for point cloud analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9397- 9406, 2018a.
Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointenn: Convolu- tion on x-transformed points. Advances in Neural Information Processing Systems, 31:820-830,
2018b.
Yiqun Lin, Zizheng Yan, Haibin Huang, Dong Du, Ligang Liu, Shuguang Cui, and Xiaoguang Han. Fpconv: Learning local flattening for point convolution. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 4293-4302, 2020a.
Zhi-Hao Lin, Sheng- Yu Huang, and Yu- Yu-Chiang Frank Wang. Convolution in the cloud: Learning deformable kernels in 3d graph convolution networks for point cloud analysis. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 1800-1809, 2020b.
Lin Liu, Jianzhuang Liu, Shanxin Yuan, Gregory Slabaugh, Aleš Leonardis, Wengang Zhou, and Qi Tian, Wavelet-based dual-branch network for image demoiréing. In Proceedings of the Euro- pean Conference on Computer Vision, pp. 86-102. Springer, 2020.
Xinhai Liu, Zhizhong Han, Yu-Shen Liu, and Matthias Zwicker. Point2sequence: Learning the shape representation of 3d point clouds with an attention-based sequence to sequence network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp.
2019a.
Yongcheng Liu, Bin Fan, Shiming Xiang, and Chunhong Pan. Relation-shape convolutional neural network for point cloud analysis. In Proceedings of the Conference Computer Vision and Pattern Recognition, pp. 8895-8904, 2019b.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the International Conference on Computer Vision, pp. 10012-10022, 2021.
Jiageng Mao, Xiaogang Wang, and Hongsheng Li. Interpolated convolutional networks for 3d point cloud understanding. In Proceedings of the International Conference on Computer Vision pp- 1578-1587 2019.
11

---

Published as a conference paper at ICLR 2022
Sunil K Narang and Antonio Ortega. Lifting based wavelet transforms graphs. In Proceedings of Asia--aaaae Asia-Pacific Signal and Information Processing Association, pp. 441-444. Asia-Pacific Signal and Information Processing Association, 2009.
Edouard Oyallon, Eugene Belilovsky, and Sergey Zagoruyko. Scaling the scattering transform: Deep hybrid networks. In Proceedings of the International Conference Computer Vision, pp-
5618-5627, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32: 8026-8037, 2019.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 652-660, 2017a.
Charles R Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J Guibas. Frustum pointnets for 3d object detection from rgb-d data. In Proceedings of the Conference on Computer Vision and Puttern Recognition, pp. 918-927 2018.
Charles Ruizhongtai Qi. Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++:: Deep hierarchical feature learning on point sets in metric space. Advances in Neural Information Processing Systems, 30, 2017b.
Michael Ramamonjisoa, Michael Firman, Jamie Watson, Vincent Lepetit, and Daniyar Turmukham- betov. Single image depth prediction with wavelet decomposition. In Proceedings of Conference on Computer Vision and Pattern Recognition, pp. 11089-11098, 2021.
Maria Ximena Bastidas Rodriguez, Adrien Gruson, Luisa Polania, Shin Fujieda, Flavio Prieto, Ko- hei Takayama, and Toshiya Hachisuka. Deep adaptive wavelet network. In Proceedings of the Winter Conference on Applications of Computer Vision, pp. 3111-3119, 2020.
Xuejian Rong, Denis Demandolx, Kevin Matzen, Priyam Chatterjee, and Yingli Tian. Burst denois- ing via temporally shifted wavelet transforms. In Proceedingy of the European Conference on Computer Vision, pp. 240-256. Springer, 2020.
Raif Rustamov and Leonidas Guibas. Wavelets on graphs via deep learning. Advances in Neural Information Processing Systems, 16:998-1006, 2013.
Fadil Santosa. Second generation wavelets: theory and application. University of Minnesota, Insti- tute for Mathematics and its Applications, 6, 2011.
Yiru Shen, Chen Feng, Yaoqing Yang. and Dong Tian. Mining point cloud local structures by kernel correlation and graph pooling. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 4548 4557, 2018.
Weijing Shi and Raj Rajkumar. Point-gnn: Graph neural network for 3d object detection in a point cloud. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 1711- 1719, 2020.
Wim Sweldens. Wavelets and the lifting scheme: A minute tour. Journal of Applied Mathematics and Mechanics, 76(2):41-44 1996.
Wim Sweldens. The lifting scheme: A construction of second generation wavelets. SIAM Journal on Mathematical Analysis, 29(2):511-546, 1998.
Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. Sparse sinkhorn attention. In International Conference on Machine Learning, pp. 9438-9447, PMLR, 2020.
Hugues Thomas, Charles R Qi, Gean-Emmnuuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the International Conference Computer Vision, pp. 641 166420, 2019.
12

---

Published as a conference paper at ICLR 2022
Mikaela Angelina Uy, Gnang-Hieu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Revis- iting point cloud classification: A new benchmark dataset and classification model on real-world data. In Proceedings of the International Conference Computer Vision, pp. 1588-1597, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Infor- mation Processing Systems, pp. 5998-6008, 2017.
Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chice Chen. Max-dceplab: End- to-end panoptic segmentation with mask transformers. In Proceedings of the Conference on Com- puter Vision and Pattern Recognition, pp. 1463-5474, 2021.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. Transactions On Graphics, 38(5):1-12, 2019.
Travis Williams and Robert Li. Wavelet pooling for convolutional neural networks. In International Conference on Learning Representations, 2018
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianx- iong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the Conference on Computer Vision and Puttern Recognition, pp. 1912-1920, 2015.
Saining Xie, Sainan Liu, Zeyu Chen, and Zhuowen Tu, Attentional shapecontextnet for point cloud recognition. In Proceedings of the Conference on Computer Vision and Puttern Recognition, pp- 4606-4615, 2018
Mingye Xu, Zhipeng Zhou, and Yu Qiao. Geometry sharing network for 3d point cloud classification and segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 12500-12507, 2020.
Mutian Xu, Runyu Ding, Hengshuang Zhao, and Xiaojuan Qi. Paconv: Position adaptive convolu- tion with dynamic kernel assembling on point clouds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3173-3182, 2021a.
Mutian Xu, Junhao Zhang, Zhipeng Zhou, Mingye Xu, Xiaojuan Qi, and Yu Qiao. Learning geomeery-dissentaggee representation for complementary understanding of 3d object point cloud. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 3056-3064, 2021b.
Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, and Yu Qiao. Spidercnn: Deep learning on point sets with parameterized convolutional filters. In Proceedings of the European Conference on Computer Vision, pp. 87-102, 2018.
Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. scalable active framework for region annotation in 3d shape collections. Transactions on Graphics, 35 2016.
Li Yi, Hao Su, Xingwen Guo, and Leonidas J Guibas. Syncspecenn: Synchronized spectral enn for 3d shape segmentation. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 2282-2290, 2017.
Xumin Yu, Yongming Rao, Ziyi Wang, Zuyan Liu, Jiwen Lu, and Jie Zhou. Pointr: Diverse point cloud completion with geometry-aware transformers. In Proceedings of the International Confer- ence on Computer Vision, pp. 12498-12507, 2021.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. Advances in Neural Information Processing Systems, 30, 2017.
Yanhong Zeng, Jianlong Fu, and Hongyang Chao. Learning joint for video inpainting. In Proceedings of the European Conference on Computer Vision, pp. 528- 543. Springer, 2020.
transformations
13

---

Published as a conference paper at ICLR 2022
Dong Zhang, Hanwang Zhang, Jinhui Tang, Meng Wang. Xiansheng Hua, and Qianru Sun. Feature pyramid transformer. Proceedings of the European Conference on Computer Vision, pp. 323- 339. Springer, 2020.
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the International Conference on Computer Vision, pp. 16259-16268, 2021.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: De- formable transformers for end-to-end object detection. In International Conference on Learning Representations, 2021.
14

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.I TABLES FIGURES FOR ABLATION STUDY
In this section, we list tables and figures for ablation study in Section 6 for better understanding of the proposed model.
Method ModeiNet-40 OBJ.ONLY OBJ_BC
randomn
89.6
82.9
83.8
No.
K-NN self
upproximation
detail voking accuracy (%)
FPS
92.0
85.3
84.2
91.4
app.
93.4
88.0
86.3
3
89.8
Table 5: Effect of different point split schemes on shape classification.
92.9
6
92.5
7
92.7
8
934
Method
Accuracy
Pre computed
91.8
Dynamic
Table 4: Results of ablating each module in AWT-Net on ModelNet40.
Table 6: Effect of adjacency matrix computation schemes.
Method
Z-rot
Xyz-rot
Number of scales
1
2
4
PointNet Qi et
201
81.6
66.3
Overall
92.5
93.4
93.1
91.3
PointNet++
2017b
90.1
87.8
Radio
90.0
83.5
75.0
75.0
SpiderCNNN Xu et al
2018
83.5
69.6
Wardrohe
75.0
71.0
DGCNN (Wang
Stool
75.0
78.0
85.0
85.2
GDA-Net Xii ef at
20216)
91.2
90.5
Person
85.0
88.5
95.0
100.0
AWT-Net (ours)
91.4
90.2
Table 7: Effect of decomposition levels on ModelNet40. Note that the results achieved without voting during test.
Table 8: Accuracy ModelNet40 against rotation. rot: randomly rotate training and test shapes along z axis; xyz-rot: randomly rotate training and test shapes along xyz axis.
1024
512
256
128
64
Figure 5: On the right, the curves show the classification results of AWT-Net tested on the 3D shapes with different numbers of randomly sampled points (shown on left).
A.2 MORE VISUALIZATION OF DECOMPOSITION OF SHAPPS
We present more visualization of detail components of multiple shapes from the test set of Model- Net40 (Wu et al. 2015) in Figure [ Such geometric representations are learned implicitly in 3D shape classification task without component supervision.
A.3 EXECUTION TIME AND MODEL PARAMETERS
We compare the number of parameters and computational overhead w.r.f. GDA-Net (Xu et al. 2021b) on ModelNet40 and the results are listed in Table 9 We measure four scales. Note that GDA-Net consists of two building blocks, which is fairly comparable "scale 2" in our model. In this case, though our model contains around 25% more parameters, the execution time is only 13% slower than GDA-Net, which shows the practicability of our model.
15

---

Published as a conference paper at ICLR 2022
Figure 6: The detail component of chair, table, guitar, sofa, stool, wardrobe, and shelf. Colors represent the absolute values of the detail coefficients per point. The larger values (redder colors), the higher probability that these points locate on high-frequency parts, e.g, edges non-flat areas.
Model
Number of Params
Time
GDA-Net (Xu et al. 2021b)
0.94 M
1.057
AWT-Net (scales I)
0.97 M
0.843
AWT-Net (scales
1.17 M
1.191
AWT-Net (scales 3)
1.38 M
1.657
AWT-Net (scales 4)
1.60 M
2.024
Table 9: The number of parameters and computational overhead (time) comparison. The compu- tational overhead is measured in second for a single forward and backward pass. All results are average over ten trails.
A.4 RANDOMNESS IN EVEN-ODD NODE SPLIT
We compare the effect of using even-odd split and random split in Algorithm I and the results are listed in Table T0 Using random split performers slightly worse than even-odd split. With even- odd split, we can separate any two connected nodes into two different sets to the best extent, thus ensuring each local area contains both even and odd nodes. However, with random split, some local areas might contain only one type of nodes, thus failing to be decomposed properly.
Number of scales
3
Even-odd split
92.5
93.4
93.1
91.3
Random split
91.7
92.9
92.6
90.9
Table 10: Classification results of using even-odd split and random split in Algorithm
16


Reviews:
[{"review_id": "jBixFE8mFQ", "reviewer": "Reviewer_tf1v", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The paper is well written and provides sufficient background knowledge.\n2. Applying wavelet transform for 3D representation learning seems quite new and is not much explored before.\n3. For the experiment results, the numbers themselves are stronger and ablation study is well conducted. \n\nCons & questions:\n1. My biggest concern is about the results comparison, even though the final number itself is strong. Table 1 shows the proposed AWT-Net achieves the best results when using Performer (AWT-Net +P). It a little bit unfair since others transformer-based prior works (e.g. PointTransformer, PCT) use vanilla Transformer. Their results might also be improved with Performer. If we look at only results for 'AWT-Net +V' in Table 1 and Table 3, the numbers fall behind prior transformer-based works like PointTransformer and PCT. Is using Performer a must? In addition, why doesn't Table 2 also show the result for 'AWT-Net + V'? I think it's better to put it on if there are no particular reasons.\n2. The module analysis of ablation study is sort of unclear to me. First, what exactly is the baseline (the first row in table 4)? It's unclear to me what's the network architecture after ablating all the modules. I feel it's better to have a supplementary document describing the details. Second, the improvement (from 92.3 to 93.4) by adding ‘approximation’ and ‘detail’ doesn't seem that much. It is because this task is already saturated? Or gives a convincing evidence that it is a considerable improvement.  \nIs there any other way to demonstrate the effeteness of using wavelet decomposition? The visualization in Figure 1 is a good example to understand the learned features but could potentially be explored more (e.g. how is hierarchy/multi-resolution reflected here).\n3. I'm also curious about the execution time. The proposed method is quite complex (with split scheme, GCN and Transformer), so I wonder if it will be much slower than previous works. A table showing the execution time and model parameters would be much helpful.\n4. Will the randomness in even-odd node split (Algorithm 1) affect the results? Just want to confirm different node partition for the same 3D shape does not much affect the learned feature.\n\nMinor point:\n- The resolution of Figure 5 right is too low. Consider using a pdf version.", "summary_of_the_paper": "The paper presents a new deep neural network architecture for 3D point cloud representation learning, based on wavelet decomposition. In particular, the authors propose a data-driven adaptive lifting scheme that introduces non-linearity into wavelet. The original linear operators update(U) and predict(P) in wavelet decomposition are replaced by non-linear graph convolutional networks (GCN). Equipped with wavelet transform and Transformers, the proposed network aims to captures and refines the holistic and complementary geometry of 3D shapes to supplement neighboring local information. Experimental results on standard benchmarks (i.e., shape classification and part segmentation) show that it achieves state-of- the-arts or competitive performance.", "summary_of_the_review": "In general, the proposed method is novel in that it combines wavelet transform into 3D representation learning (I'm not an expert in wavelet transform, so I'm not quite sure how challenging this is). Experiments and ablation study are comprehensive, and the final quantitative numbers are competitive. My biggest concern is that the improvement over prior works are quite limited (no more than 0.2%), given all the complex schemes of the method and its required usage of Performer. I understand these benchmark results are somewhat saturated and hard to improve a lot. It's ok, but the paper will be more convincing if the authors can analyze the features learned by their wavelet transform, and show how the learned features are better or different from prior works that do not use wavelet transform.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KwfEgss4Ls", "reviewer": "Reviewer_2wxQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- The authors propose a novel framework for point cloud processing, which is inspired by the lifting scheme in wavelet transformation. This new approach, equipped with performer and GCN, makes it state-of-art in point cloud classification and segmentation. \n\n- Extensive experiments on the performance of the model, especially w.r.t number of scales, point sparsity and rotations. \n\n- This paper is well written, where readers can easily understand the design of the proposed method, and is clear enough for reproducing the method without any code references. \n\nWeakness:\n\n- Major concern\n\n   1. A lack of consistent motivation of using a lifting scheme.\n    Though using the lifting scheme is novel, it is less motivated during the narrative of the paper. The only direct motivation is more or less the Fig.1, where the detail features sort of concentrates on 'junctions'. But it's still hard to interpret, and the number of qualitative results are too limited. I like the lifting scheme story, but I don't think it's being motivated enough through the paper. \n    \n    What's the unique edge of using this lifting scheme, apart from the few point gains if you use a voting scheme + performer? Does it learn meaningfully samplings of the original point could? does it highlights consistent keypoints within a semantic class?\n  \n   2. Are the comparisons fair?\n       I'm not really comfortable comparing the proposed method using the performer. Though it gives a few point in performance, I'm not sure if that's fair for evaluating the lifting scheme against other baselines. This relates closely with concern 1, since I believe the merit of having a lifting scheme should not just be the bold numbers, and it's okay if it's not: it just to be on par with state-of-the-art. \n\n   3. Why more scales hurt?\n       I don't really understand why more scale hurts. Why does it introduce `redundant information`? I think it's introducing more and more redundant representation, but not necessarily redundant information. Is it because the more scales you have, the more over-parametrized you are and therefore more prune to overfitting? \n\n\n\n\n- Minor concern\n\n1. Is the name of wavelet transform adequate?\n    This may or may not be a huge issue, but it's less intuitive for me to see this as a wavelet transformer, but more or less a lifting transformer. The wavelet in the title sort of suggests there's wavelet filters involved, but it's more like putting neural networks(transformers)  in a lifting scheme. \n\n\n", "summary_of_the_paper": "This paper proposes a novel 3D point cloud representation learning framework. At the core of this method, is a lifting scheme inspired by wavelet decomposition. The proposed method roughly splits the input data in half at each stage, producing a down-sampled approximation C and detail d. Then C is further processed by the next layer, forming a multiscale pyramid. In summary, the contribution of this paper is:\n\n1. Proposed to use the lifting scheme in point cloud processing, using graph convolution networks and transformers as backbone.\n\n2. Evaluated the method against state-of-the-art baselines and showed that the proposed scheme performs well.\n", "summary_of_the_review": "In summary, this paper proposes a method for processing point cloud using a lifting scheme. The proposed method is novel, and with some bells and whistles, it achieves the state-of-art performance on point cloud segmentation and classification. However, there's some major issues blocking me from rating this paper above the acceptance bar. First of all, it is not super clear to the reader why one would like to incorporate the lifting scheme for point cloud processing. I don't think it is just for a few points gain in the performance. Secondly, there's some issues in the evaluation due to the choice of the performer vs plain transformer.\n\nI'm willing to raise my score if the authors could address those concerns. \n\n------------------------\n\nAfter reading the response from the authors, I would like to raise my score from 5 to 6. The authors have addressed my concerns, and I think this paper meets the bar of ICLR. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HHsxAA3tgB2", "reviewer": "Reviewer_Dnca", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths: \n1) The paper is well-organized, the experiments are sufficient demonstrating the effectiveness of the proposed framework design. \n2) Relying on wavelet decomposition for 3D shape learning is theoretically sound and interesting. \n3) The authors made many efforts on the framework design, like the proposed new adaptive lifting scheme, the transformer etc. \n\nThe weaknesses:\nMy major concern is about the performance. It seems the performance is just comparable with existing methods. This brings a doubt to me: is it really necessary to use wavenet, a relatively new framework? ", "summary_of_the_paper": "This paper presents a novel framework for 3D shape representation learning, which is based on multi-scale wavelet decomposition. This is very different from existing works. A novel transformer-based neural network, AWT-Net, is also proposed.", "summary_of_the_review": "Although the proposed method is theoretically sound and the framework design makes sense, the performance gain is hard to convince me the superiority of such a different learning framework.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "H0BgWzqnH3j", "reviewer": "Reviewer_fDxy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths:\n------------\n\n1. The idea of decomposing 3D shapes into sub-bands components is really nice and to the best of my knowledge such representation has not been explored in prior work.\n2. The paper is clear, nicely written and easy to follow.\n3. The proposed model outperforms prior work on the object classification task on ModelNet40 dataset and on ScanObjectNN dataset and on the object part segmentation task on the ShapeNet Part dataset.\n\n### Weaknesses:\n-------------\n\n1. While I acknowledge that the paper outperforms prior works on both the classification and the object part segmentation task, the proposed model is marginally better than previous methods. For example for the object classification task, the proposed model is only 0.1% better than GDANet on the ModelNet40 dataset, while it performs on par with it on the ScanObjectNN dataset. Therefore, I am wondering how statistically significant are these experiments?\n2. GDA-Net by Xu et al. is performing comparably well  with the proposed method. However it is not thoroughly discussed in the Related Work section. Therefore I recommend that the authors describe the differences between their model and GDA-Net in detail so that the reader can easily understand what are the differences between the two models.\n3. Various implementation details are missing. For example what are the weights $\\lambda_1$ and $\\lambda_2$ in Eq (2)? What optimizer did the authors use? For how many epochs was their model trained? What was the learning rate? Unfortunately, I wasn't able to find these details in the main paper. However, the authors should provide them in order for the results of the paper to be reproducible.\n\n### Questions / Detailed Comments:\n---------------------------------\n\n1. From Table 1, it seems that the proposed method using the vanilla Transformer performs slightly worse than the variant that uses the Performer. Can the authors provide some intuition for this? I would have expected that using Perform would result in faster inference, however it is not clear to me why it also results in improved performance.\n2. What is the number of scales used for the classification accuracy experiment. In Table 1, the proposed model achieves 93.9 accuracy, however from the ablation study in Table 7, the best performing model achieves 93.4 with 2 scales. So how under which configuration was the performance of Table 1 achieved?\n3. In prior work, authors typically consider the shape classification task on the S3DIS dataset that contain indoor scenes. I think that the claims of the paper would be even stronger if the authors added an additional evaluation on the more challenging S3DIS dataset.\n4. For the Object Segmentation experiment, in Table 3, the authors should also include the results from Point Transformer by Zhao et al. that achieves 83.7 class mean IoU and 86.6 instance mean IoU, as stated in the original paper.\n5. For the Even-Odd Split operation, it is not clear to me, why it is necessary to have both a weighted and an unweighted adjacency matrix. Can the authors please comment on this?\n6. Some additional references are missing in the Transformers in Vision Section:\n- In addition to PCT that applies transformers for 3D pointcloud classification and segmentation, the authors should also mentions the more recent works of:\n    * Point Transformer, Zhao et al. ICCV 2021\n    * PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers, Yu et al. ICCV 2021\n- In addition to the mentioned works on efficient transformers, the authors should also mention the work of \"Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention\" by Katharopoulos et al. ICML 2019, since it was among the first to address the quadratic complexity of transformers\n\n### Minor Comments / Typos:\n-------------------------\n\n- In Page 2, in Section Introduction \"state-of-the-arts\": should be \"state-of-the-art\"\n\n", "summary_of_the_paper": "This paper proposes a new 3D shape representation learning method using multi-scale wavelet decomposition. In particular, the authors introduce a neural network architecture that decomposed 3D shapes into sub-bands components at multiple scales. In particular, starting from a pointcloud the proposed model learns to decompose it into coarse (high frequency) and detail (low frequency) components using an adaptive lifting scheme, similar to the original lifting scheme introduced for defining second-generation wavelets. Subsequently, two transformer models are used to refine the coarse and approximate geometry of the 3D shape. The proposed model achieves state-of-the-art results on the shape classification task on the ModelNet40 and the ScanObjectNN dataset and on the part segmentation task on the ShapeNet Part dataset. The concept of using such an adaptive lifting scheme seems to facilitate learning and to the best of my knowledge is novel for the case of shape representation learning.", "summary_of_the_review": "Overall, I really like this paper. It was easy to ready and nicely written. Moreover, the proposed model outperforms previous methods on the scene classification task on multiple datasets and on the part segmentation task. While the proposed model is marginally better than previous models, I really like the concept of utilizing wavelets for 3D shape modelling thus I vote for accepting this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission initially received mixed reviews. The authors presented convincing answers during the author response period, after which all reviewers recommended weak accepts. The AC has carefully read the reviews, responses, and discussions, and agreed with the reviewers' recommendation. Despite the marginal performance gains, the submission has presented a useful and inspiring way of learning shape representations. The AC, therefore, recommends acceptance.

The authors are encouraged to further revise the paper based on the reviews. In addition, the authors should use $\citep$ for all citations that are not used as a pronoun, including all citations in the tables. Please find more information here: https://journals.aas.org/natbib/
Decision: Accept (Poster)