SageMix: Saliency-Guided Mixup for Point Clouds
Authors: Sanghyeok Lee
Minkyu Jeon
Injae Kim
Yunyang Xiong
Hyunwoo J. Kim
Venue: NeurIPS 2022 Accept (2022)
Keywords: point cloud
mixup
saliency

Abstract: Data augmentation is key to improving the generalization ability of deep learning models. Mixup is a simple and widely-used data augmentation technique that has proven effective in alleviating the problems of overfitting and data scarcity. Also, recent studies of saliency-aware Mixup in the image domain show that preserving discriminative parts is beneficial to improving the generalization performance. However, these Mixup-based data augmentations are underexplored in 3D vision, especially in point clouds. In this paper, we propose SageMix, a saliency-guided Mixup for point clouds to preserve salient local structures. Specifically, we extract salient regions from two point clouds and smoothly combine them into one continuous shape. With a simple sequential sampling by re-weighted saliency scores, SageMix preserves the local structure of salient regions. Extensive experiments demonstrate that the proposed method consistently outperforms existing Mixup methods in various benchmark point cloud datasets. With PointNet++, our method achieves an accuracy gain of 2.6% and 4.0% over standard training in ModelNet40 and ScanObjectNN, respectively. In addition to generalization performance, SageMix improves robustness and uncertainty calibration. Moreover, when adopting our method to various tasks including part segmentation and standard image classification, our method achieves competitive performance. Code is available at https://github.com/mlvlab/SageMix.

SageMix: Saliency-Guided Mixup for Point Clouds
Sanghyeok Lee"
Minkyu Jeon' Korea University jmk94810ckoren ac.kr
Korea University
cat06266korea.co ac kr
Injae Kim Korea University 1000041@0orea.ccmm ac.kr
Yunyang Xiong Meta Reality Labs yunyangefb. com
Hyunwoo J. Kim
Korea University
Brunwoojkim@kore ac .rr
Abstract
Data augmentation is key to improving the generalization ability of deep learning models. Mixup simple and widely-used data augmentation technique that has proven effective in alleviating the problems of overfitting and data scarcity. Also, recent studies of saliency-aware Mixup in the image domain show that preserving discriminative parts is beneficial to improving the generalization performance. However, these Mixup-based data augmentations are underexplored 3D vision, especially in point clouds. In this paper, propose SageMix, saliency-guide Mixup for point clouds to preserve salient local structures. Specifically, we ex- tract salient regions from two point clouds and smoothly combine them into continuous shape. With a simple sequential sampling by re-weighted saliency scores, SageMix preserves the local structure of salient regions. Extensive exper- iments demonstrate that the proposed method consistently outperforms existing Mixup methods in various benchmark point cloud datasets. With PointNet++, our method achieves an accuracy gain of 2.6% and 4.0% over standard training in 3D Warehouse dataset (MN40) and ScanObjectNN, respectively. addition gener- alization performance, SageMix improves robustness and uncertainty calibration. Moreover, when adopting our method various tasks including part segmen- tation and standard 2D image classification, our method achieves competitive performance. Code is available at htttps://githbb womp:////mm//aaa////aii
1 Introduction
Deep neural networks have achieved high performance in various domains including image, video, and speech. Recent researches [ 5 have been proposed to employ deep learning model for 3D vision, especially in point clouds. However, in the point cloud domain, deep learning models prone to suffer from week--enerallizttie performance due to the limited availability of data compared to the image datasets, which contain almost millions of training samples. For alleviating the data scarcity issue, data augmentation is prevalent solution to increase the training data.
In the image domain, Mixup-based methods augmented samples. More recent Mixup methods 10 12 focus on leveraging saliency to preserve the discriminative regions such as foreground objects. Despite the success of saliency- aware Mixup methods, it has been less studied in point clouds due to its unordered and non-grid structure, Moreover, saliency-aware augmentation methods require solving an additional optimization problem [2 12] resulting in a considerable computational burden.
combine two training images to generate
First two authors have equal contribution.
Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

PointMixup
chair
guitar
RSMix
Ours
Figure 1: Comparison of Mixup methods on point clouds. Given two original samples, (left) guitar and (right) chair, generate samples by various Mixup methods. (Top) PointMixup does preserve the discriminative structure. (Middle) The samples generated by RSMix contain the local structure of each sample, but the discontinuity occurs the border. (Bottom) Our method, SageMix, generates a continuous mixture preserving the local structure of original shapes.
Recently, several works 14 PointMixup 14] enables the linear interpolation point clouds based on optimal assignment. prepeepreseriing Mixup framework that extracts and merges the rigid subsets of each sample. However, these approaches have limitations. PointMixup generates samples without preserving the local structure of the original shapes. For example, in the top row of Figure | the structure of the guitar is not preserved in generated samples. RSmix generates discontinuous samples and these artifacts often hinder effective training. Furthermore, these methods disregard the saliency, thereby causing the loss of the discriminative local structure in the original point cloud.
15 16 have attempted extend the concept of Mixup point clouds.
RSMix [15] proposed
In this paper, propose Saliency Guided Mixup for point clouds (SageMix) that preserves discriminative local structures and generates continuous samples with smoothly varying mixing ratios. For saliency estimation, measure the contribution of each point by the gradients of loss function. Through a simple sequential sampling via re- re-weighted saliency scores, SageMix samples the query points extract salient regions without solving additional optimization problems. Then, SageMix smoothly combines point clouds, considering the distance to query points to minimize the loss of discriminative local structures illustrated in Figure T (e.g., the neck of guitar and the back of chair, etc.).
In summary, contributions fourfold:
We propose a novel saliency-guided Mixup method for point clouds. To the best of our knowledge, this is the first work that utilizes saliency for Mixup point clouds.
We design a Mixup framework that preserves the salient local structure of original shapes while smoothly combining them into one continuous shape.
Extensive experiments demonstrate that SageMix brings consistent and significant improve- ments state-of-the-art Mixup methods in generalization performance, robustness, and uncertainty calibration.
We demonstrate that the proposed method is extensible to various tasks including part segmentation and standard image classification.
Preliminaries
In this section, we briefly review the basic concept of Mixup and summarize the variants of Mixup for images and point clouds [T.

---

Table 1: Variants of Mixup.
Method
Equation
mixup
i Ara + (1 X)r3
Manifold mixup 7
h Aha + (1 X)h
Image
CutMix
8
I M ④ Fa + (1 M) ④ T
SaliencyMix 13
I ( G Ia + +(1 M (Ta) 0zg
Puzzle Mix
I Z Hl To +
Z) 3
Co-Mixup [2]
I D: Ze Nr
PointMixup 14
P {Apt + (1 (1-A) X))
3 in
Point cloud
RSMix 15
P (pa - -S")
SageMiix
P (ApE + +11 >))
Vicinal risk minimization with Mixup. Given observed data D {(z,, De 9:) and function x + V, Chapelle et al. [17] learns the function f by minimizing the empirical vicinal risk: Rx(f) 1 mn
(()()))) Is). where is the loss function, and (ø, ū) is the virtual feature- target pair from the vicinal distribution v of the observed data D. To construct an effective vicinal distribution in the image domain, Zhang et al. [9] introduced Mixup:
ā a (1 N)mS
Ayo + (1 A)ya,
where (Ia: Va), (zg. y8) are two pairs of data in training dataset D, and X Beta(0, 0) is mixture ratio. Following (9] Verma et al. 771 proposed Manifold Mixup that applies Mixup in hidden representations (i.e., h = Aha + (1 A)ha) and CutMix S generates samples via cut-and and-ppate manner with binary mask M E {0, 1} (i.e. = M C za + (1 MM) @zz, where @ indicates the element-wise product). Combining with saliency, SaliencyMix [3] improved CutMix by selecting the patch M with the maximum saliency values. Puzzle Mix optimizes the mask E [0, 1] x and transport for maximizing the saliency of the mixed sample (i.e., _ Z 2 1 za + (1- oIl zz).
Beyond two samples, Co-Mixup mixes multiple samples in a mini-batch by optimizing multiple masks (ie., ã = Σ. Z, ① 24).
While saliency-aware Mixup methods boost the generalization of deep learning models, they originally designed for the image domain. Hence, these methods are not directly applicable to point clouds due the unordered and irregular structure.
Mixup in point cloud. Several works 14 15] tried to leverage the Mixup in point cloud. Point- Mixup [4] linearly interpolates two point clouds by
p DPp + (1- 2)
(1 (1)) ū= Xy + (1 N)y
(2)
n Σ lp Pl1
6- argmin 6 +
(3)
where Pd = {pl .... p the set of points with E fa, 3). n is the number of points, and 6" {1..... n) + {I,..., n) is the optimal bijective assignment between two point clouds. In RSMix [15,, they generate an augmented sample by merging the subsets of two objects, defined as p (pa 1 SS)) U 5 where St c pt is the rigid subset and S denotes S translated to the center of S
Although these methods have shown that Mixup is effective for point clouds, some limitations have remained unresolved: loss of original structures, discontinuity at the boundary, and loss of discrininative regions. Here, to address these issues, we propose a new Mixup framework in the following section.

---

Eq (4)
(6)
w a
Pa
Eq (8)
B
ll - 9-1l
Eq (5)
(6)
P8
1ll d" ||s
w
8
Figure 2: Illustration of SageMix pipeline. Given pair of samples Pa, and pa, SageMix sequentially samples the query points q², and de (green) based on saliency using Equation and Equation ( respectively. Then, using smoothly varying weights with respect to the distance to query points, SageMix generates an augmented sample P preserving the salient local structures.
Method
The main goal of SageMix is to generate an augmented sample that 1) maximally preserves the salient parts, 2) keeps the local structure of original shapes, and 3) maintains the continuity in the boundary, To achieve this goal, SageMix extracts salient regions centered at query points and smoothly combines them through continuous weights in Euclidean space. The overall pipeline is illustrated in Figure and pseudocode is provided Algorithm []. In this section, we delineate the process for selecting the query point based on the saliency map in Section 3.1 Mixup methods for preserving the salient local structure in Section 3.2
3.1 Saliency-guided sequential sampling
We first introduce query point y' E pp that is considered the center of the region for preserving the local shape. A naive random sampling for a query point is simple, but does not guarantee that is placed in a salient region. Thus, we propose saliency- -uiided query point selection for maximally maintaining the salient part. We denote the saliency St of the input p' by the norm of the gradient (i.e., S p (f(()) y²) II), following [2] Deterministically selecting a query point with maximum saliency score (i.e., 9º ph.. where i" = arg max;
max;(S!)) (S!)) sounds promising but in practice, this makes it difficult to generate diverse samples since query points are always located at the same position. Moreover, if two query points q", are closely located in the Euclidean space, is challenging preserve the local structure of each sample because of the significant overlap.
To address these issues, SageMix sequentially selects the query point based the saliency scores. The first step is to extract the query point gd = pf.. To maximize the diversity, the query point is sampled with respect to the probability distribution defined
Pr( i) Σ sp
(4)
where s" E S" is the saliency of pf and _" is random variable for index (i* ~ I*), That is, the points salient region have high chance to be chosen as query point. This simple sampling method efficiently provides diverse query points, thereby minimizing redundant selections. Given query point 9t the next step define a sampling method for q alleviate the overlap between selected parts. We encourage the sampler to distance q from o by reweighting the saliency scores:
1/ 9al 1
Pr =) = Σn 11 9rd ||s
(5)

---

Algorithm A saliency-guided Mixup for point clouds
Input: pa, ,P", yf y , P= (PI,..., ....
0, .5,
"P.....PPP. Pn set of points, y : target, S {s1, sn) saliency values, bandwidth, ø assignment function, f shape parameter
Output: P.
Draw q" P from Pa w.r.t. Pr( =)
of
Equation
2: Draw q² = p from Pd w.E.t Pr(1 =1) ΣE 1p -9 ||o
11p - 11
Equation
3:
for
1 to do
4:
w wi f Ko (ptt 9²), ; Ko (olpi (P.
>. Equation 6
5:
A σuς +(1 (1-1))1)) 77o 616)
> Beta(0, 0). Equation (9
6: pj f
Xxp, + (1 A,)p 6(i)
> Equation
7: end for
x + n 1 A3
9: + Ay + (1 (1-1) 2)y
10: return P = fp..... P.u.
8 Assuming the point pa d", the probability of selecting the point oe decreases. This implies that SageMix samples a query point considering both distance and saliency. Finally, with Equation 4 and Equation (5), we can obtain the remotely located query points with high saliency scores, resulting in augmented samples preserving the discriminative structures of original shapes.
3.2 Shape-preserving continuous Mixup
Since our objective is preserve the region around the query point , we need impose high weight on the points near the query point. Further, to alleviate the discontinuity between two samples, the weight should smoothly vary in Euclidean space. Herein, we use Gaussian Radial Basis Function (RBF) kernel to calculate weights:
wf Ko(..g') = exp (- (-Ilp)- 1 2'113) (
20
where w/ is the weight on the point p\ and a E R. is bandwidth for kernel. The weight wi smoothly increases the distance to q decreases, resulting in the region around the query point being prone to respect its original shape. As shown in (Figure 2). the parts with higher weights maintain their original local structure more in mixed sample, e.g., the body of the guitar and the tail of the airplane.
Given two point clouds (Pa, pd) and their corresponding weights (wwf N {wr }r), generate an augmented sample via wose interpolation. For differentially mixing points, we define the mixing ratio for the i-th point pair
1
X
((1)
w
6(1)
(1)
where ò is the assignment in Equation 33. Note, that the point mixing ratio X is the ratio between the weight of two paired points (i.e., 1 ((1) Ag). thus enabling linear interpolation. For
simplicity, we use the notation A4o (1 A₂) instead of Ag 2 pray Then, we generate the virtual point cloud by modifying the Equation (2 to
p (ur + N
1 Ay + (1 A)y²,
(8)
where X >= - X; can be interpreted as the overall mixing ratio that is used label interpolation. In Mixup, the distribution of À is a key factor for model training [9. To control the distribution of X,
5

---

Table 2: 3D shape classification performance on MN40/OBJ_ONLL/PP) B_00___TS_S___
Dataset
Model
Method
MN40
OBJ_ONLY
PB_T50_RS
Base
89.2
79.1
65.4
PointMixup RSMix [15
14
89.9
79.4
65.7
PointNet
88.7
77.8
65.7
SageMix
90.3
79.5
66.1
Base
90.7
86.5
79.7
PointMixup 14
92.3
87.6
80.2
PointNet++
RSMix [S]
91.6
87.4
81.1
SageMix
93.3
88.7
83.7
Base
92.9
86.2
79.9
+ PointMixup 14
92.9
86.9
82.5
DGCNN
RSMix [15
93.5
86.6
82.2
SageMix
93.6
88.0
83.6
we introduce prior factor ~ Beta (o,a(O,0) 0):
TWF
A TUT + (1 - π wu (1))
(9)
Remarks. Our SageMix emulate both PointMixup and RSMix depending the bandwidth If the bandwidth a is sufficiently large, Ko (pi. g¹) Re K (pf, q²) = c, Vigj, where c E R4, resulting in That is, all points mixed in the same ratio as PointMixup. Conversely, when the bandwidth becomes smaller, SageMix changes the mixing ratio drastically around the boundary between the two shapes and generates augmented sample like RSMix. Qualitative results are available in Section See Section 4.2 for qualitative results. Additionally, with minor change, SageMix is applicable in feature space as Manifold Mixup, see Appendix B.2 for more discussion.
A; ne+(I=a}e
4 Experiments
In this section, we demonstrate the effectiveness of our proposed method SageMix with various benchmark datasets. First, for 3D shape classification, we evaluate the generalization performance, robustness, and calibration error in Section 4.1 Next, provide ablation study and analy- ses of SageMix Section 4.2 Lastly, we study the extensibility of our method in Section 4.3 Implementation details provided in Appendix A.
Data. We use two benchmark dataset: 3D Warehouse dataset (MN40) 18 and ScanObjectNN 19 MN40 is a synthetic dataset containing 9,843 CAD models for training and 2,468 CAD models for evaluation. Each CAD model of MN40 is obtained from 3D Warehouse (20] ScanObjectNN: obtained from SceneNN (2]] and ScanNet (22), is a real-world dataset that split into 80% for training and 20% for evaluation. Among the variants of ScanObeectNN, (OBJ ONLY) and the most challenging version (PB T50 RS). For training models, we use only coordinates (x.y.z) of 1024 points without additional information such the normal vector.
adopt the simplest version
For comparison with previous studies, we use three backbone models: PointNet 3 PointNet++ and DGCNN (.. We compare SageMix with the model under default augmentation in 3 4 19 (Base), and other Mixup approaches (PointMixup [1) RSMix (15)). We report
Baselines.

---

Table 3: Robustness and calibration with DGCNN on OBJ_ONLX
Gaussian noise.
Rotation 180
Scaling
Dropout
Calibration
Method
d' 0.01 o': 0.05
X-axis Z-axis
x0.6 x2.0
25% 50%
ECE(%)
DGCNN 6
84.9
48.4
32.5
32.4
73.7
73.0
83.3
75.7
19.8
PointMixup
14
85.0
61.3
31.7
32.7
73.8
73.0
84.2
74.9
6.8
RSMix [S
84.2
49.1
32.7
32.6
75.0
74.5
84.0
73.6
18.9
SageMiix
85.7
51.2
36.5
37.9
75.6
75.2
84.9
79.0
5.1
Table Ablation of Saliency-guided sequential sampling.
Metric
DGCNN
6
Uniform
Max
Saliency only
SageMix
OA
86.2
86.8
86.1
87.8
88.0
the performance in overall accuracy. We highlight the best performance red and second-best performance in yellow.
4.1 3D shape classification.
Generalization performance. Table summarizes the experimental results of 3D shape classi- fication three datasets. Our framework significantly outperforms all of the previous methods in every dataset and model. Although the datasets are quite saturated, the averages of the improvements against the Base with PointNet, PointNet++, and DGCNN are 0.7%, 2.9%, and 2.1%. respectively. With PointNet+++, SageMix improves the overall accuracy by 2.6%, 2.2%, and 4.0% compared to Base MN40, OBJ ONLY, and PB T50 RS, respectively. We observe similar performance improvements in DGCNN by 0.7%, 1.8%, and 3.7% over Base. These consistent improvements demonstrate the effectiveness of our framework.
Robustness. We adopt DGCNN and OBJ_ONL to evaluate the robustness of models trained by
our method. We compare our method with previous methods [5] on four types of corruption: (1) jittering the point cloud with Gaussian noise (o' E 0.01, 0.05), (2) Rotation 180" (3) Scaling with a factor (0.6, 2.0), and (4) Dropout 25% or 50% all points. As shown in Table 3 SageMix consistently improves the robustness in various corruption. Except for Gaussian noise with o' = 0.05,
DGCNN trained with SageMix shows the best robustness with significant gains compared to previous methods. Specifically, SageMix outperforms the other methods with gains of 5.5% for rotation (Z-axis), 2.2% for scaling (x2.0), and 3.3% for dropout (50%) over base DGCNN.
Calibration. Previous works 23 have proven that deep neural networks tend be over- confident resulting in poorly calibrated models. In other words, a well-calibrated model should provide accurate probability according to its predictions. Here, we report Expected Calibration Error (ECE) (33 to estimate the uncertainty calibration of models. We use the same setting as the robustness test for reporting ECE (Table Overall, this result reveals that SageMix provides the best performance on uncertainty calibration compared to the other methods. See Appendix B.3 for more results.
4.2 Ablation study and analyses
We provide various quantitative and qualitative analyses for better understanding of SageMix. We use DGCNN and QBJ_ONLY for an ablation study and MN40 for visualization.
Ablation on query point sampling. We explore the effectiveness of saliency-guides sequential query point sampling. Table shows the results with various sampling methods for query points. A naïve uniform sampling for query point (Uniform), without considering the saliency, introduces +0.6% gains over base DGCNN. Interestingly, when SageMix (deterministically) selects the query point with maximum saliency values (Max), the performance is even degraded (-0.1%) 10).. With our
7

---

Table
0.1
== 0.3
0.5
= 0.7
π = 0.9
Airplane
σ = 0.25
σ = 0.5
σ = 0.75
o 0=1
0 =
Figure 3: Qualitative analyses prior factor and the bandwidth Given two samples (left) table and airplane, SageMix generates sample based on (Top) various и with a fixed 0.3 and (Bottom) various with fixed ı = 0.5. Note that the green points indicate query points.
Table Quantitative analysis on the bandwidth
0.1
0.3
0.5
1.0
2.0
OA
87.2
88.0
87.6
87.3
87.6
saliency-guide random sampling for two query points based only on Equation (4) (Saliency Only), the performance is significantly improved 87.8% (+1.6%). These results suggest that although saliency is key to effective training, we also have to consider the diversity for generating new sample. Finally, the best accuracy 88.0% (+1.8%) is obtained by our saliency-guie guided sequential sampling considering saliency and the distance between query points together.
Prior factor. We introduce the parameter π prior factor for mixing ratio X. The top row of Fig is the visualization of samples generated by SageMix with various prior factor æ. Given two samples and their corresponding query points colored in green, SageMix generates a sample with the salient local regions from both samples, egg, the head and right-wing of airplane and the back of table are preserved. We observe that the generated sample gets close to the airplane as - 1, and vice versa. short, SageMix controls the distribution of mixing ratio based on π.
Bandwidth. The bandwidth a of the RBF kernel controls the change of point-wise mixing ratios in SageMix. As mentioned in Section 4.2 when the bandwidth is sufficiently large (a 2). SageMix emulates PointMixup 14] SageMix tends to yield globally even weights and constant mixing ratios for all points rather than focusing local parts. In contrast, as the bandwidth gets smaller, SageMix tends to impose higher weights around the query point preserving the local structure more precisely. These well exemplified in the bottom row of Figure 3 For instance, when 0.25, notice the steep change in the boundary two samples while preserving the salient local structure. This allows generating augmented samples like RSMix. In short, our SageMix
exhibit similar behaviors as PointMixup and RSMix depending on the bandwidth σ. We also share the quantitative analysis the bandwidth with DGCNN and OBJ ONLY in Table [] We observed that SageMix with wide range of bandwidth (0.1 to 2.0) consistently outperforms previous Mixup methods (e.g., 86.9%, 9%, 86.6% for PointMixup, RSMix).
4.3
Extensions to part segmentation and 2D image classification.
Part segmentation. For part segmentation, train DGCNN on 31) Warehouse (SN-Parts) 20 18 In part segmentation, since a model predicts a label for each point p., we generate wisnt-wise ground truth, i.e., J. = A + (1 -x)) Also, for a comparison with previous methods, we used the official code by the authors of PointMixup and RSMix with minor modifications for generating

---

Table 6: Part segmentation performance on SN-Parts 18
Model
Base
PointMixup [14
RSMix [1
SageMix
PointNet++
85.1
85.5
85.4
85.7
DGCNN 6
85.1
85.3
85.2
85.4
Table 7: 2D image classification performance with PreActResNec118 25 on CIFAR-100.
Dataset
Vanilla Mixup Manifold CutMix SaliencyMix Puzzle Mix Co-Mixup SageMix
CIFAR-100)
76.41
77.57
78.36
78.71
79.06
79.38
80.13
80.16
point-wise ground truth. We follow the settings in 4 6 to evaluate our method and reports the results Table Note that although the gain seems small, SageMix outperforms previous Mixup methods. Also, considering the already saturated performance, we believe that the improvement (+0.6%, +0.3% PointNet++, DGCNN) over the base model is noteworthy.
2D classification. Our framework also applicable 2D image classification. Following 12] we used PreAActReNNe118 [25] in the CIFAR-101 for our experiments. We compare our method Although our method is designed for point clouds, it shows competitive performance. SageMix achieves the best accuracy of 80.16% with PreActResNet188 on the CIFAR-I00 dataset (Table 7. is worth noting that PuzzleMix and Co-Mixup require an additional optimization which introduces considerable computational overhead. Specifically, SageMix is x6.05 faster than Co-Mixup [2] per epoch in the CIFAR- 100 dataset. We believe that our simple sampling technique is helpful to improve the generalization power of the model, For more details, see Appendix B.4.
with several Mixup baselines in the image domain I I2
13 g g).
Related works
Deep learning on point clouds. PointNet 3 is a pioneering work that designs a novel deep neural
network for processing unordered 3D point sets with a multi-layer perceptron. Inspired by CNNs, Qi
et al. [4] propose PointNet++ with a hierarchical architecture. In DGCNN, Wang et al. [6] introduce
EdgeConv which utilizes edge features from the dynamically updated graph. Additionally, various
works have focused on point-wise multi-layer perceptron [26] [2 28), convolution [29 30 31 [ 2
5,32), and graph-based methods
34 to process point clouds. Parallel to these approaches, other
recent works [14 35
36 8 focus on data augmentation to improve the generalization power of
deep neural networks in point clouds.
Mixup. Mixup g is widely used regularization technique, which linearly interpolates pair of images to generate an augmented sample. Following this work, Verma et al. 7 propose Manifold Mixup that extends Mixup to the hidden representations. CutMix & replaces a part of an image with part of another one. More recent studies 11 13 have been proposed to preserve the saliency while mixing samples. point clouds, PointMixup [44 is the first approach that adapts the concept of Mixup in point clouds with the optimal assignments. Instead of linear interpolation, Lee et al. [T5] propose RSMix which merges the subsets two point clouds inspired by CutMix.
Saliency. Measuring the saliency of data using neural networks has been studied obtain more precise saliency map 37 38 39]. The saliency has been prevalent in various fields such as object
detection, segmentation, and speech recognition [TO 40
42 43 Similarly, PointCloud Saliency
Map [44 constructed the saliency map identify the critical points via building gradient-
saliency map. Recently, saliency has been used in Mixup framework generating samples only with background irrelevant regions to the target objects.
I 12
13
to prevent

---

6 Conclusion
We propose SageMix, a novel caliency-guided Mixup for point clouds to preserve salient local structures. Our method generates augmented sample with continuous boundary while preserving the discriminative regions. Additionally, with a simple saliency-guides sequential sampling, SageMix achieves Mfae-off-hhe-r performance in various metrics (e.g.. generalization, robustness, and uncer- tainty calibration). Moreover, we demonstrate that the proposed method is extensible to various tasks: part segmentation and standard 2D image classification. The visualization supports that SageMix generates a continuous mixture while respecting the salient local structure.
Acknowledgments and Disclosure of Funding
This work was partly supported by ICT Creative Consilience program supervised by the IITP; the National Supercomputing Center with supercomputing resources including technical support (KSC-2222CCR--028 )); and IIT grant funded by the Korea government (MSIT) NNo.2021-0-0233 Efficient Meta-learning Based Training Method and Multipurpose Multi-modal Artificial Neural Networks for Drone Al).
References
[1] Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Convolution x-transformed points. NeurIPS, 2018.
Pointenn:
[2] Yongcheng Liu, Bin Fan, Shiming Xiang, and Chunhong Pan. Relation-shape convolutional neural network for point cloud analysis. In CVPR, 2019.
[3] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, 2017.
[4] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. NeurIPS, 2017.
[5] Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In ICCV, 2019.
[6] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph enn for learning point clouds. TOG, 2019.
[7] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In ICML, 2019.
[8] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
[9] Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
[10] Chengyue Gong, Dilin Wang, Meng Li, Vikas Chandra, and Qiang Liu. Keepaugment: A simple cooamation--reservvinn data augmentation approach. In CVPR, 2021.
[11) Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statisties for optimal mixup. In ICML., 2020.
[12] Jang-Hyun Kim, Wonho Choo, Hosan Jeong, and Hyun Oh Song. Co-mixup: Saliency guided joint mixup with supermodular diversity. In ICLR, 2021.
[13] A. F. M. Shahab Uddin, Mst. Sirazam Monira, Wheemyung Shin, TaeChoong Chung, and Sung- Ho Bae. Saliencymix: A saliency guided data augmentation strategy for better regularization. In ICLR, 2021.
[14] Yunlu Chen, Vincent Tao Hu, Efstratios Gavves, Thomas Mensink, Pascal Mettes, Pengwan Yang, and Cees GM Snock. Pointmixup: Augmentation for point clouds. In ECCV,
10

---

[15] Dogyoon Lee, Jaeha Lee, Junhyeop Lee, Hyeongmin Lee, Minhyeok Lee, Sungmin Woo, and Sangyoun Lee. Regularization strategy for point cloud via rigidly mixed sample. In CVPR, 2021.
[16] Jinlai Zhang, Lyujie Chen, Bo Ouyang, Binbin Liu, Jihong Zhu, Yujin Chen, Yanmei Meng, and Danfeng Wu. Pointcutmix: Regularization strategy for point cloud classification. Neuro- computing, 2022.
[17] Olivier Chapelle, Jason Weston, Léon Bottou, and Vladimir Vapnik. Vicinal risk minimization. NeurIPS, 2000.
[18] Chenfeng Xu, Shijia Yang, Bohan Zhai, Bichen Wu, Xiangyu Yue, Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka. Image2point: 3d point-cloud understanding with pretrained 2d convnets. arXiv preprint ar arXiv:2/06.04180 2021.
[19] Mikaela Angelina Uy Quang-Hieu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In ICCV, 2019.
[20] SketchUp. 3D Warehouse, 2022. URL https:///ddarreooseeeeeee /cnetcchp.ccom/
[21]
Binh-Son Hua, Quang-Hieu Pham, Duc Thanh Nguyen, Minh- Khoi Tran, Lap-Fai Yu, and Sai-Kit Yeung. Scenenn: A scene meshes dataset with annotations. In 3DV, 2016.
[22] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017.
[23] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. ICML, 2017.
[24] Aviral Kumar, Sunita Sarawagi. and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In ICML, 2018.
[25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. ECCV, 2016.
[26] Mor Joseph-Rive Rivlin, Alon Zvirin, and Ron Kimmel. Momen (e) t: Flavor the moments in learning to classify shapes. ICCV, 2019.
[27] Xu Yan, Chaoda Zheng, Zhen Li, Sheng Wang, and Shuguang Cui. Pointasnl:
Robust point
clouds processing using nonlocal neural networks with adaptive sampling. In CVPR, 2020.
[28] Hengshuang Zhao, Li Jiang, Chi-Wing Fu, and Jiaya Jia. Pointweb: Enhancing local neighbor- hood features for point cloud processing. In CVPR, 2019.
[29] Binh-Son Hua, Minh-Khoi Tran, and Sai-Kit Yeung. Pointwise convolutional neural networks. In CVPR, 2018.
[30] Shiyi Lan, Ruichi Yu, Gang Yu, and Larry S Davis. Modeling local geometric structure of 3d point clouds using geo-cnn. CVPR, 2019.
[31] Huan Lei, Naveed Akhtar, and Ajmal Mian. Octree guided enn with spherical kernels for 3d point clouds. In CVPR, 2019.
[32] Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks 3d point clouds. In CVPR, 2019.
[33] Yiru Shen, Chen Feng, Yaoqing Yang, and Dong Tian. Mining point cloud local structures by kernel correlation and graph pooling. In CVPR, 2018.
[34] Yingxue Zhang and Michael Rabbat. graph-enn for 3d point cloud classification. In ICASSP, 2018.
[35] Sihyeon Kim, Sanghyeok Lee, Dasol Hwang, Jaewon Lee, Seong Jae Hwang, and Hyunwoo J Kim. Point cloud augmentation with weighted local transformations. In ICCV, 2021.
[36] Ruihui Li, Xianzhi Li, Pheng-Ann Heng, and Chi- Chi-Wing Fu. Pointaugment: an auto- augmentation framework for point cloud classification. In CVPR, 2020.
[37] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV, 2017.

---

[38] Lijun Wang, Huchuan Lu, Xiang Ruan, and Ming-Hsuan Yang. Deep networks for salieney detection via local estimation and global search. In CVPR, 2015.
[39] Rui Zhao, Wanli Ouyang, Hongsheng Li, and Xiaogang Wang. Saliency detection by multi- context deep learning. In CVPR, 2015.
[40] Chanho Jung and Changick Kim. unified spectral-dome domain approach for saliency detection and its application to automatic object segmentation. TIP, 2011.
[41] Ozlem Kalinli and Shrikanth S Narayanan. A saliency- based auditory attention model with applications to unsupervised prominent syllable detection in speech. In INTERSPEECH, 2007.
[42] Zhixiang Ren, Shenghua Gao, Liang-Tien Chia, and Ivor Wai-Hung Tsang. Region-based saliency detection and its application in object recognition. TCSVT, 2013.
[43] Yunchao Wei, Jiashi Feng, Xiaodan Liang, Ming-Ming Cheng, Yao Zhao, and Shuicheng Yan. Object region mining with adversarial erasing: A simple classification to semantic segmentation approach. CVPR, 2017.
[44] Tianhang Zheng, Changyou Chen, Junsong Yuan, Bo Li, and Kui Ren. Pointcloud saliency maps. In /CCV, 2019.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See abstract and Section !
(b) Did you describe the limitations of your work? [Yes] See supplement.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See supplement.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] See supplement.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See supplement.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See supplement.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See supplement.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See supplement.
(b) Did you mention the license of the assets? [Yes] See supplement.
(c) Did you include any assets either in the supplemental material URL? [Yes]
See supplement.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] We the publicly available benchmark datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes] We use the publicly available benchmark datasets.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "t8jOQt-9ges", "reviewer": "Reviewer_VLSt", "summary": "The paper proposes SageMix, a data augmentation technique for point clouds. Similar to the Mixup family of data augmentations, SageMix mixes two point clouds. It tries to mix point clouds in a saliency-guided way to preserve the salient local structures. Experiments have been conducted to show the efficacy of the method.", "questions": "Refer to the weakness section for questions. Overall, I am ambivalent about the paper. The method could be useful but I am unable to conclude from the experiments because of reasons mentioned in the weakness sections. Hence, I suggest a borderline rating for now. I will update the score based on the rebuttal.", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper is well-written and easy to follow.\n- It is nice that this data augmentation can lead to more robust networks as shown in Table 3. \n\nWeakness:\n\n- A critical weakness is that experimental findings do not appear to be very significant. The performance difference between various techniques is not very large and no error margin has been reported (Table 2). It would be nice if the paper could provide some mean +- std measures. This could be done by running the same experiments multiple times (with random initialization) and reporting the mean and variance. This is particularly important as point-based benchmark methods can have significant variations across runs.\n\n- Most experiments in the paper are limited to point cloud classification. The experiment on part-segmentation has not been described in detail. It would be nice to see experiments similar to Table 2 for segmentation as well. This would help in showing that the technique can be used beyond classification.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "t0G2R-HiXqL", "reviewer": "Reviewer_yU1N", "summary": "This paper proposes a novel saliency guided mixup method for point clouds. It first utilizes saliency to find a query point for each of the two point clouds. Then it uses an RBF kernel (around the query point) to calculate the blending weights for each of the points. This method generalizes PointMixup [2], and shows superior performance against existing ones.", "questions": "- About the robustness experiments in Table3. Why don’t the authors present all tests presented in Table & in RSMix [15] ? e.g. Rotation 90, Rotation Y, Scale 1.4 are missing.\n- The authors mentioned their varying performance with different $\\sigma$ in Appendix D.2. However, they didn’t quantitatively experiment the performance except for a qualitative one in Figure 3. How does $\\sigma$ affect the performance of SageMix?\n- Another minor issue is missing reference of a paper on arxiv (non peer reviewed, but with decent citations). PointCutMix: Regularization Strategy for Point Cloud Classification. In ArXiv. https://arxiv.org/abs/2101.01461.", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The idea of introducing saliency guidance to the mixup method in point clouds sounds reasonable, as it has been proven to be useful in image-based methods [11] [12] [26].\n- The design of sequentially sample two remotely located query (salient) points is delicate. It avoids the overlapping problem while preserving the important local structure of the two point clouds.\n- The experiments seem sufficient.\n\nWeaknesses\n- I didn’t see any major weaknesses in this paper. One potential improvement to the method could be rotating and translating the whole point clouds to move the two query points far away from each other. In my opinion, it’s a more elegant way. Nevertheless, I think it’s fine to leave it as a future work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7-5xyNQO2Ie", "reviewer": "Reviewer_YgrL", "summary": "This paper presents a method for the data augmentation of 3D point clouds. The proposed Saliency-Guided Mixup for point clouds (SageMix) preserves discriminative local structures and generates continuous samples with smoothly varying mixing ratios. Here saliency is computed based on the impact to the corresponding task, measured through the gradients of the loss. Experimental results show that SageMix brings consistent and significant improvements over state-of-the-art Mixup methods.", "questions": "1. Why 3D mixup is more challenging than 2D mixup? What is the particular difficulty of extending mixup from 2D to 3D?\n2. The computation of saliency is similar to the popular CAM approach. What is the impact of using different methods for saliency compuation?\n3. I suggest the authors make a more solid evaluation for the proposed method. ", "limitations": "n/a", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The saliency-guided sequential sampling is technically novel.\n2. There are some ablation studies to demonstrate the effect of the proposed method.\n3. Overall, the paper is well organised.\n\nWeaknesses:\n1. The difference between 2D and 3D mixup-based methods is not insightfully analysed in the introduction.\n2. The shape-preserving continuous mixup component just follows the mainstream method and thus the novelty is limited.\n3. The experimental results is not extensive. The proposed method is only quantitatively compared with PointMixup and RSMix. And there is also a lack of qualitative results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept