SUG: Single-dataset Unified Generalization for 3D Point Cloud Classification
Authors: Siyuan Huang
Bo Zhang
Botian Shi
Peng Gao
Tao Chen
Hongsheng Li
Yikang LI
Venue: Submitted to ICLR 2023 (2023)
Keywords: 3D Point Cloud Classification
Domain Adaptation

Abstract: In recent years, research on zero-shot domain adaptation, namely Domain Generalization (DG), which aims to adapt a well-trained source domain model to unseen target domains without accessing any target sample, has been fast-growing in the 2D image tasks such as classification and object detection. However, its exploration on 3D point cloud data is still insufficient and challenged by more complex and uncertain cross-domain variances with irregular point data structures and uneven inter-class modality distribution. In this paper, different from previous 2D DG works, we focus on the 3D DG problem, and propose a Single-dataset Unified Generalization (SUG) framework that only leverages the source domain data to alleviate the unforeseen domain differences faced by the well-pretrained source model. Specifically, we first design a Multi-grained Sub-domain Alignment (MSA) method that can constrain the learned representations to be domain-agnostic and discriminative, by performing a multi-grained feature alignment process between the splitted sub-domains from the single source dataset. Then, a Sample-level Domain-aware Attention (SDA) strategy is presented, which can selectively enhance easy-to-adapt samples from different sub-domains according to the sample-level inter-domain distance, to avoid the negative transfer. Extensive experiments are conducted on three common 3D point cloud benchmarks. The experimental results demonstrate that SUG framework is effective to boost the model generalization ability for unseen target domains, even outperforming the existing unsupervised domain adaptation methods that have to access extensive target domain data, where we significantly improve classification accuracy by 7.7% on ModelNet-to-ScanNet setting and 2.3% on ShapeNet-to-ScanNet setting. Our code will be available.

Under review as a conference paper at ICLR 2023
SUG: SINGLE-DATASET UNIFIED GENERALIZATION FOR 3D POINT CLOUD CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
In recent years, research on zero-shot domain adaptation, namely Domain Gen- eralization (DG), which aims to adapt a well-trained source domain model to un- seen target domains without accessing any target sample, has been fast-growing in the 2D image tasks such as classification and object detection. However, its exploration 3D point cloud data is still insufficient and challenged by more complex and uncertain cross- variances with irregular point data structures and uneven inter-class modality distribution. In this paper, different from previ- ous 2D DG works, we focus on the 3D DG problem, and propose a Single-dataset Unified Generalization (SUG) framework that only leverages the souree domain data to alleviate the unforeseen domain differences faced by the well-pretrained source model. Specifically, we first design Multi- Sub-domain Align- ment (MSA) method that can constrain the learned representations to be domain- agnostic and discriminative, by performing multi-grained feature alignment pro- cess between the splitted sub-domains from the single source dataset. Then, Sample-level Domain-awe Attention (SDA) strategy is presented, which can selectively enhance easy-to-adapt samples from different sub-domains according to the sample-level inter-domain distance, to avoid the negative transfer. Exten- sive experiments are conducted three common 3D point cloud benchmarks. The experimental results demonstrate that SUG framework is effective to boost the model generalization ability for unseen target domains, even outperforming the existing unsupervised domain adaptation methods that have to access exten- sive target domain data, where we significantly improve classification accuracy by 7.7% on SodelNet-to-Scan.. setting and 2.3% on ShapeNet-to- to-canNNe setting. Our code will be available.
INTRODUCTION
As commonly-used data format describing the real world, point clouds-based representations pre- serve more geometrie information residing in 3D scenes, and have become one of the most important data types for 3D scene perception and real applications such as robotics (Rusu et al., 2008; Rusu & Cousins, 2011), 1), autonomous driving (Sun al., 2020; Shi et al., 2020), and augmented and virtual reality (Tredinnick et al., 2016), giving a better understanding of the surrounding environment for machines. In recent years, point clouds-based vision tasks (Shi al., 2020) have achieved great progress on the public benchmarks (Vishwanath et al., 2009; Chang et al., 2015; Dai et al., 2017), which largely owes to the fact that the collected point clouds are carefully annotated, sufficiently large, and low level noised. But in the real world, acquiring such data from a new target domain and manually labeling these extensive 3D data are highly dependent on professionals in this filed, which makes the data acquisition and annotation more difficult, labor-intensive, and time-consuming.
One effective solution to transfer the model from fully-labeled source domain to a new domain with- out extra human labor is Unsupervised Domain Adaptation (UDA) (Shen et al., 2022; Zou et al., 2021; Fan et al., 2022; Yang et al., 2021), whose purpose is to learn a more generalizable represen- tation between the labeled source domain and unlabeled target domain, such that the model can be adapted to the data distribution of the target domain. For example, when point cloud data distribu- tion from the target domain undergoes serious geometric variances (Shen et al., 2022), performing a correct source-to to-teeetttttet correspondence can boost the model's adaptability. Besides, GAST (Zou et al., 2021) learns domain-shared representation for different semantic categories, while vot-

---

Under review as a conference paper at ICLR 2023
ing reweighting method is designed (Fan et al., 2022) that can assign reliable target domain pseudo labels. However, these techniques are highly dependent on the accessibility of the target domain data, which is a strong assumption and prerequisite for the models running in an unprecedented cir- cumstance, such as autonomous driving system and medical scenarios. Thus, it is meaningful and important to investigate the model's cross-domain generalization ability under the zero-shot target domain constraint, which derivates the task of Domain Generalization (DG) for 3D scenario.
However, achieving such zero-shot domain adaptation, i.e., DG, is more challenging in 3D scenario mainly due to the following reasons. (1) Unknown vorainevariance Challenge: 3D point cloud data collected from different sensors or geospatial regions with different data distributions often present serious domain discrepancies. Due to the inaccessibility of the target domain data (or sen- sor), modeling of source-to-target domain variance intangible. (2) Uneven Domain Adaptation Challenge: Considering that our goal is to learn a transferable representation that can be generalized to multiple target domains, robust model needs to perform an even domain adaptation, rather than lean to fit the data distribution one of the multiple target domains. But for 3D point cloud data with more complex sample-level modality variances, how to ensure an even model adaptation under the zero-shot target domains setting still remains challenging.
To tackle the above challenges, we study the typical DG problem in 3D scenario, and introduce a Singe-dataset Unified Generalization (SUG) framework for addressing the 3D point cloud general- ization problem. We study a one-to-many domain generalization problem, where the model can be trained on only single 3D dataset, and is required be simultaneously generalized multiple target datasets, Different from previous DG works in 2D scenarios (Shankar et al., 2018; Piratla al., 2020; Chen al., 2021), 3D point cloud data have more irregular data structure and di- verse data distribution within a single dataset, which provides the possibility to exploit the modality and sub-domain changes without accessing any target-domain datasets. To be specific, our SUG framework consists of a Multi-graine grained Sub-domain Alignment (MSA) method and a Sample-level Domain-aware Attention (SDA) strategy. To address the unknown domain-variance variance challenge, the MSA method first splits the selected single dataset into different sub-domains. And then, based on the splitted different sub-domains from a single dataset, the bascline model is constrained to simulate as many domain variances as possible from multi-grained features, so that the bascline model can learn multi-gran muutti-raiiedd and multi-domains agnostic representations. To solve the uneven do- main adaptation challenge, the SDA is developed, which assumes that the instances from different sub-domains often present different adaptation difficulties. Thus, we add sample-leve constraints to the whole sub-domain alignment process according the dynamically changing sample-level inter-domain distance, leading to an even inter-domain adaptation process.
We conduct extensive experiments on several common benchmarks (Qin al., 2019) under the single-dataset DG scenario, which includes three sub-datasets and our experiments cover the follow- 1e---SSapeeeett///SeeSe
meaning that the model is only trained on ModelNet- and directly evaluated on both ShapeNet- 10 and ScanNet- 2) ShapeNet- Experimental re- sults demonstrate the effectiveness of SUG framework in learning generalizable features of 3D point clouds, and it can also significantly boost the DG ability for many selected baseline models.
ing three scenarios: 1)
10 ---Moodeleeet-0SSaaeeetaaat
3)
The main contributions of this paper can be summarized as follows:
1) From a new perspective of one-to-many 3D DG, we explore the possibilities of adapting a model from its original source domain to many unseen domains, and study how to leverage the feature's multi-modal information residing in single dataset.
2) We propose a SUG to tackle the one-to-many 3D DG problem. The SUG consists of a designed MSA method to learn the donain-agnostic and discriminative features during the source-domain training phase, and a SDA strategy to calculate the sample-level inter-domain distance and bal- ance the adaptation degree of different sub-domains with different inter-domain distances.
2 RELATED WORKS
2. 2D IMAGE-BASED DOMAIN ADAPTATION AND GENERALIZATION
Recent Domain Adaptation (DA) works can be roughly categorized into two types: 1) Adversarial learning-based methods (Ganin & Lempitsky, 2014; Tzeng et al., 2017; Long et al., 2018b; Kang
2

---

Under review as a conference paper at ICLR 2023
et al., 2020) that focus leveraging domain label discriminator to reduce the inter-domain dis- crepancy; 2) Moment matching- based methods (Long et al., 2018a; 2015; Sun & Saenko, 2016) that refer to aligning the first-order or second-order moments of feature distribution. But under the Domain Generalization (DG) setting where the data distribution of the target domain is unavailable, the above DA methods cannot be directly applied to address the DG problem.
For this reason, some researchers (Shankar et al., 2018; Piratla et al., 2020; Chen et al., 2021) start to explore how to adapt the pre-trained model from its source domain to out-of-distribur only using source data. For example, some works (Zhu al., 2022: Zhang et al., 2017) try to boost the model generalization ability using mix-up domains, which generates novel data distribution from the mixtures of multi-domains. Besides, self-supervised learning (SSL.) (Wang et al., 2020; Carlucci et al., 2019) also is applied to DG problem to enhance transferable features by leveraging the designed pretext tasks. Although these DG methods have been extensively studied in 2D image tasks, the research DG problem in 3D point cloud scenarios still remains under- explorexp,ored, which motivates us to investigate the zero-shot generalization ability of 3D point cloud models.
domain
2.2 3D POINT CLOUD CLASSIFICATION
The existing 3D point cloud classification methods can be divided into: 1) Projection- and 2) Point-based methods. The projection-bade based methods first covert irregular points into structured representations, such as multi-view images (Su et al., 2015; Yu et al., 2018), voxels (Riegler et al., 2017), and spherical (Rao et al., 2019). And then, a 2D or 3D neural network is utilized to extract dense features of the structured representations. In contrast, point-based methods (Qi et al., 201 7a;b; Wang et al., 2019) directly learn features from the irregular point clouds. This kind of methods can effectively explore the point-wisc relations using the designed network such PointNet (Qi et al., 2017a), which is the first work that directly takes original point clouds as the input and achieves permutation invariance with symmetric module. Further, considering that point clouds have variable density at different areas, PointNet++ (Qi et al., 2017b) learns 3D features from multiple semantic levels according the set abstraction. However, these data-driven point cloud models still face substantial recognition accuracy drop when they are deployed to an unknown domain or dataset.
2.3 DOMAIN ADAPTATION FOR 3D POINT CLOUD CLASSIFICATION
To investigate how equip 3D point cloud model with good domain generalization capability, we have reviewed existing domain adaptation- based (Qin et al., 2019; Luo et al., 2021; Shen et al., 2022: Achituve et al., 2021; Yang et al., 2021) transfer learning- -basdd 3D point cloud works (Ye et al., 2022), and find that they mainly focus on DA study and fail to generalize to unseen target domains. These DA-based works can be mainly categorized into two classes: 1) Self-supervised adaptation and 2) Domain-level feature alignment.
For self-supervised adaptation methods (Luo et al., 2021; Shen et al., 2022; Achituve et al., 2021; Yang et al., 2021), they try to design a pretext task to address the common geometric deforma- tions caused by the variances in scanning point clouds. For example, by deforming region shape of points and reconstructing the original regions of the shape, DefRee (Achituve et al., 2021) can achieve good domain adaptation result under different domain shift scenarios. Recently, geometry-aware DA method (Shen et al., 2022) is proposed, which employs the underlying ge- ometric information from points. The domain-level feature alignment method (Qin et al., 2019) focuses on designing an adversarial network to model the discriminative local structures for align- ment cross-domain features. Specifically, PointDAN (Qin et al., 2019) proposes Self-Ade self-Adaptive (SA) node learning with a node-level attention to present geometric shape information for points.
Overall, when performing the cross cross-domain adaptation, the above works need to collect extensive target samples in advance for supporting the adaptation process, which is infeasible for many real applications where the target domain is inaccessible or even unknown before deploying the pre- trained model. Thus, it is indispensable that data-driven models trained on a single dataset can handle domain shifts to a certain extent.
3 THE PROPOSED METHOD
The overall SUG framework is illustrated in Fig. 1. For easy understanding, we first give the problem definition of Domain Generalization (DG) for 3D point cloud classification, and then introduce SUG framework including aware Attention (SDA) modules. Finally, the overall loss function and DG strategy are described.
Sub-domain Alignment (MSA) and Sample-level Domain-
3

---

Under review as a conference paper at ICLR 2023
Faby Labeke Sumple
F
I) Sence Pra-draliing
Cleatio C fhrentobe
Ca
The Thes albch-deesk Tioe
2) MSA Mithed
3) 5DA Stratigs
Sease Yeak
Cazeh
Zuom
LyDD
Dkaribation
Figure 1: SUG framework, consisting of MSA and SDA to tackle the one-to-many DG.
3.1 PRELIMINARIES
Problem Definition. Suppose that a domain is defined by a joint distribution Pxy on X x V where X' and y' stand for the input image and label space, respectively. In the scope of DG, K source domains S = { = { (x(*), (y())) distinct domain associated with one distribution P And the goal of DG to obtain model f + V, trained on the source domain(s), which would have minimized prediction errors on the unseen target domain(s).
are available for the training process, where each
Point cloud data is a set of unordered 3D points x {p: i= I...., normally represented by its 3D coordinate (z, v, and is the number of sampling points of one 3D object. We use (x, y) to denote one training sample pair, and y is its label.
n). where each point p; is
Sangle-dataset DG. In the 3D point-based single-dataset DG setting, the training model can only access one labeled dataset S, and is required to be evaluated on M unseen target datasets T (usually > 1). The corresponding joint distribution could be described with T AM4 mp=1 Also, PXY * p(R) xXY, vk E {l...., K}. Ym E {I....,My In
{T.m = { (x(m)) } our problem set, V/s and D/ share the same label space. The goal of 3D DG is to improve the per- formance of source-trained model f on the unseen target domain(s) with the following objectives:
min E (7 e(J((x. v),
where e is the cross-entropy error in our classification task, which can be further defined as:
Eyi- log p(ū c [x]]
(2)
with the prediction that can be obtained with:
p = c z) = softmax (Co (F)()))),
where x is the input point cloud instance, ý is the predicted label. The F is the embedding network parameterized by d, and C is the classifier parameterized by 6.
3.2 SUG: A SINGLE-DATASET UNIFIED GENERALIZATION FRAMEWORK
To overcome the two challenges discussed in Sec. I, we introduce SUG framework consisting of two novel plug-and-play modules, e.g. Multi-grained Sub-domain Alignment (MSA) and Sample-level Domain-aware Attention (SDA), which can be inserted into existing 3D backbones to learn more domain- aagnosicc representations, to be elaborated in Sec. 3.2.1 and 3.2.2, respectively.
First, the single source dataset is fed into designed split module to get multiple sub-domain of the original
based on pre-defined heuristics. Then, the embedding network F takes all the splitted sub-domains the network input, and converts the point cloud instance into multi- level feature vectors fi Fx.((x) and fn Fo, (ft). where f E IR xx and fi and fn denote the learned low-level and high-level representations. To handle feature discrepancies from different sub- domains, the MSA module is applied to align the multi-grained features, both at low- and high-level, which can constrain the network to focus on the -agastic representations. Meanwhile, the SDA module is used to selectively enhance the alignment constraints rising from the easy-to-transfer samples to ensure an even adaptation across different sub-domains.
4

---

Under review as a conference paper at ICLR 2023
3.2.1
MULTI-GRAINED SUB-DOMAIN ALIGNMENT (MSA)
Class distribution alignment. The 3D point clouds have been deployed in plenty of application scenarios where the ob- jects' distribution shifts significantly, re- sulting in different distribution patterns re- siding in different objects, as shown in Fig. 2. To handle such cross-dataset class-imbalance issue, we incorporate the class-wise sample weighting w with the original classification loss (refer Eq. 2), and the complete weighted classification loss can be written as follow:
Bed Casoee
Montor 5ots
910
Vocellel
Shappleet
Scan
Figure 2: Class distribution shifting across datasets in PointDAN.
Lets (B) -
- - Ea(y)L(0xx) (3)
where B denotes a data batch. The weighting vector a could be set following different heuristics, like FocalLoss (Lin et al., 2017) and DLSA (Xu al., 2022), etc. Here, we follow the definition in DLSA (Xu et al., 2022), where samples are weighted by:
m
(4)
Σ, my
where m, is the number of training samples of the class and is positive number controlling the weights distribution. The optimization objective of previous methods such as FocalLoss (Lin et al., 2017) and DLSA (Xu et al., 2022), is to tackle the class imbalance problem within single dataset, while the optimization function of our method is to tackle the cross-dataset class-wise imbalance issue, which is illustrated in Fig. 2. Note that different 3D datasets present an inconsistent class- distribution, which motivates us to use Eq. 4 to Iearn a uniform and even class-distribution by re- weighting class-distribution for each dataset. Such way is beneficial to learn more generalizable representations that can avoid to overfit to the class-distribution of the soucre dataset.
Geometric shifting alignment. Due the objects' geometric variances in different scenarios and inconsistent data acquisition procedures, the objects from the same class across different datasets present diverse geometric appearances, as illustrated in Fig. 3(a) across different rows. Meanwhile, the objects' geometric appearance varies greatly with a certain class or a single dataset, which offers the potential that we could use the geometric variances within single dataset to effectively simulate the ones between different datasets.
To be more specific, we take the low-level feature vector fi from the shallow layer the embedding module F, and minimize the Maximum Mean Discrepancy (MM))Borgwardd et al., 2006) (Long et al., 2013) loss to align the geometric features from different sub-domains as follows:
LMMDCco 7lg78
k (fiis
+
Σ K f(f
ThT Σ R fii
(5)
ne i,=1
where & is the kernel function, and its superscript and denote two different sub-domains sam- pled from a single dataset. We use the Radial Basis Function (RBF) kernel in our SUG, which is consistent with previous work Qin et al. (2019).
Semantic variance alignment. After the high-level features from F are obtained, the semantic variance alignment is applied to minimize the semantic-level discrepancy between features across different sub-domains before feeding into the classifier. The intuition of the semantic alignment rises from the observation that there exist samples from different classes that could have the similar geometric appearances. As illustrated in Fig. 3(b). the class Table and Cabinet resemble some samples in Chair class as they are all four-legged items. And by conducting semantic variance alignment, the model will learn less single-domain geometric bias yet discriminative representations. The semantic alignment constraints LMM Dsom can be easily calculated by employing the fhj and fr as the input in Eq. 5.
5

---

Under review as a conference paper at ICLR 2023
Similar Didaeit Modality
Diffeent Chur Cru
the to thar
ModelNet
Same Class
SeapeNet
Difffrrnt Class Class
ScanNet
(
(b)
Illustration of distinct characteristics of data in 3D datasets. (a) Geometric and semantic- level domain variances within and between datasets. (b) Geometric similarity comparisons within and between classes.
Figure 3:
3.2.2 SAMPLE-LEVEL DOMAIN-AWARF ATTENTION (SDA)
The aforementioned MSA module guides the model learn more domain-agnostr representations. However, the features inside one mini-batch from different sub-domains do not contribute equally to the sub-domain alignment process, since they could contain distinct feature distributions. Ignoring such diversity and imposing equal importance for different samples would result in the hard-to- transfer samples deteriorating the generalization procedure. Meanwhile, the designed domain split module in SUG framework inevitably introduces randomness to different sub- domains with differ- ent domain variances, which could also hurt the model generalization performance. Towards safer transfer, we propose the SDA module to enhance the alignment constraints from easy-to-transfer samples. To be more specific, we add sample-level weights w to the alignment constraints, inverse proportional to the domain distance d, expressed as:
1
Marigirriered w LMMD
LMMD.
(6)
d
where d could be realized by using either Eq. 7 or Eq. 8. As for the geometric shifting alignment, we use the 3D reconstruction metric as the distance function. In our implementation, Chamfer Distance (CD) is used, which can be formulated follows:
dx Σ min yEY
min || v112 + Σ
v112.
ZE
VEY
where X and Y are two point cloud instances. The geometric weights deD focus on the geometric consistency explicit, as shown the first column of Fig. 3(a), where the samples with geometric similarity have relative small CD distance even they could come from different classes. While for the samples with distinct geometric appearances, the CD distance is higher and the corresponding MMD constraints would be relaxed.
For the semantic variance alignment, we adopt the Jensen-Shannow (JS) divergence as our metric. And for symmetric usage, the JS -isiistance dss X, Y) is defined
1 2
(Y where (xl(X is the discrete format of KL divergence, represented as:
d Y) D D
(8)
((c)
DKL(X Excc) log
Y(c)
where X(c) and (c) are the probability of predicting sample belonging to the class In contrast to the geometric weighting, dxr. emphasizes more semantic consistency and tends on conducting the alignments among samples belonging to the same class.
3.3 OVERALL OBJECTIVES AND DOMAIN GENERALIZATION STRATEGY
Overall Objectives. With the MMD constraints introduced in Sec. 3.2.1 and the corresponding weights stated in Sec. 3.2.2, the complete MMD loss could be defined as
LMMD WGeo LMMDGes + Wsem LLM
(10)
The overall training loss consists of the classification loss as described in Eq. 3 and the MMD loss in Eq. 10, which can be written follows:

---

Under review as a conference paper at ICLR 2023
= Lals + X LMDD:
(11)
where is the weighting factor to balance the classification task and the alignment process. Domain Generalization Strategy. We train our model in an end-to-end manner, and the training procedure consists of two steps follows.
Step I: Firstly, the model is trained using classification loss as defined in Eq. 3, which can ensure that the model learns discriminative features for the subsequent domain transfer. Step 2: Secondly, to learn robust representation that can be generalized to different target datasets, we train the baseline model with the complete loss L as defined in Eq. 11, aiming to constraint the learned representations to be Nomain--ggnosee and discriminative.
4 EXPERIMENTS
4.1 DATASETS AND IMPLEMENTATION DETAILS
Datasets. In order to conduct the experimental evaluation for domain adaptation setting, Point- DAN (Qin et al., 2019) extracts point cloud samples of 10 shared classes from ModelNet40 (Vish- wanath et al., 2009), ShapeNet (Chang et al., 2015), and ScanNet (Dai et al., 2017). We follow the work (Qin et al., 2019) and select the same datasets to verify the effectiveness of the proposed method. ModelNet-10 (M) contains a total of 4183 training samples and 856 test samples of 10 classes, which are collected using 3D CAD model. ShapeNet-10 (S) has 17378 frames for train- ing and 2492 frames for testing, and these frames are produced using a 3D CAD model. ScanNet-10 (S*) includes total of 7879 samples that are re-scanned from real-world indoor scene.
Implementation Details. For our SUG framework, we employ the PointNet (Qi et al., 2017a) and DGCNN (Wang et al., 2019) as the feature embedding network while the classifier Co is constructed with a Multi-Layer Perceptron (MLP) using a three-layer fully-connected network. The sample weighting control q in Eq. and the hyper-parameeers of X are set be 0.2 and 0.5, respectively, and the comparison results are shown in Appendix. During the training phase, we use the common naive data augmentations as described in the work (Qin al., 2019). The Adam optimizer (Kingma & Ba, 2014) is utilized using an initial learning rate of 0.001 and 0.0001, weight decay of 0.00005 and 0.0001 for DGCNN and PointNet backbones, respectively. When testing the generalization performance, to make a fair comparison with the works Zou et al. (2021); Shen et al. (2022), we align each object along x and y axes for the DGCNN backbone and no alignment procedure is applied for experiments on PointNet backbone. During the DG adaptation process, we mainly judge whether the model adaptation state reaches optimal by the designed cross-domain MMD loss. When the change of the MMD loss tends to be stable and has less fluctuation, the adaptation process ends. For all our experiments, we report mean value over the three runs, And we use FFss third layer and Co's 's second layer as the low- and high-level features, respectively.
4.2 HOW TO SPLIT: DOMAIN SPLIT MODULE DESCRIPTIONS
In this section, we describe the tnooldenoweede based domain split modules and employ the corre- sponding modules to conduct experiments, as shown in Table 1. Note that our splitting procedure is conducted via a class-wise manner within a source dataset to ensure that each sub-domain contains all categories of the dataset. Please refer to Appendix for more discussions about the hand-designed domain split modules.
Random Splitting. We conduct the random sampling and split a single source dataset into different sub-domains with the same sample size, where domain characteristic of each sub-domain is identical with that of the original one.
Geometric Splitting. In our practice, we randomly select one sample as the anchor sample of a cer- tain class, then compute the Iterative Closest Point (ICP) registration score between other samples of the current class and the selected anchor sample. After getting all registration scores of all samples, the current class is clustered into K sub-domains according to the calculated score.
Entropy Splitting. We quantify the uncertainty of classifier's predictions with the entropy criterion (g) -
c= X. log Xe, as defined in Eq. 9. Note that the classifier used here is pre-trained model on the source domain. For the domain split module, the single dataset is clustered into K sub-domains based on the entropy scores of all samples.
Feature Clustering Splitting. We infer the whole dataset with the pre-trained model on source dataset, and save the feature maps before feeding them to the classifier. After that, we use Principal
7

---

Under review as a conference paper at ICLR 2023
Table 1: Results on different domain split methods under the one-to-many Domain Generalization (DG) setting. Avg denotes the mean adaptation accuracy across all target domains. The results of Random Splitting are averaged over three runs, and we report mean value over the three runs.
Domain Splic Method
Setting
Rac Rackehome
Ad
31 Source Domain
S as Source Donain
M S
Avg.
S" 3/
S" 0s Source Domain
Arg.
Aclapt
Souree-hily
PointNer
42.5
22.3
39.9
31.7
34,2
46.9
40.6
PoinDANN (NoutPS199)
UDA
PointNer
64.2
33.0
47.6
401.8
56.6
Rundom Splitting (Sec. 4.3)
36.3
37.8
49.0
Entropy Splitting (Se.. 4.2)
Geometric splliting
PointNer
57.4
55,4
41.7
42.5
49.0
36.5
30.3
27.7
31.0
32.1
41.7
49.9
41.3
45.8
Peature Clustering Splitting (Sec. 4.2)
00.4
36.1
48.3
45.4
31.7
38.6
37.5
45.0
41.6
Kandom Splitting tSer. 4.21
80.8
53.2
67.0
09.4
49.5
29.5
01.4
57.6
59.5
Entropy Splitting (Sec. 4.2)
Geometric Splitting (Sec. 4.2)
DG
DGCNN
73.6
79.3
19.3
49.9
64.6
61.5
72.8
56.7
50.3
53.2
61.6
55.0
42.9
40.5
60.9
64.4
51.9
52.5
Feature Clastering Splitting (See. 4.2)
77.8
52.9
65.4
71.0
47.6
59.3
63.0
59.3
61.2
Table 2: Results on PointDA-10 under the one-to-many Domain Generalization (DG) setting. Note that our SUG can be sinultaneously generalized to multiple target domains without accessing any target samples. In contrast, UDA methods can only be adapted to a single target domain. For example, GAST model performs an adaptation from the domain M to another domain S, but the adapted model cannot perform well in a new domain S
Method
Setting
Backhone
M- S
M as Source Domain
S*
Avg.
5 Mf
S as Source Domain
Avg.
S*
S"
as Source Domain
S"
Avg.
w/o Adapt
Source- Only
PointNet DGCNN
22.3
32.4
39.9
23.5
31.7
34.2
46.9
40.6
83.3
43.8
63.6
75.5
42.5
59.0
63.8
64.2
64.0
PoinIDAN (NeuriPS'V 19)
UDA
PointNet
64.2
33.0
48.6
47.6
33.9
40.8
49.1
64.1
56.6
GAST (ICCV'21)
UDA UDA
DGGNN
8488
59.8
72.3
80.8
56.7
68.8
81.1
74.9
78.0
SLT (CVPR 22)
DGCNN
86.2
58.6
72.4
81.4
56.9
69.2
81.5
74.4
71.9
SUG
IX
DGCNN
PointNet
61.3
40.7
52.5
44.0
74.8
36.2
10.1
41.5
51.7
49.6
Component Analysis (PCA) with t-SNE (Van der Maaten & Hinton, 2008) get the dimensional reduced representations and apply K K-means to get K sub-domain clusters.
4.3 How TO ALIGN: DOMAIN- AGNOSTIC FEATURE LEARNING
DG Baseline Implementation. In this part, we study how to use the off-the-shelf UDA technique to achieve unseen domain generalization. First, we use the domain split module to generate different sub-domain data. Then, when source dataset is clustered into K sub-domains, 3D UDA methods such as PointDAN (Qin ct al., 2019) can be used to perform a sub-domain adaptation within a single dataset. In our baseline practice, we directly use the implementation from PointDAN (Qin et al., 2019) without any further modification, in order to align the feature gaps between different splitted sub-domains. can be seen from Table that, by leveraging the above domain split modules to split a single dataset into different sub-domains, the off-the-shelf UDA method (Qin et al., 2019) can simultaneously boost the model generalization ability for multiple unseen datasets. It also can be concluded that, multi-modal distribution exists within a single-source dataset. As result, a hand-designed domain split method coupled with the uff-the-shelf UDA baseline can significantly boost unseen domain generalization. Besides, we also observe that the classification accuracy of the model in the target domain is related to the selected network structure. This is intuitive since different network structures have different model capacities that can learn features with the different sensitivities to the source-to-Be target feature variations.
SUG Implementation. Although naive UDA baseline coupled with designed domain split modules can enhance the model's s zero-shot recognition ability, it is still important for one-to-many adaptation to exploit multi-modal feature variations across different sub-domains, and further learn as many domain variances as possible. We conduct the experiments using the designed MSA and SDA, and the results are shown in Table and Random Splitting applied to obtain sub-domains.
Medefinnd to Nea3
500
Figure 4: tSNE results (PointNet). Different colors denote different classes.
8

---

Under review as a conference paper at ICLR 2023
Table 3: Results on down-sampling the whole source dataset using different methods.
Down-xampling Methods
Data Diversity
M
S
M
8
Avg.
Source Accuracy
Train Sample Size
Split& Select (Feature Clustering Splitting)
Low
48.9
33.6
41.3
8T.2
T015
Split & Select (Geometric Spliting)
Law
53.7
45.0
49.4
S0.1
975
Random Sampling
High
55.4
45.2
50.3
88.8
1044
Table 4: Ablation studies of tlass-wise classification accuracy, where the model is trained on
ModelNet-10 10 and directly evaluated on ScanNet- 10 (M (M->S*). PointNet is used as backbone.
Mellods
Align.
GS Aliga.
SV Align.
SIA
Hanhiih
Red
Beokshelr
Cahiset
Lamp
Monine
Pam
Tabl
Avg.
Supervaed
47.8
96.6
93.7
57.1
X3.5
Adapt
59.4
18.4
55.7
43.5
84.8
60.0
39.7
37.3
PoinDAAN
$4.7
1.6
19.0
1.3
81.9
63.3
90.5
82.3
2.2
82.9
51.0
68.1
2.6
20
0.0
49.0
53.9
95.3
86.4
0.3
79.3
45.5
64.5
5.6
17.0
0.7
75.2
61.4
96.5
78.6
0.2
88.4
48.2
SUG
64.1
0.0
17.8
1.43
75.9
55.7
92.0
90.0
0.0
85.0
48.2
63.9
34,0
74.1
91.7
77.3
86.7
69.7
73.3
93.7
80.9
48.4
76.9
2.0
25.0
81.5
89.7
88.2
85.0
50.8
First, our results show that the state-of-the-art, 3D-based UDA methods (Zou et al., 2021; Shen et al., 2022) cannot work well under the one-to-many generalization scenario. For example, GAST (Zou et al., 2021) can obtain a relatively high result (84.8% under M + S setting, but the adapted model has a serious accuracy drop (only 40.1%) under another target domain M 5 for our experiments. This is mainly because these methods often try to perform the explicit cross-domain alignment between the source domain and specific target domain, which is hard ensure that the adapted model has an even generalization toward different domains. In contrast, our SUG achieves higher one-to-many zero-shot generalization results for different target domains (e.g. 82.8% for S and 27 for 5).
To validate that SUG can be generalized to different point cloud backbones, we also conduct the DG experiments on many backbones (DGCNN, Point Transformer Zhao et al. (2021), and KP- Conv Thomas et al. (2019)). It can be seen from Table 2 that SUG yields consistent accuracy gains.
SUG Limitation. Our SUG framework assumes that the source domain dataset presents multi- modal feature distributions which can be implicitly exploited to model the feature distribution dif- ferences residing in the modti distributions. In 3D scenario, our assumption holds since the 3D point cloud samples for each class often have diverse appearances, geometric shapes, and etc, as shown in Fig. 3. Here, we further discuss the limitation cases of our SUG from: the diversity of source domain distribution gradually decreases.
To this end, we first split the given single dataset into M sub-domains, and then select one of the sub- domains from the splitting results (I out of splits) as the training set, which is described in Sec. 4.2 and denoted as Split& & Select. For comparison, we also randomly sample from the complete set of the given single dataset, which denoted as Random Sampling. The biggest difference between the above down-sampling methods is that single split (sub-domain) has much less data diversity and domain variances inside than the randomly sampled one that has a similar distribution status as the original dataset. It can be seen from Table 3 that, when the data distribution within the sampled sub-domain becomes more undiversified, the zero-shot generalization ability of model from a source domain to multi-target domains will drop.
4.4 FURTHER ANALYSES
Ablation Studies. In Table 4, we conduct the ablation studies from the following two aspects: 1) MSA method that consists of Class Distribution (CD Align.), Geometric Shifting (GS Align.), and Semantic Variance (SV Align.) alignments; 2) SDA strategy. First, MSA learns the domain-agnostic features from various granularities including class-level, geometry-level, and semantic-level. We observe that each newly-added alignment constraint can bring accuracy gains. Besides, we also conduct experiments of removing the SDA to investigate the effectiveness of the designed SDA. The results shown in Table 4 demonstrate that, by enhancing some casy-to-adapt instances to keep an even adaptation, SDA significantly boosts the generalization accuracy from 48.4% to 50.8%.
ISNE Results. We visualize features from source-only model and our SUG under in Fig. 4. The visualizations show that features learned by SUG can improve the model discriminability of different classes' features from unseen domains. For more visualization results, please see Fig. 7-8 in Sec. A.
CONCLUSION We have proposed SUG framework tackle the one-to-many Domain Generalization (DG) prob- lem in 3D scenario. SUG consists of a MSA method to exploit the data diversity residing in a given source dataset and further learn domain-agnostic and discriminative representations, a SDA strategy to selectively increase the domain adaptation degree for casy-to-adapt instances. Experiments are conducted on public benchmarks to show the effectiveness of SUG in tackling the 3D DG problem.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Idan Achituve, Haggai Maron, and Gal Chechik. Self-supervised learning for domain adaptation point clouds. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 123-133, 2021.
Kelsey Allen, Evan Shelhamer, Hanul Shin, and Joshua Tenenbaum. Infinite mixture prototypes for few-shot learning. In ICMI., 2019.
Karsten M Borgwardt, Arthur Gretton, Malte J Rasch, Hans-Peter Kriegel, Bernhard Schölkopf, and Alex J Smola. Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics, 22(14):e49 2006.
Fabio M Carlucci, Antonio D Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Do- main generalization by solving jigsaw puzzles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2019.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song. Hao Su, et al. Shapenet: An information-riee 3d model repository. arXiv preprint 15 2015.
Wuyang Chen, Zhiding Yu, Shalini De Mello, Sifei Liu, Jose M Alvarez, Zhangyang Wang, and Anima Anandkumar. Contrastive syn-to-real generalization. In ICLR, 2021.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5828-5839, 2017.
Hehe Fan, Xiaojun Chang, Wanyue Zhang, Yi Cheng, Ying Sun, and Mohan Kankanhalli. Self- supervised global-local structure modeling for point cloud domain adaptation with reliable voted pseudo labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6377-6386, 2022. 2022.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International Conference on Machine Learning, 2014.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recogni- tion (CVPR'06) volume 2. pp. 1735-1742, IEEE, 2006.
Guoliang Kang. Lu Jiang, Yunchao Wei, Yi Yang, and Alexander Hauptmann. Contrastive adap- tation network for single-and multi-source domain adaptation. IEEE transactions on pattern analysis and machine intelligence, 2020.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On latce-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv: /60900433. 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. abs/1442.69800, 2014.
CoRR,
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp- 2980-2988, 2017.
Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip S Yu. Transfer feature learning with joint distribution adaptation. In Proceedings of the IEEE international conference on computer vision, pp. 2200-2207, 2013.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International Conference Machine Learning. 2015.
Mingsheng Long, Yue Cao, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Transferable representation learning with deep adaptation networks. IEEE transactions on pattern analysis and machine intelligence, (112)33771-3088 2018a.
10

---

Under review as a conference paper at ICLR 2023
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Conditional adversarial do- main adaptation. In Advances in Neural Information Processing Systems, pp. 1640-1650, 2018b.
Xiaoyuan Luo, Shaolei Liu, Kexue Fu, Manning Wang, and Zhijian Song. A learnable self-supervised task for unsupervised domain adaptation on point clouds. arXiv preprint arXiv:2/04.05164, 2021.
Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Efficient domain generalization via common-specific low-rank decomposition. In International Conference on Machine Learning, pp. 7728-7738. PMLR. 2020.
Pytorch. Pytorch cosine embedding loss, 2022. URL https://pytorchh ortp:/cs...r//ooss//aoooooooo nn Cwsi CossineEmbeddinggos htmm.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652 660, 2017a.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++: Deep hierarchical fea- ture learning point sets in metric space. Advances in neural information processing systems, 30, 2017b.
Can Qin, Haoxuan You, Lichen Wang, C-C Kuo, and Yun Fu. Pointdan: A multi-scale 3d domain adaption network for point cloud representation. Advances in Neural Information Processing Systems, 32, 2019.
Yongming Rao, Jiwen Lu, and Jie Zhou. Spherical fractal convolutional neural networks for point cloud recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 452 460, 2019.
Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. Octnet: Learning deep 3d representa- tions at high resolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3577-3586, 2017.
Radu Bogdan Rusu and Steve Cousins. 3d is here: Point cloud library (pel). In 201 IEEE interna- tional conference on robotics and automation, pp. 1-4. IEEE, 201
Radu Bogdan Rusu, Zoltan Csaba Marton, Nico Blodow, Mihai Dolha, and Michael Beetz. To- wards 3d point cloud based object maps for household environments. Robotics and Autonomous Systems, 56(1 1):92-9941, 2008.
Shiv Shankar, Vihari Piratla. Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross grasditt training. In ICLR, 2018.
Yuefan Shen, Yanchao Yang, Mi Yan, He Wang, Youyi Zheng, and Leonidas J Guibas. Domain adaptation on point clouds via geometry-aware implicits. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp. 7223-7232, 2022.
Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng
Li. Pv-renn: Point-voxel feature set abstraction for 3d object detection. Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 10529-10538, 2020.
Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. Multi-view convo- lutional neural networks for 3d shape recognition. In Proceedings of the IEEE international conference on computer vision, pp. 945-953, 2015.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision, pp. 443 450. Springer, 2016.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for au- tonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on com- pufer vision and pattern recognition, pp. 2446-2455 2020.

---

Under review as a conference paper at ICLR 2023
Hugues Thomas, Charles R Qi, -aaa-mema Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings the the IEEE/CVF international conference computer vision, pp. 6411-64200 2019.
Ross Tredinnick, Markus Broecker, and Kevin Ponto. Progressive feedback point cloud rendering for virtual reality display. In 2016 IEEE Virtual Reality (VR), pp. 301-302. IEEE, 2016.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7167-7176, 2017.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911),, 2008.
Kashi Venkatesh Vishwanath, Diwaker Gupta, Amin Vahdat, and Ken Yocum. Modelnet: Towards datacenter emulation environment. In 2009 IEEE Ninth International Conference on Peer-to-Peed Peer Computing, pp. 8 81-82. IEEE, 2009.
Shujun Wang, Lequan Yu, Caizi Li, Chi-Wing Fu, and Pheng-Ann Heng. Learning from extrin- sie and intrinsic supervisions for domain generalization. In European Conference on Computer Vision, pp. 159-176. Springer, 2020.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. Acm Transactions On Graphics (tog), 38(5): 1-12,2013
Yue Xu, Yong- Lu Li, Jiefeng Li, and Cewu Lu. Constructing balance from imbalance for long-tailes image recognition. arXiv preprint arXiv:2208. 02567, 2022.
Jihan Yang. Shaoshuai Shi, Zhe Wang. Hongsheng Li, and Xiaojuan Qi. St3d: Self-training for unsupervised domain adaptation on 3d object detection. Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp. 18368-0078,, 2021.
Chuangguan Ye, Hongyuan Zhu, Yongbin Liao, Yanggang Zhang, Tao Chen, and Jiayuan Fan. What makes for effective few-shot point cloud classification? In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1829-1838, 2022.
Tan Yu, Jingjing Meng, and Junsong Yuan. Multi-view harmonized bilinear network for 3d object recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 186-194, 2018
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint
2017.
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16259-16268, 2021.
Wei Zhu, Le Lu, Jing Xiao, Mei Han, Jiebo Luo, and Adam P Harrison. Localized adversarial domain generalization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 7108-7118, 2022.
Longkun Zou, Hui Tang, Ke Chen, and Kui Jia. Geometry-awand self-training for unsupervised domain adaptation object point clouds. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pp. 6403-6412, 2021.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 1 DISCUSSION OF THE HAND-DESIGNED DOMAIN SPLIT MODULES
In Table of the main text, we conduct the extensive experiments show the Domain General- ization (DG) results for different domain split modules. However, through extensive experiments, we observe that these DG results achieved by the domain split modules are unstable for different cross-domain settings. Here, we give two main reasons for such instability as follows.
1) The distribution shift patterns across datasets are quite different. ModelNet and ShapeNet are both CAD-generated datasets. As result, they contain similar geometric characteristics, and at least both of them are without occlusions and follow a similar appearance. In this way, using "Fea- ture Clustering" to emphasize the semantic discrepancy and alignment would bring more gains (As reported in Table 1 for M + S and S - M experiments). In contrast, ScanNet obtained from the real world and originally designed for segmentation tasks. In other words, it is quite different both in semantic and geometric views, as shown in Fig. 3. In such a situation, emphasizing solely geometric or semantic discrepancy is optimal while random splitting is strong baseline to conduct the alignment operation, and this phenomenon is consistent with our experiment results in S + M, S + S, and S S* cross-domain settings.
2) The split results achieved by the domain splitting strategy are quite imbalance along the sample size. Take "Entropy Clustering" as an example, since the pre-trained model (on source-dataset) with source domain-related distribution characteristics is used, the model will be quite confident on predicting the source-domain samples, resulting in a quite imbalance clustering result. For example, the PointNet backbone on ScanNet dataset will get 3504 samples vs. 2606 samples splitting results for each sub-domain. But it will get worse on the easier dataset like ModelNet, where 2542 samples vs. 1641 samples splitting results for each sub-domain. Such imbalance is harmful for model training since that will bring the bias from the source dataset characteristic to the training procedure.
Table 5: The number of parameters of the backbone networks employed by SUG.
Network
Parameters
PointNetQi et al. (2017a)
3.5M
DGCNN Wang et al. (2019)
1,8M
PointTransformerr/haa et al. (2021)
9.6M
KPConyThomas et al. (2019)
5.3M
Table 6: Results on PointDA-10 under the one-to-many Domain Generalization (DG) setting with
additional backbones e.g. KPConv and Point Transformer.
M as Source Domain
Method
Setting
Backbone
M + S
M + S*
Avg.
KPConv
81.76
46.06
63.91
w/o Adapt
Source-Only
Point Transformer
84.11
54.83
69.47
KPConv
81.08
47.67
64.38
our SUG
DG
Point Transformer
83.36
58.35
70.86
A.2 COMBINATION WITH POINT TRANSFORMER AND KPCONV
To further verify the superiority of our SUG in boosting more baseline models, we select two state- of-the-art 3D point-cloud backbone networks, e.g., Point Transformers Zhao et al. (2021) and KP- Conv Thomas al. (2019) to conduct the one-to-many DG study. The corresponding experimental results are shown in Table 6.
According the above experimental results shown in Table 6, we summarize the following two main empirical findings.
1) By coupling with method, the Point Transformer Zhao et al. (2021) can achieve a better one- to-many DG classification performance gain, such 1.39 for M + S, M + S* settings. But it should be pointed out that the accuracy gain of Point Transformer is relatively slight compared with that
13

---

Under review as a conference paper at ICLR 2023
of the DGCNN backbone. This is mainly due to the fact that the cansformer-basee methods could learn much discriminative features during the model training phase, which is consistent with the observations in Allen et al. (2019). But Point Transformer Zhao et al. (2021) also takes much more time in model training and hyper-parameter tuning (As reported in Table 5).
2) The DG classification performance gain KPConv Thomas al. (2019) is quite minor, which is mainly due to that, the dataset-related parameter settings like query radius, are sensitive to different larget domains. Besides, we observe that, during the inference process where the points selected by the kernel of KPConv in ModelNet are generally more than 100 points (the first layer), but less than 80 points selection could happen if we did not change that parameters when used for ScanNet. The cross-domain feature alignment process brings more negative effects towards source-similar ModelNet than positive gains towards source-dissimilar ScanNet, which results in lower average classification accuracy across different datasets.
A.3 QUALITATIVE ANALYSES
More tSNE results between source-only model and our SUG. In the main text, we have shown the tSNE visualization results of high-level features learned by the source-only model and our SUG, respectively. In this part, we give more iSNE visualization results for more cross-domain settings such as the adaptation from ShapeNet-10 illustrated in Fig. 8 to 7, these visualization results demonstrate that the features from an unseen target domain (e.g., ModelNet- have distinct feature discrimination for different classes, further verifying that the learned features are domain-agnostic and discriminative for unseen domains.
10, ShapeNet-10 ModelNet-
ScanNet-10, etc. As
degpehe df So
02
43
Figure 5: ASNE results of ModelNet-10 and ScanNet- 10 datasets, where the model is trained on ShapeNet- 10 dataset and different colors denote different classes.
62
05
SEG/mm)
Figure 6: tSNE results of ModelNet-10 10 and ShapeNet-10 10 datasets, where the model is trained on Sca dataset and different colors denote different classes.
More tSNE results of domain split modules. In this part, we split the training dataset (source domain) into two sub-domains. And then we use the baseline model and the proposed SUG to train on those two sub-domains, respectively. After the training process ends, we visualize and compare the extracted features of the baseline model and the proposed SUG by t-SNE, respectively. The visualization results are shown in Fig. 8.
More ISNE results of sub-domains characteristics using random splitting module. Moreover, in order to validate the consistency of the distribution from sub-domains characteristics with the Random Splitting module, we split single source dataset into different sub-domains using the random sampling strategy. Then we use the pre-trained model to extract features from each sub- domain and tSNE is applied compare the features. The visualization results shown in Fig. 9.
14

---

Under review as a conference paper at ICLR 2023
Maldhhe-1l 18 ScatNe568
3010
SEG tan))
Figure 7: tSNE results of ScanNet- 10 and ShapeNet- 10 datasets, where the model is trained on ModelNet-10 10 dataset and different colors denote different classes.
w/
62
with
ModelNet
ScanNet
ShapeNet
Figure 8: iSNE results of sub-domains without and with alignment module. The first and second rows show the learned features without or with the feature alignment process, respectively. Different colors denote features from different sub-domains.
A.4 DISCUSSION ON THE ALIGNMENT CONSTRAINTS.
Discussion the scope of MMD constraints. Generally speaking, the alignment should be con- ducted between different modalities. And as shown in Fig. 3(b), similar modalities exist across classes while different modalities exist within single class. As result, exploit multiple modality information from both intra- and inter-classes, and thus do not perform hard class-wise MMD-based alignment. Specifically, avoid losing the label information, we first turn the class label into scaled one-hot vector and then concatenate it with the feature maps before conducting the MMD alignment, which is termed as the "Soft-MMD".
expected to fully
Besides, we have implemented different MMD- based alignment methods by changing the class-label information constraint, such as FHard-MMDD which means that only samples from the same class are aligned, and "Max-Hard MMD" which means that we first re-order the samples from different domains let them have most class overlapping, and then conduct the Hard-MMI. Through our experiments, we found that Soft-MMD outperforms other MMD-based alignment designs, as shown Table 7.
Comparison between Contrastive Loss and MMD loss. Contrastive Loss (CL) is also known for its capability of constraining learned features. We conducted experiments to compare the perfor- mance between CL and MMD loss used in SUG framework. Actually, for the implementation of Contrastive Loss, we directly use the Pytorch Implementation Pytorch (2022), which variation of the Hadsell al. (2006) with cosine distance.
15

---

Under review as a conference paper at ICLR 2023
ModelNet
ScanNet
ShapeNet
(SNE results of different sub-domains divided by Random Splitting module without using domain alignment. Different colors denote features from different sub-domains.
Figure 9:
Table 7:
Average results of unseen domains and S among different MMD-hased alignment methods, and we employ the M as the source domain.
Alignment Soft-MMD
Avg. Results
52.5
Hard-MMD
51.8
Max- MMD
52.3
Specifically, for the CL. have explicitly define the positive and negative pairs, which is quite complex in our setting. Positive pairs are the samples with similar geometric appearances for geo- metric alignments, regardless whether they are from the same class. contrast, the positives always from the same class for semantic alignments. For simplicity, we directly take the geometric features as negative pairs when they come from different classes under the CL constraint.
Experimentally, we use ModelNet as the source domain and evaluate on ShapeNet and ModelNet. We report the average results on these two datasets. Note that we have yet to tune the parameter for CL. loss much.
Based on the above experimental results, we summarize the following empirical findings.
1) As we can see from Table 8, when we replace the MMD loss with CL loss for semantic-level alignment, the final results are still competitive since both CL and MMD can make learned features to be domain-invariant. However, the results for CL loss for geometric-level alignment are much worse. The main reason behind those accuracy differences that CL focuses on capturing the high- level feature variances while it tends to ignore some low-level information for describing domain shift.
2) Based on the experiments in Table 8, we are delighted that the SUG has the potential to be unified framework where the sub-domain alignment module could be replaced using other recently- proposed alignment loss function such Contrastive Loss.
Table 8: Average results of unseen domains S and S* using the Contrastive Loss (CL) and MMD alignment designs, and we employ the M as the source domain. Geo and Sem stand for geometric and semantic alignment.
Alignment
Gco-MMD
Geo-CL
Sem-MMD
Sem-CL
Avg. Results
1-MMD Original
X
X
0.5245
2-CL
X
X
0.4598
3-MIX
X
X
0,5234
4-Mix
X
0.4725
A.5 DISCUSSION ON IN SUG.
16

---

Under review as a conference paper at ICLR 2023
The experiments in this part are conducted employing PointNet the backbone and ModelNet as the source domain. We report the average prediction results on ShapeNet and ScanNet.
The À in Eq. 11 achieves trade-off between the classification task and the alignment process. We conduct the ablation studies to study the sensitivity of the À value setting on our SUG performance. The corresponding results are shown in Table. 9.
Weight A.
We also conduct the experiments of changing the batch-size value, where we keep other settings as default. The results are shown in Table. 10.
Batch Size B.
Table 9; Average results of unseen domains S and S using different À values in Eq. 11, and we
employ the M as the source domain.
X
Avg. Results
0.25
44.7
0.50
52.5
0.75
51.2
1.0
47.8
2.0
49.1
3.0
48.5
4.0
45.9
5.0
44.7
Table 10: Average results of unseen domains S and S trained with different batch sizes, and we employ the M as the source domain.
Batch Size
Avg. Results
16
51.35
32
52.86
64-Default
52.45
128
50.45
256
50.52
512
47.51
Table 11: Average results of unseen domains S and Sh trained with different layer selection settings, and we employ the M as the source domain.
Embedding Module Layer
Avg. Result Classification Module Layer
Avg. Result
Layer-1
48.8
Layer-1
49,9
Layer-2
49.7
Layer- 2(Defull))
52.5
Layerr-3(Deaaull
52.5
Layer-3
48.2
Layer-4
49.4
Layer-5
48.2
According to the above experimental results shown in Table 10, we find that our method can achieve good generalization ability across different batch-size settings. For the batch-size setting with small value, the mini-batch data could not contain enough information related to the domain distribution. As result, the SUG could not learn the served that the degradation of generalization's 's ability when we continuously enlarge the batch size, which is mainly due to that the large-batch training procedure tends converge sharp minimiz- ers Keskar et al. (2016).
features well. In contrast, it can be ob-
Layer selection for low-level and high-level features. In the default SUG setting, we use F's third layer and Co's second layer as the low and high-level features, respectively. To further explore how the layer selection for features would affect the SUG performance, we change the selection choices of the layers. Specifically, in order to validate the choice for geometric features, we use the features from {1, 2, 3, 4, 5)-layer of the embedding module as the geometric features while keeping the second layer of the classification module default. For semantic features experiments, we used the features from {1, 2, 3) llyer of the classification module while keeping the third layer of the embedding module default.
17

---

Under review as a conference paper at ICLR 2023
Based on the above experimental results, summarize the following empirical findings.
1) For Embedding Module Layer Selection: The features from too shallow layers (e.g.. Layer- 1) contain much less information and would be sensitive to noise. In contrast, if we choose the features from too deeper layers, the geometric and fine-grained information would be overtaken by the deep semantic information. At the same time, when we choose that deeper features, the geometric alignment would be much similar to semantic alignment and thus lose its discriminability.
2) For Classification Module Layer Selection: The features from the shallow layer (e.g., Layer-1) are similar to the geometric ones and would lose semantic alignment ability. At the same time, the last layer's features are too high-level and lose lot of semantic information.
18


Reviews:
[{"review_id": "ugRGVaj0taF", "reviewer": "Reviewer_AoJ5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the single domain generalization problem for 3D point cloud classification, which aims to generalize the model trained on a single source domain to an unknown target domain. The method splits the training domain into two subdomains, and models the alignment between the two sub-domains, expecting the model to learn domain-generalized features. The evaluation is conducted on ModelNet-10, shapeNet10, and ScanNet-10 by considering two networks, PointNet and DGCNN.", "summary_of_the_review": "The addressed problem is interesting and important but the experiments are not adequate.\n\n\n=================== post-rebuttal =====================\nIn the response, the authors have made efforts to conduct additional experiments to address my questions. However, I still have a main conern: different split strategies have different impacts for performance on different datasets and networks.", "strength_and_weaknesses": "=== Strength === \n\n1. The studied problem is interesting and important, since domain generalization in 2D image has been studied extensively while in 3D point cloud still under-explored.\n\n2. The method is introduced clearly and is easy to follow.\n\n=== Weaknesses ===\n\n1. The sub-domain splitting strategy. In Tab.1, we can find that the performance of different splitting strategies is not stable, as the performance varies from different datasets. A reasonable explanation should be given. In addition, the features of the sub-domains with/without alignment should be compared. For example, extract a validation set from the training set, then train two networks (baseline, the proposed), then visualize the features and make a comparison. I am wondering whether the subdomain splitting makes sense or not.\n\n2. Following the above comment, in fact, the performance of the splitting strategy is also related to the network structure, which makes me confused about the effectiveness of the strategy.\n\n3. This paper only evaluates two very old networks, PointNet and DGCNN. As many SOTA networks are developed, more evaluations are expected, such as KPConv, and Point Transformer.\n\n4. In Page 5, the paper claims that the motivation behind weighted classification loss is different from previous methods. I am so confused. In fact, in this paper, this loss still addresses the class imbalance problem, and there is no difference from other methods.\n\n5. other concerns: how to select the best model for evaluation? how about the variances of the results over three runs? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "k0p8nlQTwfv", "reviewer": "Reviewer_i2R8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method for domain generalization for 3D point clouds-based deep learning (i.e., pre-training a 3D model on a source domain to well generalize on a new and unseen domain without accessing any data points from the new domain). The proposed method includes three main components: domain splitting (to split a source domain into sub-domains), multi-grained sub-domain alignment (to align features at different levels from sub-domains), and sample-level domain-aware attention (to weight samples based on their distances to their domain). The proposed method was experimented with two point cloud-based backbones (PointNet, DGCNN) and evaluated on benchmark datasets (ModelNet, ShapeNet, and ScanNet), and also compared with existing ones on these datasets. ", "summary_of_the_review": "My detailed comments for the paper are as follows.\n\nA. Presentation:\n- Many technical details are missing. For instance, it is not clear how the mini-batches can be created to include data points from different sub-domains. It is also not clear what layers should be considered to extract low-level and high-level features. What is the kernel function used in Eq (5) in experiments? Do i and j in Eq (5) need to represent samples of the same class? Also, Eq (5) assumes that there are two sub-domains, but what if there are more than 2 domains? What are X and Y in Eq (7) and Eq (8), are they from the same class or different classes? If X and Y come from different classes, then d in Eq (6) cannot show the domain distance. What is lambda in \"Implementation Details\"? What setting (e.g., splitting method) is used for SUG in Table 2 and Table 4? Why the results of SUG in Table 2 do not look like any results of SUG in Table 1?\n- Notation is not used consistently, leading to confusion. For instance, in Eq (4), n is referred to as the number of samples in a class but it is not so in Eq (5). Also, do n_s and n_t represent the number of samples in sub-domains s and t? The same notation for omega in Eq (4) and Eq (6), but, I believe, these omega parameters are different in these two equations. In 3.1, K is referred to as sub-domains, while denoted as M in 4.3. \n\nB. Methodology\n- For geometric splitting, what if Chamfer distance is used to compare two different point clouds?\n- I was wondering what if a contrastive loss is used for the L_MMD_Geo in Eq (5) as contrastive loss is also known for the capability of constraining learnt features. A comparison between the MMD and contrastive loss would be useful to consolidate the proposed solution.\n\nC. Experiments and results\n- How many sub-domains are used in experiments and how is the method affected by the number of sub-domains?\n- I was wondering how the method performs under different batch sizes, as the method may be affected when the batches do not well reflect the data distributions in sub-domains. \n- It is important to experiment the method with different selections of the layers for the low-level and high-level features\n- As mentioned in comments for methodology, it would be great to see a comparison between MMD and contrastive loss.\n- ScanObjectNN in [a] is a real-world dataset including practical challenges such as occlusions, background intervention, etc., that do not exist in ModelNet, and therefore would be ideal to validate the domain generalization ability of the proposed method. Note that, this dataset also shows that many models, successful on ModelNet, fail on it. \n- The proposed method does not outperform state-of-the-art (e.g., SLT)\n- The role of SDA is not very clear (given experimental results). And, it may also be affected by batch setting. This should be verified.\n- Also in terms of performance, SUG is well below supervised learning approach. On the one hand, I do understand that the studied problem is much more challenging than supervised learning approach. On the other hand, the performance of SUG shows that its current status is not at the applicable level. \n\nD. Others\n- It would be clear to explicitly state that d in Eq (6) can be realized using either Eq (7) or Eq (8).\n- It is not clear how the domain characteristic of each sub-domain is identical with the original domain from the following claim \"We conduct the random sampling and split a single source dataset into different sub-domains with the same sample size, where domain characteristic of each sub-domain is identical with that of the original one\". It may be over claimed.\n\nE. Missing reference\n- [a] Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data, ICCV19.\n", "strength_and_weaknesses": "A. Strength\n- The studied problem is important.\n- The proposed method and its components are well-motivated.\n\nB. Weaknesses\n- Presentation needs improvement (many technical details are missing; notation is not used consistently)\n- Experiments are lacking in some aspects.\n- The proposed method does not outperform state-of-the-art (SLT, CVPR22)\n- The performance is quite low compared with supervised learning approach, limiting the practicality of the proposed method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YD2PKwjKDy", "reviewer": "Reviewer_Yd9F", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a Single-dataset Unified Generalization (SUG) framework for the challenging one-to-many domain generalization task in 3D point clouds. A Multi-grained Sub-domain Alignment (MSA) is proposed for spliting the single dataset into multiple sub-domains and then constraining the learned representations to be domain-agnostic and discriminative. A Sample-level Domain-aware Attention (SDA) is further proposed for adapting the uneven domain to an even inter-domain. Extensive experiments on several benchmarks illustrate the effectiveness of SUG.", "summary_of_the_review": "Overall this work is well-motivated and fits well in the one-to-many domain generalization task in 3D point clouds. ", "strength_and_weaknesses": "Strengths:\n1.\tThis work is technically sound and shows promising results. \n2.\tThe design of the proposed SUG framework fits well in the one-to-many DG task. \n3.\tThe extensive experiments on 3 well-established datasets verify the efficiency of the proposed approach.\n\nWeakness:\n1.\tThe differenences, characteristics and comparisons of different datasets has not been fully discussed.\n2.\tLack of introduction to DGCNN and motivation for choosing to use DGCNN as the backbone network.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject