Domain-invariant Feature Exploration for Domain Generalization
Authors: Wang Lu
Jindong Wang
Haoliang Li
Yiqiang Chen
Xing Xie
Venue: Accepted by TMLR ()

Abstract: Deep learning has achieved great success in the past few years. However, the performance of deep learning is likely to impede in face of non-IID situations. Domain generalization (DG) enables a model to generalize to an unseen test distribution, i.e., to learn domain-invariant representations. In this paper, we argue that domain-invariant features should be originating from both internal and mutual sides. Internal invariance means that the features can be learned with a single domain and the features capture intrinsic semantics of data, i.e., the property within a domain, which is agnostic to other domains. Mutual invariance means that the features can be learned with multiple domains (cross-domain) and the features contain common information, i.e., the transferable features w.r.t. other domains. We then propose DIFEX for Domain-Invariant Feature EXploration. DIFEX employs a knowledge distillation framework to capture the high-level Fourier phase as the internally-invariant features and learn cross-domain correlation alignment as the mutually-invariant features. We further design an exploration loss to increase the feature diversity for better generalization. Extensive experiments on both time-series and visual benchmarks demonstrate that the proposed DIFEX achieves state-of-the-art performance.

Published in Transactions on Machine Learning Research (07/2022)
Domain-invariant Feature Exploration for Domain General- ization
Wang Lu Institute of Computing Technology, Chinese Academy of Sciences
kuvangBict.oc.or
Jindong Wang"
jandong.wong wwwg@mmicroooof..ccomm com
Microsoft Research Asia
Haoliang Li City University of Hong Kong
Rtaliang.li@csttyn.eed
Yiqiang Chen
wgchen@ict.ac.co
Institute of Computing Technology, Chinese Academy of Sciences. Peng Cheng Laboratory
Xing Xie
zingz@microsoft..com com
Microsoft Research Asia
Reviewed on OpenReview: https: // openrevie.......... net/ forum? id= OOENE7H4
Abstract
Deep learning has achieved great success in the past few years. However, the performance of deep learning is likely to impede in face of non-llI situations. Domain generalization (DG) enables model to generalize invariant representations. In this paper, we argue that domain-invariaat features should be originating from both internal and mutual sides. Internal invariance means that the features can be learned with single domain and the features capture intrinsic semanties of data, i... the property within a domain, which is agnostic to other domains. Mutual invariance means that the features can be learned with multiple domains (cross-domain) and the fcatures contain common information, i.c., the transferable fcatures w.r.t. other domains. We then propose DIFEX for Nomainn-Ivarrinn employs knowledge distillation framework to capture the high-level Fourier phase as the internally-invariant features and learn cross-domain correlation alignment the mutually- invariant features. We further design exploration loss to increase the feature diversity for better generalization. Extensive experiments both time-series and visual benchmarks demonstrate that the proposed DIFEX achieves state-of-the-art performance.
an unseen test distribution, i.e., to learn domain-
Feature EXploration. DIFEX
1 Introduction
Over the past years, machine learning, especially deep learning has achieved remarkable success across wide application arcas such as computer vision (He et al.) 2016) and natural language processing (Vaswani et al. 2017). However, machine learning generally assumes that the training and test datasets are identically and independently distributed (IID) (Esfandiari et al.) 2021 Xu et al., 2019), which may not hold in reality. Such non-lID issue more practical and challenging. For instance, we expect that a model that recognizes the activities of a child can generalize well on the data from an adult, even if their data distributions are different due to different lifestyles and body shapes.
When the target data is available for training (e.g,, the adult data is accessible), domain adaptation (DA) Wilson & Cook 2020) can be employed to handle the non-IID issue and learn an adapted model
* *Corresponding author. Code: http:///..c....oooo/////////eenr

---

Published in Transactions on Machine Learning Research (07/2022)
for the target domain. However, a more practical situation is when the target domain is unseen in training, which makes existing DA approaches not feasible. Domain generalization (DG), or out-of-distribution gen- eralization is one of the most popular research topics that aims to solve such problems (Wang et al. 2022).
DG learns a generalized model from multiple training datasets that can generalize well on an unseen dataset.
There are several approaches for DG, such as data augmentation (Tobin et al. 2017; Shankar el al. 2018
Lu et al.) 2022) that increase data diversity and meta-learning (Li et al. 2018a Balaji et al. 2018 that learns generally transferable knowledge by simulating multiple tasks.
Different from these two approaches, Nomain-invarien feature learning is popular DG strategy that aims to learn representations that remain invariant across different domains, thus benefiting cross-domain gener- alization. For instance, Ganin et al.] (2016) designed the domain--adversariaal neural network (DANN) using adversarial training, where they tried to confuse the domain classifier such that it could not distinguish which domains the features belonged to, thus achieving domain-invariant learning. Similar to DANN, many other methods are proposed to learn ionaintinvariant features for DG and achieved great success (Muandet
et al. 2013 Li al. 2018b; Matsuura & Harada 2020 Sun & Saenko 2016).
The popularity and effectiveness of domain-invarien learning naturally motivate to seek the rationale behind this kind of approach: what are the invariatn features and how to further improve its perfor- mance on DG? Previously, Zhao et al. (2019) showed that feature alignments across domains are not enough for domain adaptation and they paid attention to label functions. However, due to unseen targets in DG, label functions cannot be accessed and Bui et al. (2021) utilized meta- Peeta-Dmmain Specific-Domade Invariant to solve the above problem. In this paper, we take deep analysis of the domain-invariant features. Specifi- cally, we argue that domain-invariant features should be originating from both internal and mutual sides: the internally-invariann features capture the intrinsic semantic knowledge of the data while the mutually- invariant features learn the cross-domain transferable knowledge. On the one hand, the internally-invariant features are born with the input data that do not change with the existence of other domains. On the other hand, the mutually-invariiat features focus on harnessing the cross-domain transfcrable knowledge that is mined from different distributions. Therefore, the inlegration of these features can ensure better generalization to unseen domains.
We propose DIFEX for Domain-Invariant Invariant Feature EXploration of the internally- and muntually-invariat representations. Specifically, for internal features, DIFEX employs a knowledge distillation framework to capture the high-level semantics using Fourier transform. For mutual features, DIF EX utilizes correlation alignment to align the feature distributions of any two domains. To allow mutual feature exploration, DIFEX further adds a regularization to maximize their divergence. We conduct extensive experiments across both image data and time series datasets to comprehensively show the advantage of DIFEX. Results show that our DIFEX outperforms other recent baselines in all datasets. Our contributions are mainly three-fold:
1. We propose DIFEX for Domain-Invvrriat Feature EXploration of both internally- and mutually- invariant representations. We further develop a regularization to maximize their divergence to allow for more feature exploration! !
2. Comprehensive experiments on both visual and time series in several different settings demonst the superiority and universality of DIFEX.
3. Our method can inspire the community to explore more diverse features from both inter- and intra- domain simultaneously, which may be a promising direction for future DG research.
2 Related Work
2.1
Domain Generalization
Existing domain generalization approaches (Wang et al. 2022 mainly consist of data augmentation (Shankar
et al.
2018 Tobin et al. 2017).
feature learning (Muandet et al. 2013 Ganin et al.] 2016
TTo the best of our knowledge, DIFEX is the first work that proposes these two concepts (internally- and mutually-invar invariatt representations), and utilizes inheritance and exploration simultaneously in DG.
2

---

Published in Transactions on Machine Learning Research (07/2022)
Li et al. 2018b), and meta-learning (Li et al. 2018a Balaji et al. 2018 techniques. Most of existing work pays itt to some specific applications such as computer vision and reinforcement learning. Our work belongs to Nomain-invariant feature learning-basee based DG, where we mainly focus on common feature learning methods which can be directly applied to different tasks.
2.2
Domain-invariant Representation Learning
Domain-invariant representation learning (Johansson et al., 2019; Ben-David al.) 2010 has long been
popular solution to addressing domain shift problems such domain adaptation GGanin & Lempitsky
2015
Ganin et al. 2016; Zhao et al. 2019) and domain generalization {Li et al. 2018b Muandet et al.
2013
Matsuura & Harada, 2020). Specifically for DG, learning invarin representations is critical
since DG focuses on the invariance across domains. Such domain invariance is achieved can be achieved by explicit feature alignment (Li et al. 2018b Motiian et al., 2017), adversarial learning (Ganin et al., 2016 2018c: Rahman al. 2020), or disentanglement (Liu al. 2021 Piratla et al.) [020:] Isse et al, 2.20 Recently, some work pointed that only simple alignments maybe not enough for generalization (Bui ct al. 2021 Our work shares the same goal with them, but with more possibilitics to explore other invariant features that allow more diversities to increase the generalization ability. Our DIFEX may seem similar to the disentanglement framework from many existing efforts since it also learns two kinds of features and then combines them for final prediction. However, it significantly differs from them since we are actually learning two kinds of domain-invariant features, while disentanglement often learns invariant and specific features (Zhao al. 2019).
Li et al.
2.3 Fourier Features
Xu et al. 2021). FDA (Yang & Soatto, 2020) introduced the Fourier perspective into domain adaptation. It could generate target-like images for training by simply replacing small area in the centralized amplitude spectrum of a source image with that of target image. FACT (Xu et al.) 2021 illustrated that Fourier phase information contained high-level semantics and was not easily affected by domain shifts. It utilized the mixup 4Zhang et al. 2018) technique to enlarge the diversity of Fourier amplitude without changing labels and thereby learned model that was insensitive to Fourier amplitude. Another work (Xingchen et al. 2021) utilized Fourier amplitude (style) to calibrate the largel. domain style on the fly. However, il required some operations when testing. Some other work tried to control Fourier amplitude during training for better gencralization (Lin et al., 2022: Zheng ct al.] 2022). These work are designed for the visual field, although much work (Oppenheim et al. 1979 Oppenheim & Lim 1981 Hansen & Hess 2007) have demonstrated that the phase component retains most of the high-level semantics in original signals, which suggests that Fourier phase may be a kind of universal Nomain-invariian features.
Recently, some work paid attention to Fourier phase in visual recognition (Yang & Soatto 2020:
3 Methodology
3.1 Problem Formulation
We follow the problem definition in (Wang et al. 2022) to formulate domain generalization in a C-class classification setting. In domain generalization, multiple labeled source domains, = (8 = 1, M) are given, where M is the number of sources. S = [(x), 1f denotes the ith domain, where ny denotes the mumber of instances in St. The joint distributions between each pair of domains are different: Pxx # Plr:1 < i $ < M. The goal of domain generalization is to Icarn robust and generalized predictive function h X' + y from the M training sources to achieve minimum prediction error on an unseen target domain Stest with unknown joint distribution, i.e., min, E(x.)) [e(h(x), v)], where is the expectation and ((.,)) the loss function. All domains, including source domains and unseen target domains, have the same input and output spaces, i.c., X space, and y .... y.M yT (1, ,C), where y is the label space.
X XT E T where X' is the m-dimensional instance
3

---

Published in Transactions on Machine Learning Research (07 (07/202) /022)
3.2 Motivation
Existing literature YYng et al. 2020 Yang & Soatto 2020 Xu et al 2021 show that Fourier phase information contains high-level semantics that are not casily affected by domain shifts in the visual field. Many early studies (Oppenheim et al. 1979 Oppenheim & Lim 1981 Hansen & Hess 2007] also conclude that in the Fourier spectrum of signals, the phase component relains most of the high-level semanttcs in the original signals, while the amplitude component mainly contains low-level statistics. Thus, Fourier phase features can act the internally-invari fcatures, which, of course, are domain-invariant.
Raw data
Fourier phase
Fourier amplitude
Restored data via phase Restored data
A
VV
VV
B
Figure 1: Fourier features internally-invarian representations.
Is Fourier phase sufficient. for DG? Figure 1 shows sensor readings of the walking activity collected from two persons in the UCI DSADS dataset (Barshan & Yüksek 2014] We apply Fourier transform to these two samples and compute their corresponding phase and amplitude. From the results, it is obvious that Fourier phases are similar between two different persons while there exist large gaps in Fourier amplitude. Therefore, Fourier phase information can perfectly act as domain-invariant features to help generalization. We clearly see that the raw data of and B contains similar periodicity information that could further utilized for generalization. However, when we restore data with Fourier phases and the sample amplitude, the two restored samples both lose some information compared with raw data such the periodicity of walking. Thus, Fourier phase along would fail to capture the commonness from multiple training domains since they only focus on their own spectrum without considering the multi-domain statistics. Moreover, some exist.ngg work pointed that only simple alignments maybe not enough for generalization (Bui et al. 2021).
3.3 DIFEX
We propose Domain-Invariant Feature EXploration to learn both internally- and mutually-invariant features for domain generalization, short as DIFEX. As shown in Figure DIFEX takes inputs from multiple training domains. Then, after common feature extractor, we enforce the network to learn the internally- and mutually-invariaat features. Subsequently, these features concatenated to form the invariant features, which can then be used for classification. Furthermore, to allow the network to explore more diversity, we propose the exploration loss to regularize these features by maximizing their divergence.
Internally- invariant
Feature extractor
Invariant feature
O
Classifier
Max divergence
Mutually- invariant
Input
Figure 2: The framework of DIFEX.
We perform the shift operation, but the schematics have demonstrated that amplitade contains very little information.
4

---

Published in Transactions on Machine Learning Research (07 (07/2022)
3.3.1 A distillation framework to learn internally-invariant features
We show how to obtain internally-invariant features with Fourier phase information. As shown Fig- ure we utilize a distillation framework to obtaïn Fourier phase information for classification with raw data. Knowledge distillation is simple framework encourage different networks to contain particular
charactcristics (Hinton et al. 2015
Romero et al. 2014
Teacher
O
Transform
Fourier
Feature
Classifier
Raw
Phase
Liaa
Student
Feature
Classifier
Raw
Figure 3: Distillation framework to learn internally invariatt features using Fourier transform.
Concretely speaking, the teacher network utilizes Fourier phase information and class labels inputs and outputs, respectively, to obtain Fourier phase information features for classification. According to (Xu et al. 2021), the Fourier transformation F(x) for a single-channel two-dimensional data x is formulated as:
#-1 W Σ
F(x)(u,
whe 12) # ##)
(1)
h=1 w=0
where u and v are indices. H and W are the height and the width, respectively. Fourier transformation can be calculated with the FFT algorithm Nussbaumer expressed as:
1981) efficiently. The phase component is then
P(z)(u,v) arctan (())(u,))
(2)
(u,)
where R(x) and ((x) represent the real and imaginary parts of F(x), respectively. For data with several channels, the Fourier transformation for each channel is computed independently to obtain the corresponding phase information. We denote Fourier phase of x as &, then, the teacher network is trained using (&, y):
min B(x.)) (Gr (x)) 04.05
where of and 0z are the learnable parameters of feature extractor (G// and classification layer (Gt) of the teacher network. pir is distribution of training data, denotes expectation, and Late is cross-entropy loss which is a common function for classification.
Once obtaining the teacher network, Gr, we feature knowledge distillation to guide the student network to learn the Fourier information. Such distillation is formulated as:
min E. (x)). + X Cmse (G (x),)), (Z)),
8 05
where B5 and f are the learnable parameters of feature extractor (G and classification layer (G%) of the student network. A is tradeoff hyperparameter and Lmse is the MSE loss which can make features of the student network close to features of the teacher work.
5

---

Published in Transactions on Machine Learning Research (07/2022)
3.3.2 Explore mutually-invariant features
As discussed early, Fourier phase features alone are insufficient to obtain enough discriminative features for classification. Thus, we explore the mutually-invarianl features by leveraging the cross-domain knowledge contained in multiple training domains [ Specifically, given two domains S 8²,8', we align their second-order statistics (corrclations) using the correlation alignment approach (Sun & Saenko 2016
N
Catign
2 N (N - 1)
Σ
IC- C
(5)
where C' = -,- x - (1'X)) is the covariance matrix and ||x denotes the matrix
Frobenius norm.
Since there may exist duplication and redundancy between the internally-invariant features and mutually- invariant features, we expect that two parts can extract different invariant features as much as possible. This allows more diversities features that are beneficial to generalization. Towards this goal, we regularize the distance between the internally-invariant (21) and mutually-invarriat (z1) features by maximizing their divergence, which call the exploration loss:
Leap 22) -d12 22),
(6)
where d(.,-) distance function and we simply utilize the L2 distance for simplicity: Lozp - -1|2| 22
3.3.3 Summary of DIFEX
To sum up, our method split into two steps. First, we optimize Eq. 1 Second, we optimize the following objective:
+ Auso (2,, + deLatign + >z 22),
min 0y
where Ge and Gy are the classification layer and the feature net respectively while 0. and 0 are corresponding parameters. X1, X2, and λ are hyperparameters. Eq. contains four objectives: classification, internally- invariant feature learning, autuully--nvariin and the third terms are two common objectives in invariant representation learning for domain generalization, and they are not enough according to the existing work Zhao et al. (2019); Bui et al. (202I). With the help of exploration of internally-invariant features and diverse fcatures (the second and the last terms), we can alleviate the above problems to achieve better performance, which proved in later experiments. For the current implementation, we mainly tune the hyperparameters, which balances these four terms. In the future, we plan to design more heuristic ways to automatically determine the hyperparameters.
feature learning, and exploration of diverse features. The first
As for inference, when data sample x comes,
predict its label by simple forward-pass:
y arg min Gx)G)(x)).
(8)
3.4 Discussions
An alternative to learning Fourier features is to directly use the teacher network, and then use another feature extractor to extract mutuully-invvriian features. However, this would seriously increase the model size for inference as it requires two feature extractors. In addition, it also requires performing FFT for each sample as another input which is timo-consuming for inference. Although our distillation framework would introduce more parameters the training phase, it will not increase the model size and additional inputs for inference, which we believe is more useful in a real deployment.
One may argue that DIFEX may fail if there is only one training domain thus mutually-invarian features. We provide a positive answer to this situation with virtual domain labels, i..,, domain alignment training on one domain by giving random domain labels. This simple trick seems effective in real experiments.
3 *Mutually-invariint features focus on harnessing the cross- doss-domain transferable knowledge that mined from different. distributions. We think alignments among sources can bring common knowledge of sources.

---

Published in Transactions on Machine Learning Research (07 (0722)
4 Experiments
We extensively evaluate our method in both visual and time-series domains including image classification, sensor-based human activity recognition, EMG recognition, and single-domain generalization. The training data are randomly split into two parts: 80% for training and 20% for validation. The best model on the validation split is selected evaluate the target domain. For fairness, we re-implement seven recent
strong comparison methods following DomainBed (Gulrajani & Lopez-Paz 2021 ERM, DANN (Ganin 2016), CORAL (Sun & Saenko 2016). Mixup (Zhang et al.) 2018), GroupDRO (Sagawa et al. 2020
et al.
RSC
Huang et al. 2020), ANDMask (Parascandolo et al. 2021), FACT (Xu et al. 2021 and SWAD (Cha
et al.
2021). In addition, for specific applications, we also add some latest comparison methods to compare. Please nole that although our method contains two parts of Tealures, we maintain the same architecture lo features and the dimension
predict for fairness, which means that both the dimension of internally-invarii of the mutuaaly--nvariint features are only half of the dimension of the other methods.
4.1 Image classification
Table 1: Accuracy on DGG The bold and underline items are the best and the second-best results.
Results in the parentheses are the performance reported in other papers.
Source
Target
ERM
DANN
CORAL
Mixup
GroupDRO
RSC
ANDMask
SWAD
DIFEX
MM.SV.Y
M
97.55(95.80)
97.77
97.62
97.50(94.20)
97.48
97.78
96.85
97.37
97.82
M.SVSY
MM
55.52(68.80)
55.62
57.68
57.95(56.50)
53.47
56.27
56.00
55.77
57.90
M,M.SY
SV
59.98(61.70)
61.85
57.82
54.75(63.30)
55.63
62.38
59.47
59.00
64.30
M.M.SV
SY
89.25(78.60)
89.37
90.12
89.80(76.70) 70)
92.15
89.25
88.17
87.92
89.98
AVG
75.58(73.70)
76.15
75.81
75.00(72.70)
74.68
76.42
75.12
75.02
77.50
Table 2: Accuracy on PACS. The bold and underline items are the best and the second-best results.
Source
Target
ERM
DANN
CORAL
Mixup
GroupDRO
RSC
ANDMask
SWAD
DIFEX
C.P.S
77.00(77.63)
78.71
77.78
79.10(76.80)
76.03
79.74(83.43)
76.22
81.93
80.86
A.P.S
76.77)76.77)
75.30
77.05
73.46(74.900
76.07
76.11(80.31)
73.81
76.45
77.60
A,C.S
95.51 85)
94.01
92.63
94.49(95.60) 80)
91.20
95.57 (95.99)
91.56
94.67
95.57
A,C,P
77.86(69.50)
77.83
80.55
76.71 60)
79.05
76.64 (80.85)
78.06
76.79
79.49
AVG
81.22(79.94)
81.46
82.00
80.94(78.50)
80.59
82.01(85.13)
79.91
82.46
83.38
Table 3: Accuracy VLCS. The bold and underline items are the best and the second results.
Source
Target
ERM
DANN
CORAL
Mixup
GroupDRO
RSC
ANDMask
SWAD
DIFEX
L,S.V
C
93.64
94.49
96.33
96.18
97.81
96.89
93.50
94.56
96.61
C,S.V
60.05
64.34
64.42
63.55
62.24
62.91
64.83
61.78
67.21
C,L.V
68.46
67.15
68.65
68.86
69.23
69.07
63.28
67.82
74.31
C,L,S V
74.02
72.69
70.73
71.95
70.73
70.38
68.87
67.89
75.24
AVG
74.04
74.67
75.03
75.13
75.00
74.81
72.62
73.01
78.34
Datasets and implementation details We adopt three conventional DG benchmarks. (1) Digits- DG (Zhou et al. 2020) contains four digit datasets: MNIST {LeCun et al. 1998), MNIST-M (Ganin & Lempitsky 2015), SVHN (Netzer al. 2011). SYN (LeCun et al. 1998). The four datasets differ in font style, background, and image quality. Following (Zhou et al. 20201, we utilize their selected 600 images per class per datasct. (2) PACS [Li et al. 2017) is an object classification benchmark with four domains (photo, art-painting, cartoon, sketch). There exist. large discrepancies in image styles among different domains. Each domain contains seven classes and there are 9,991 images in total. (3) VLCS Fang et al. (013)) comprises photographic domains (Caltoch101, LabelMe, SUN09, VOC2007). contains 10,729 examples of 5 classcs.
For each dataset, we simply leave one domain the test domain which is unseen training while the others the training domains.

---

Published in Transactions on Machine Learning Research (07/2022)
For the architecture, we use ResNet-18 for PACS and VLCS, and Digits-DG dataset uses DTN the backbone followingLiang et al. (2020). Since Sincesplits, learning epochs, and some other factors all have influences on results, we mainly compare with the methods extended by ourselves for fairness. The maximum training epoch is set to 120. The initial learning rate for Digits-DG is 0.01 while 0.005 for the other two datasets. The learning rate decayed by 0.1 twice at the 70% and 90% of the max epoch respectively.
Results The results on three image benchmarks are shown in Table 2 and respectively. Overall, our method has the average improvements of 1.08%, 0.92%, and 3% average accuracy than the second best method three datasets. Visual classification for domain generalization is challenging task, and it is difficult to have an improvement over 1%. As can be seen in these tables, the sccond-best method only has a slight improvement compared to the third one. This demonstrates the great performance of our approach in these datasets. Moreover, we see that alignments across domains and exploiting characteristics of data own can both bring remarkable improvements. In addition, some latest methods such as ANDMask even perform worse than ERM on some benchmarks, indicating that generalizing to unseen domains really challenging.
4.2 Sensor-basee human activity recognition
Datasets and implementation details We also evaluate our method on a cross-datasel DG benchmark with four different datasets on human activity recognition (HAR). The four datasets are: UCT daily and sports dataset (DSADS) (Barshan & Yüksek] 2014) consists of 19 activities collected from 8 subjects wearing body-worn sensors 5 body parts. And it contains about 1140000 samples with three sensors. USC-HAD (Zhang & Sawchuk 2012). USC-SIPT human activity dataset (USC-HAD) composes of 14 subjects (7 male, 7 female, aged from 21 to 49) executing 12 activitios with a sensor tied on the front right hip. And it contains 5441000 samples with two sensors. UCI-HAR (Anguila et al. 2012). UCI human activity recognition using smartphones data set (UCI-HAR) is collected by 30 subjects performing 6 daily living activitics with a waist-mounted smartphone. And it contains 1310000 samples with two sensors. PAMAP2 (Reiss & Stricker 2012). PAMAP2 physical activity monitoring dataset (PAMAP2) contains data of 18 different physical activities, performed by 9 subjects wearing 3 sensors. And it has 3850505 samples with three sensors.
To evaluate our method thoroughly for time-series data, we design three settings: Person, Cross- Position, and Datssea generalization with these four datasets.:
Cross-Person Generalzation. We split DSADS, USC-HAD, and PAMAP For each dataset, we split data into four parts according to the persons and utilize 0,1,2, and 3 to denote four domains. For example, 14 persons in USC-HAD are divided into four groups, [1, 0), (,, 9, [7, [7,13,,, 13, 10), [4, 12] where different numbers denole different persons. We try our best make each domain have a similar number of data.
Cross-Position Generalization. We choose DSADS since it contains sensors worn on five different positions. Therefore, we split DSADS into five domains according to sensor positions.
Cross-Dataset Generalization. To perform Cross-dataset generalization for HAR, we need to unify inputs and labels of each dataset first. Each dataset corresponds to different domain. Six common classes are selected. Two sensors from each dataset that belong to the same position are selected and data is down-sampled to ensure the dimension of data same.
The HAR model contains two blocks. Each has convolution layer, one pool layer, and one batch nor- malization layer. A single-fully-conneetee layers layer serves as the classifier. All methods are implemented with Pororc (Paszke et al. 2019). The maximum training epoch is set to 150. The Adam optimizer with weight decay x 10 used. The learning rate for the rest methods is 10-2, We tune hyperparameters for each method. Moreover, we add another two latest methods, GILE (Qian et al. 2021 and AdaRNN (Dul et al. 2021), ; for comparison in Cross-Person generalization setting.
"The values in parentheses the results reported in other papers. They do not mean anything due different splits, optimizers, settings, etc.
STe baselines for UCI-HAR are good enough, so we da run experiments on it.
8

---

Published in Transactions on Machine Learning Research (07/2022)
Results The results are shown in Table 4 and P for three settings, respectively. Our method achieves the best average performance compared to the other state-of-the-ar methods in all three settings. In the Cross-Person setting, DIFEX has about 2.8%, 5.5%, 1.5% improvement compared to the second-best method for DSADS, USC-HAD, and PAMAP respoctively. DIFEX has an improvement with about 2.3% in the Pross-Position setting while has an improvement with about 7.6% the Cross-Datase setting. The results demonstrate DIFEX has good generalization capability for time-serics classification.
Table 4: Classification accuracy on HAR in Cross- Cross-Perosor Peroson setting, The bold items are the best results.
Dataset
DSADS
USC
PAMAP
Target
AVG
ERM
83.11
79.30
87.85
70.96
80.30
80.98
57.75 .7
74.03
65.86 86
69.66 66
89.98
T8.00 78.08
55.77
84.44
77.07
DANN
89.12
81.17
85.92
83.38
85,
81.22
57.88
76.69
70.72
71.63
82.18
78.00
55.39
87.26
75.
CORAL
90.96
85.83
86.62
78.16
85.39
78. 82
58.93
75.02
53.72
66.62
86.16
77.85
19.00
87.81
75,20
Mixup
89.56
82.19
89.17
86.89
86.95
79.98
64.14
74.32
61.28
69.93
89.44
80. 30.30
58.45
87.68
78.97
GroupDRO
91.75
85.92
87.59
78. 25
85.88
80.12
55.51
74.69
59.97
67.57
85.22
T7.69
56.19
84.95
76.01
RSC
84.91
82.28
86.75
77.72
82.92
81.88
57.94
73.39
65.13
69.59
87.11
76.92
60.26
87.84
78.03
ANDMask
$5.04
75.79
87.02
77.59
81.36
79.98
55.32
74.47
65.04
08.68
86.74
76.41
43.61
85.56
73.09
SWAD
56.56
71.07
85.18
T4.93
GILE
$1.00
75.00
77.00
66.00
7/1.5
78. 00
62.00
77.00
63.00
70.00
83.00
68.00
76.00
67.50
AdaRNN
$0.92
75.48
90.18
75.48
80.52
78.62
55.25
60.89
73.68
68.62
81.64
71.75
15.42
82.71
70.38
DIFEX
94.30
87.24
92.15
85.35
89.76
83.52
69.16
76.45
79.42
77.14
90.65
82.35
59.90
89.25
80.54
Table 5: Classification accuracy DSADS in Cross- Posilion setting.
The bold items are the best results.
Target
0
2
3
4
AVG
ERM
41.52
26.7 73
35.81
21.45
27.
30.56
DANN
45.45
25.36
38.06
28.89
25.05
32.56
CORAL
33.22
25.18
25.81
22.32
20.64
25.43
Mixup
48.77
34.19
37.49
29.50
29.95
35.98
GroupDRO
27.12
26.66
24.34
18.39
24.82
24.27
RSC
46.56
27.37
35.93
27.04
29.82
33.34
ANDMask
47.51
31.06
39.17
30.22
29.90
35.57
SWAD
43.83
30.18
39.35
25.38
27.58
33.26
DIFEX
49.55
32.73
41.75
33.43
34.20
38.33
Table 6: Accuracy on HAR. The bold and underline items are the best and the second-best results.
Source
Target
ERM
DANN
CORAL
Mixup
GroupDRO
RSC
ANDMask
SWAD
DIFEX
USC HAD, UCI HAMAP
DSADS
26.35
29.73
39.46
37.35
51.41
33.10
41.66
28.30
46.90
DSADS,UCIHAIE UCI-HAR,,AMAL
USC-HAD
29.58
45.33
41.82
17.39
36.74
39.70
33,83
40.99
49.28
DSADS,USC-HIS PAMAP
UCI-HAR
41.44
46.06
39.10
40.24
33.20
45.28
43.22
43.94
46.41
DSADS,USC-HAAD UCIIIAR AVG
PAMAP
32.93
43.84
36.61
23.12
33.80
45.94
40.17
16.11
52.72
33.32
41.24
39.25
37.03
38.79
41.01
39.72
32.33
48.83
Moreover, we have some more insightful findings.
Can the simple alignments always bring benefits? Obviously, the answer no. Although CORAL bring improvements compared to ERM for DSADS in the Cross-Person setting and in the Cross-Dataset setting, it performs worse than ERM for the other benchmarks.
Different aligrments bring different effects. seems that DANN performs better than CORAL, which inspires that we can be able to replace CORAL with DANN to capture mutually-invariant features in the future work.
Some methods proposed for visual classification initially also work for time series. From Table and we can see that RSC brings improvements in most circunstances although was initially designed for visual classification, which demonstrates there may exist commonness bet these
9

---

Published in Transactions on Machine Learning Research (07 (07/2022)
two different modalities. However, compared to remarkable improvements for visual classification, the improvements for time-series data are not significant anymore. Tt illustrates that these met may be not general enough.
4. The same method for different DG benchmarks may have completely different performances. ERM, the baseline without any generalized techniques, sometimes performs better than other state-of- the-art methods, c.g., for USC in the Cross-Porson setting while it performs worse than others sometimes. This phenomenon illustrates that domain generalization is a hard task and we need to design different methods for various situations. In this condition, our method that can achieve the best performance almost on all tasks is inspiring.
For data with fewer row channels and for more difficult benchmarks, learning more diverse features brings better improvements. Tn the Cross-Person setting, USC-HAD is the most difficult. benchmark containing only 6 channels. In this situation, how to exploit more diversity and useful features is more important. Thus, our method with both ineernally-ivvaiinn and mutually-invariant features can bring much larger improvements. similar phenomenon exists in the
setting.
4.3 EMG gesture recognition
To further prove the superiority of our methods on time-series data, we also validate our method on a more challenging benchmark, EMG for gestures Data Set Lobov et al. (2018) which contains raw EMG data recorded by MYO Thalmic bracele Electromyogra- phy (EMG) is based on bioclectric signals and is common type of time-series data used in many fields, such as healthcare and en- tertainment. For the dataset, the bracelet is equipped with eight sensors equally spaced around the forearm and eight sensors simul- tancously acquire myographic signals when 36 subjects performed series of static hand gestures. The dataset, contains 40, 000 50, 000 recordings in each column with 7
Table 7: Results EMG in Cross- Person sctting.
Target
AVG
ERM
64.08
58.05
52.51
57.29
57.96
DANN
60.85
70.40
61.06
52.39
61.88
CORAL
62.97
73.70
71.53
70.17
69.59
Mixup
59,21
66.51
61.54
61.62
62.96
GroapDR
63.62
67.
72.19
67.99
67.86
RSC
65.02
75:18
60.86
61.25
67.83
ANDMack
59.53
66.15
71,83
65.09
65.60
SWAD
42.0/2
57.
53.89
48.73
50.42
DIFEX
71.83
82.8
76.91
75.84
76.84
classes and we select 6 common classes for our experiments. We randomly divide 36 subjects into four domains without overlapping and cach domain contains data of 9 persons. The baseline methods and imple- mentation details are similar to those for HAR. Experiments are done in three random trials, thus mitigating influences of unfair splits. FMG data comes from bioelectric signals, and thereby it is affected by many fac- tors, such as environments and devices, which means the same person may generate different sensor data even when performing the same activity with the same device at a different time or with the different devices at the same time. Therefore, this benchmark more challenging.
Table 7 shows that our method achieves the best average performance and is 7.25% better than the second-best method. Just similar to results mentioned above, alignments among domains can bring improvements in most circumstances while only traditional alignments are not enough, which proves the need of both internally-invarien features. Moreover, from the first task in Table 7 we see that sometimes simple domain alignments may deteriorate performance. But ours performs the best on all tasks.
70
65
Accuracy
60
55
features and mutually-invarian
50
45
ERM
RSC
GroupDRO
ANDMasK Ours
Figure 4: Results on USC- HAD for single-person DG.
4.4 Single domain generalization
In this section, we perform single domain generalization to demonstrate superiority of our method.
USC-HAD We randomly choose two subjects from USC-HAD and each subject serves as one domain. As shown in Figure 4 DIFEX achieves the best performance with over 9% improvements compared to four her stale-of-the-art methods. It demonstrates DIFEX can make full use of both internally and mutually domain-invariant features. To sum up, DIFEX is effective in both image and time-series benchmarks with multiple and single domains, indicating that it is a general approach for domain generalization.
6 For more details about FMG, please refer to httpp///ssssseee.....e..eee////s
10

---

Published in Transactions on Machine Learning Research (07 (07/2022)
Raw Data
Fourier amplitude
Fourier phase
Restored data
Figure 5: Fourier features as internally-invariant representations for images.
Table Accuracy on PAGS. The bold and underline items are the best and the second-best results.
Source
Target
ERM
Mixup
GroupDRO
RSC
ANDMask
DIFEX
A
35.50
36.67
38.53
36.57
35.50
46.68
A
C
57.25
58.49
59.94
62.50
57.25
64.46
P
81.86
82.87
84.55
84.49
81.86
86.17
S
31.48
32.25
34.11
33.16
31.48
56.81
AVG
51.52
52.57
51.25
54.18
51.52
63.53
We choose two datasets from PACS each time. We treat one selected datasct as the source domain and the other as the larget. As shown in Table our method achieves the best performance with over 9% improvements on average compared to five latest srate-of-the-ars methods. In this situation, GroupDRO shows its ability of generalization since is designed for out-of-distribution without many source domains while RSC also shows an acceptable performance since it is a general method. The results demonstrate our method can perform well single domain generalization for image classification.
PACS
4.5 Analysis
Ablation Study We perform ablation study in this section.
84
Does it work if using interrallg-nnvariantt or contually-invariar features alone? As is shown in Figure 6 directly utilizing mutually-invariant features brings a slight improvement compared to ERM while only utiliz- ing internally-ivvariant features distilled from a teacher net has the same performance ERM. Combining both two kind of features bring a much better performance. This demonstrates that only pveennlly-invariant or mutually-invarian features may be not discriminative enough.
83
Accuracy
82
81
80
ERM
CORAL
FFT Phase Ours
Figure 6: Ablation study on the PACS dataset.
Do lhre parts of Eq. 7 all work? As shown in Table g without one part still has an improvement compared to ERM. Compared to mutually- invariant features and exploration, internally-invariant fcatures play slightly less important role. And combining three parts can bring the best performance, showing that each part is essential for generalization.
11

---

Published in Transactions on Machine Learning Research (07 (07/2022)
Table 9: Ablation study.
Benchmark
PACS with multiple sources
HAR Croes-Datase
PACS with single source
Target
C
P
S
AVG
D
U
H
P
AVG
A
C
P
AVG
ERM
77.00
74.53
95.51
77.96
81.22
26.35
29.58
11.11
32.93
33.32
35.50
57.25
81.86
31.48
51.52
w./o Intern.
50.66
76.51
95.33
18.19
82.42
46.90
17.26
15.76
51.35
47.82
41.75
63.52
84.91
30.42
60.15
w.fo Mutual.
50.47
75.60
95.33
78.06
82.36
42.61
44.12
45.12
46.49
44.58
43.16
63.01
84.91
41.77
58.21
w.fa Exp.
95.33
$2.55
38.21
48.63
45.88
51.25
45.99
63.99
DIFEX
80.86
77.60
95.57
79.49
83.38
46.90
49.28
46.44
52.72
48.83
46.68
64.46
86.17
56.81
63.53
Motivation Examples for Visual Images We have shown example that some Fourier features be viewed as internally invariily representations for time-series data, and give example for visual data. As shown in Figure 5 Fourier features for images are similar to features for sensors. Two figures in the second column demonstrate that Fourier amplitude may be useless for classification. Two figures in the last column are data restored with only Fourier phase. They illustrate that Fourier phase contains most of classification information since we can vaguely see dogs while still loses some information compared to the original data since the dogs are not clear compared to original raw data (two figures in the first column). [
Parameter Sensitivity There are mainly three hyperparameters in our method: A for distillation knowl- edge from teacher net with Fourier phase information, Xg for the CORAL alignment loss, and X for the exploration loss. We evaluate the parameter sensitivily of our method in Figure where we change one parameter and fix the other to record the results. From these results, we can see that our method achieves better performance near the optimal point, demonstrating that our method is insensitive to some hyperpa- rameter choices to some extent. We also note that A3 for exploration loss bit sensitive which may need attention in real applications
85
86
86
Ours
80
85
ERM
accuracy (%6)
84
75
accuracy
accurat)
70-
82
80
Ours
Ours
ERM
81
ERM
55
78
0.001
0.01
0.1
0.5
1
10
0.01
0.1
0.5
10
0.01
0.1
0.5
Az
Aa
(a) A
(b) Ag
(c)
A
Figure 7: Parameter sensitivity (C-P in PACS for single domain generalization).
5
Conclusion
In this paper, we propose DIFEX to learn both internally-invarian and mutuaaly--nvvriian features for domain generalization. DIFEX utilizes inheritance and exploration framework to combine internally- invariant fcatures distilled from a teacher net with Fourier phase information and mutually-invarian fca- tures obtained from domain aliguments. Extensive experiments on both image and time-series classification demonstrate the superiority of DIFEX.
In the future, we plan to combine more internally-invan features and mutually-invaiin features for better generalization. Moreover, we may utilize normalization techniques or some other distance to guide explorations for more stable optimization.
7 Although restored data 6 have many checkboard artifacts, they have little influence performance. We first utilize teacher network learn uscful information for classification. And then extract internally-invariant representations with the help of the teacher net which has filtered noisy information. In this step, also utilize the classification loss reduce the in influence of useless information.
8 When Ag smaller, e.g. 0, the performance will drop, as shown Table
12

---

Published in Transactions on Machine Learning Research (07 (0722)
6 Acknowledgments
This work is supported by the National Key Research and Development Plan of China (No. 2019YFB1404700), Natural Science Foundation of China (No. 61972383), Science and Technology Ser- vice Network Initiative, Chinese Academy of Sciences (No. 4FL-STTSQQYZD-22221111-0000 Grant Council (RGC) of Hong Kong through Early Career Scheme (ECS) under the Grant 21200522 and CityU Applied Research Grant (ARG) 9667244.
and the Research
References
Davide Anguila, Alessandro Ghio, Luca Onelo, Xavier Parra, and Jorge L Reyes-Ortiz Ortiz. Human activity recognition on smartphones using a multiclass hardware-friendly support vector machine. In International workshop on ambient assisted leing, pp. 216 223. Springer, 2012.
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain generalization using meta-regularizztioon Advances in Neural Informalion Processing Systems, 31:998 1008, 2018.
Billur Barshan and Murat Cihan Yüksek. Recognizing daily and sports activities in two open source machine learning environments using body-worn sensor units. The Computer Journal, (7(11):1649-1660 2014.
Shai Ben- Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, (9(1):151 111151115,,111111111111111111111 2010.
Manh-Ha Bui, Toan Tran, Anh Tran, and Dinh Phung. Exploiting domain-specific features to enhance domain generalization. Advances in Newral Information Processing Systems, 34, 2021.
Junbum Cha, Sanghyuk Chun, Kyungjac Lec, Han- Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrac Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34:22405-222 22418, 2021.
Yuntao Du, Jindong Wang, Wenjie Feng, Sinno Pan, Tao Qin, and Chongjun Wang. Adarnn: Adaptive learning and forccasting of time scries. In ACM International Conference on Knowledge and Information Management (CIKM), 2021.
Yasaman Esfandiari, Sin Yong Tan, Zhanhong Jiang, Aditya Balu, Ethan Herron, Chinmay Hegde, and Soumik Sarkar. Cross-gradient aggregation for decentralized learning from non-iid data. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 Proceedings of Machine Learning Reseurch, pp. 3036 3046. PMLR, 2021.
Chen Fang, Ye Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In Proceedings of the IEEE International Conference Computer Vision, pp. 1657-1664, 2013.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Interna- tional conference on machine learning, pp. 1180-1189. PMLR, 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. comain-adversarial training of neural networks. The journal of machine learning research, 17(1):2096-200 2030, 2016.
Ishaan Gulrajani and David Jopez-Pazz In search of lost domain generalization. In International Conference on Learning Representations, 2021.
Bruce C Hansen and Robert F Hess. Structural sparseness and spatial phase alignment in natural scenes. JOSA A, 24(7):1873 1885, 2007.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 770-778. 2016.
13

---

Published in Transactions on Machine Learning Research (07/2022)
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. In NIPS, 2015.
Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross-domain general- ization. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 124-140. Springer, 2020.
Maximilian ssse, Jakub M Tomezak, Christos Louizos, and Max Welling. Diva: Domain invariant variational autoencoders. In Medical Imaging with Deep Learning, pp. 322 348. PMLR, 2020.
Fredrik D Johansson, David Sontag, and Rajesh Ranganath. Support and invertibility in domain-invariant representations. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 527- 536. PMLR, 2019.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to docu- ment recognition. Proceedings of the IEEE, 86(11):2278 22222222372, 1998.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain gener- alization. In Proceedings of the IEEE international conference on computer vision, pp. 5542 5550, 2017.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta-learning for domain generalization. In Phirty-Second AAAI Conference on Artificial Intelligence, 2018a.
Ya Li, Mingming Gong, Xinmei Tian, Tongliang Liu, and Dacheng Tao. Domain generalization via con- ditional invariant representations. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 32, 2018b.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 624-639, 2018c.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, pp. 6028-6039. PMLR, 2020.
Shiqi Lin, Zhizheng Zhang, Zhipeng Huang, Yan Lu, Cuiling Lan, Peng Chu, Quanzeng You, Jiang Wang, Zicheng Liu, Amey Parulkar, al. Deep frequency filtering for domain generalization. arXio preprinl arX1v:220.112198, 2022.
Chang Liu, Xinwei Sun, Jindong Wang, Haoyue Tang, Tao Li, Tao Qin, Wei Chen, and Tie- Yan Liu. Learning cansal semantic representation for out-of-distribution prediction. In NewrIPS, 2021.
Sergey Lobov, Nadia Krilova, Innokentiy Kastalskiy, Victor Kazantsev, and Valeri A Makarov. Latent factors limiting the performance of semg-interfaces. Sensors, 18(4):1122, 2018.
Wang Lu, Jindong Wang, Yiqiang Chen, Sinno Pan, Chunyu Hu, and Xin Qin. Semantic-iisrrimnatiee mixup for generalizable sensor-based cross-domain activity recognition. Proceedings of the ACM on Inler- active, Mobile, Wearable, and Ubiquitous Technologies (IMWUT), 2022.
Toshihiko Matsuura and Tatsuya Harada. Domain generalization using a mixture of multiple latent domains. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 34, pp. 11749-11756, 2020.
Saeid Motiian, Marco Piccirilli, Donald A Adjeroh, and Gianfranco Doretto. Unified deep supervised domain adaptation and generalization. In Proceedings of the IEEE international conference on computer vision pp. 5715-5725, 2017.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning, pp. 10-18. PMLR, 2013.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
14

---

Published in Transactions on Machine Learning Research (07 (07/2022)
Henri J Nussbaumer. The fast fourier transform. In Fast Fourier Transform and Convolution Algorithms, pp. 80-111. Springer, 1981.
A Oppenheim, Jae Lim, Gary Kopec, and SC Pohlig, Phase in speech and pictures. In ICASSP"79. IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 4, pp. 632-637. IEEE, 1979.
Alan V Oppenheim and Jae Lim. The importance of phase in signals. Proceedings of the IEEE, 69(5): 529 541, 1981.
Giambattista Parascandolo, Alexander Neitz, Antonio Orvieto, Luigi Gresele, and Bernhard Schölkopf. Learning explanations that are hard to vary. In ICLR, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. volume 32, pp. 8026-8037, 2019.
Vihari Piratla; Praneeth Netrapalli, and Sunita. Sarawagi. Efficientt. domain generalization via common- specific low-rank decomposition. In International Conference on Machine Learning, pp. 7728 7738. PMLR, 2020.
Hangwei Qian, Sinno Jialin Pan, Chunyan Miao, H Qian, SJ Pan, and C Miao. Latent independent excitation for generalizable sensor-based cross-person activity recognition. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 35, pp. 11921 11929, 2021.
Mohammad Mahfujur Rahman, Clinton Fookes, Mahsa Baktashmotlagh, and Sridha Sridharan. Correlation- aware adversarial domain adaptation and generalization. Pullern Recognition, 100:107124, 2020.
Attila Reiss and Didier Stricker. Introducing a new benchmarked dataset for activity monitoring. In 2012 16th international symposium on wearable computers, pp. 108-109. TEEE, 2012.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitncts: Hints for thin deep nets. arXiv preprint orXiv:1412.6550, 2014.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang, Distributionally robust neural net- works for group shifts: On the importance regularization for worst-case generalization. In International Conference on Learning Representations (ICLR), 2020.
Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross-gradient training, In International Conference on Learn- ing Representations, 2018.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision, pp. 443 450. Springer, 2016.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schncider, Wojeiech Zaremba, and Picter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RS) international conference on intelligent robots and systems (IROS), pp. 23 30. IEEE, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advonces in newral information processing systems, pp. 5998 6008, 2017.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to unseen domains: A survey on domain generalization. IEEE Transactions on Knowledge and Data Engineering, 2022.
Garrett Wilson and Diane J Cook. A survey of unsupervised deep domain adaptation. ACM Transactions on Intelligent Systems and Technology (TIST), 11(5):1-46, 2020.
15

---

Published in Transactions on Machine Learning Research (07 (2222)
Zhao Xingchen, Liu Chang, Sicilia Anthony, Jac Hwang Scong, and Fu Yun. Test-time fourier style calibration for domain generalization. In International Joinl Conference on Arlificial Intelligence. (IJCAT), 2021.
Hongzuo Xu, Yongjun Wang, Zhiyue Wu, and Yijie Wang. Embedding based complex feature value coupling learning for detecting outliers in non-iid categorical data. In Proceedings of the AAAI Conference Arlificial Intelligence, volume 33, pp. 5541-5548 2019.
Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, and Qi Tian. fourier-based framework for domain generalization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pullern Recognition, pp. 14383 14392, 2021.
Yanchao Yang and Stefano Soatto. Fda: Fourier domain adaptation for semantic segmentation. In Proceedings of the IEEE/CVI Conference on Compuler Vision and Pallern Recognilion, pp. 4085 1085-4095, 4095, 2020.
Yanchao Yang, Dong Lao, Ganesh Sundaramoorthi, and Stefano Soatto. Phase consistent ecological domain adaptation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9011-9020, 2020.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Topez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Mi Zhang and Alexander A Sawchuk. Usc-had: daily activity dataset for ubiquitous activity recognition using wearable sensors. In Proceedings of the 2012 ACM conference on ubiquitous computing, pp. 1036- 1043, 2012.
Han Zhao, Remi Tachet Des Combes, Kun Zhang, and GeolTrey Gordon. On learning invariant representa- tions for domain adaptation. In International Conference on Machine Learning, pp. 2019.
PMLR,
Kecheng Zheng, Yang Cao, Kai Zhu, Ruijing Zhao, and Zueng-Jun Zha. Famlp: A frequency-aware mlp-like architecture for domain generalization. arXiv preprint ar Xiv:2200 1 12899, 2022.
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang, Deep domain-ad image genera- tion for domain generalisation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 13025- 13032, 2020.
A More details about DIFEX
A.1 Novelty
Learning the private and shared reprosentations, or disentanglement, a quite popular research direction for domain adaptation ietiaatiionneeeeeaiaaaiooo However, the novelty is never the disentanglement idea itself, but how can achieve such representation separation. From this point of view, method is novel since it is the first work that splits the representations into internally and mutually invariant aspects. Most of the existing disentanglement works are based on a variational autoencoder structure, which is totally different from ours.
Fourier features are also popular in recent years, but the problem is that the novelty lies how adopt Fourier features for DG. To this end, our approach totally different from existing research:
Please note that Xingchen et al. 2021 is not for a traditional DG setting. The model changes when testing. Xu al. (2021) utilized Fourier amplitude for data augmentation. Our method directly utilizes knowledge distillation to extract internally-invarian features, i.c. Fourier phase information. It is no doubt that ours has totally different implementation.
16

---

Published in Transactions on Machine Learning Research (07 (07/2022)
Applying the high-level Fourier features is only part of our method. It is for intrrially- features. Our method also includes mutually-invariant feature learning. These concepts and the combination are novel. Moreover, we also include exploration. To the best of our knowledge, few methods pay attention to the exploration among different types of features for diversity.
Overall, our method novel enough.
A.2 Compare to other Fourier-based methods, e.g. FACT
There several existing approaches exploiting Fourier-based features for DG, discussed in related work section. More specifically, although FACT Xu et al. (2021) and ours share similar ideas, we are totally different in methodology:
FACT utilizes amplitude Mixup perform data augmentation and teacher-student framework for consistency alignment. It can be seen as an adaptation of the data augmentation technique and the model optimization technique.
Although our DIFEX also tries to utilize Fourier phase information, focus on the feature ex- traction. And our teacher network directly extracts the Fourier phase information for the student network to learn.
Some other mothods utilizing Fourier for domain generalization are mentioned in Section 2.3 of the main paper. However, none of them directly utilizes the Fourier phase to guide the part of feature learning (internally invariant features).
We offer some comparison results between our method and FACT in Figure Obviously, our method achieves the better results, especially in time-seriese benchmarks.
A.3 How the mutually-invariant and the internally-invariant features are extracted.
As shown in Figure this operation is done by directly splitting the last layer of the feature network into two parts: one for mutually-invariiat and the other for internally-invariant features, respectively. This be easily implemented by the neural network design. Then, utilize the learning goals (i.e. distillation, alignment, and exploration) to guide the feature extraction process. The feature extractor network, except for the last layer, is shared by these two parts (the blue block in the figure). The summation operation (i.e., the circled '+' sign) in the figure denotes concatenating two parts. We believe that utilizing independent networks for these two parts respectively can bring better performance. However, it may be unfair since it utilizes larger feature network.
B Implementation details
We try our best to ensure fair comparisons. As mentioned in the main paper, we utilize the same backbone, the same optimizer, the same splits, and some other same tricks, e.g. data transformation (The epoch is so large that each method can reach convergence.). We tune hyperparameters for each method and select the best to compare (we even tune some base hyperparameters, c.g. Ir, for some methods but we report the mainstream of the base hyperparameters.)
Moreover, note that DomainBed is popular benchmark but not the only and totally random parameters, which are not suitable for some methods. Therefore, lots of methods, e.g. FACT, do not utilize it. In addition, utilizing the results reported in the original paper are also not suitable. Different tricks, different data splits, and even different initial states can have large influences on the final results (As shown DomainBed, decline seriously.). We try to get balance between totally random and good performance. We believe ours achieve better performances with more careful tuning. But we focus methods and fairness.
Domainbed utilizes Adam
the totally same and random environment, some methods
17

---

Published in Transactions on Machine Learning Research (07/2022)
Ours
Accuracy (%)
(%)
Accuracy (%)
(a) Digits-DG
(b) PACS
(c) VLCS
FACT
Ours
FACT
Ours
FACT
Ours
Accuracy (%)
Accuracy (%)
Accuracy
AVG
AVG
3
AVG
Task
Task
Task
(d) DSADS
(e) USC-HAD
(I) PAMAP
FACT
Accuracy (%)
Accuracy
AVG
DSADS USC-HADUCHI HAR PAMAP Task
AVG
2
AVG
Tack
Task
(g) Cross-Pasition Position
(h) Cross-Dataset
(i) RMG
Figure Comparison results between our method and FACT.
18

---

Published in Transactions on Machine Learning Research (07 (072222))
C
Limitations
Table 10: Classification accuracy with different exploration distance computation ways.
Dataset
Target
A
P
S
AVG
L2
80.86
77.60
95.57
79.49
83.38
PACS
norm- L1
81.30
78.41
96.17
80.02
83.98
Dataset
Target
1
2
3
AVG
L2
90.65
82.35
59.90
89.25
80.54
PAMAP
norm-L1
90.85
82.51
63.71
89.32
81.60
Dataset
Target
DSADS
USC-HAD
UCI-HAR
PAMAP
AVG
L2
46.90
49.28
46.44
52.72
48.83
norm-L1
56.25
49.36
46.35
56.43
52.10
Dataset
Target
1
2
3
4
AVG
L2
49.55
32.73
41.75
33.43
34.20
38.33
Cross-Position
norm-11
50.87
33.85
43.49
34.71
33.46
39.28
We have mentioned in the main paper that exploring with L2 distance may bring difficulty to optimize (L2 can be infinity). Therefore, we try another computation way, normalization distance (norm-L1),
Leep1 (21, 22) = -1 2
Z2
(9)
21
|22
The results are shown in Table 10 and we can see that norm-L1 can bring improvements, especially for difficult problems, e.g. Cross-Datase! the future, we may try some other distance, such as cosine.
Visualization study
We present some visualizations to show the reasons behind the rationales of our method. As shown Figure we perform t-SNE show the learned features of training data and target data directly. Figure ((a) demonstrates that ERM without any generalization operations leads that the learned features of test data have different distribution from the learned features of training data and target data are hard to classify, Figure 9() and Figure 9(c) illustrate that learning with alignments alleviate the above problem but some classes still suffer from the same issuc, especially for CORAL Since mutually-invarian features are Iearned with the same technique as CORAL, a similar phenomenon exists in Figure ((T) which again emphasizes the necessity the other techniques. And Figure [(e) proves that internally invariant features may make sense for generalization. Figure [[d] demonstrates that our methods can further alleviate distribution shifts compared to traditional methods with domain alignments, and thereby can bring better performance.
Moreover, with the last fully-connect layer, can assign different. weights to internally-invariant features and mutually-in inntually-ivvariant invariant features for better performance in different situations.
9please that method with L.2 has achieved best results compared other methods and normalization techniques mainly for better performance and easier optimization
19

---

Published in Transactions on Machine Learning Research (07/2022)
(a) ERM
(b) DANN
(c) CORAL
(d) Ours
(e) Internally-invariantt features
(f) Mutually-invariant features
Figure 9: Visualization of the t-SNE embeddings of learned feature spaces for PACS with different methods. Different colors correspond to different classes and different shapes correspond to different domains. Circle and plus correspond to test data and training data respectively. Best viewed in color and zoom in.
20


Reviews:
[{"review_id": "Max5gmw8Xx", "reviewer": "Reviewer_Mw6s", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+ Knowledge distillation has not been widely applied to the DG problem. Using the Fourier transformation to extract the high-frequent features seems a straightforward way for internal representation extraction. \n\n+ The overall framework is simple and effective. It has outperformed most of the baseline methods by a considerable margin. There are solid experiments, including multi/single domain DG and EMG tasks with detailed analyses to enrich the content of this paper.\n\n+ This paper is written in clear logic with a comprehensive survey of related literature. Even the most recent works are included in the discussion.\n\nWeaknesses:\n\n- From my point of view, separating the raw representations into domain private/shared ones is not new. Similar ideas have been well explored in domain adaptation/few-shot learning communities. (I have not listed the refs here, but a similar concept is pretty popular.)\n\n- Applying the high-level Fourier features is also not new in the DG area [1, 2]. Despite the differences claimed in the paper, the overall contributions/novelties are marginal.\n\n[1] Test-time Fourier Style Calibration for Domain Generalization. IJCAI 22.\n\n[2] A Fourier-based Framework for Domain Generalization. CVPR 21.\n\n- In some DG scenarios, the proposed DIFEX cannot outperform the previous methods. Some recent techniques, such as [1,2], are suggested to be included for comparison in PACS. And some critical benchmarks, such as Office-home or DomainNet, are missing.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is built upon the assumption that domain-invariant features should be originating both internally and mutually. The internal representation captures the intrinsic/private semantics of the data, and the mutual ones conclude the domain-invariant knowledge. The authors propose a framework DIFEX with the help of knowledge distillation to learn the high-level Fourier phase as internal features and the cross-domain alignment for domain-invariant learning. The proposed model has been evaluated over multiple benchmarks, including Digits, PACS, and VLCS, with competitive performance.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Suggestions:\n\n1. Please consider including more recent methods for comparison. \n\n2. And some critical benchmarks, such as Office-home or DomainNet, are missing. ", "broader_impact_concerns": "I have not seen any concerns about border impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aKv6FIOtAQ", "reviewer": "Reviewer_B7sJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n- The approach is interesting, coupling standard alignment techniques (i.e. CORAL) with Fourier-based ones (e.g. FACT). Exploring potential redundancy between the two is an interesting topic of research that might shed light on how to perform domain invariant learning on DG. \n- The analysis in section 3.2 well motivates the benefits and drawbacks of using only Fourier transformed inputs.\n\n**Weaknesses**:\n1. In the method it is unclear how the mutually invariant and the internally invariant features are extracted. The reason is that while Fig.2 shows two different features (i.e. yellow and green blocks), Fig. 3 shows that there is no particular feature extractor for the internally invariant one. Thus,  in Eq. (6) it is unclear how $\\mathbf{z}_2$ is obtained and how it differs from $\\mathbf{z}_1$, given that they share the same feature extractor. Note also that Fig. 2 hints at the presence of a summation operation between the two, which is not reflected in the training/inference as described in Eq.(7) and Eq.(8). It is of utmost importance that these aspects are revised to avoid ambiguities and make DIFEX clear to the reader.\n1. The approach is very similar in spirit to FACT (Xu et al. 2021) where also a teacher-student framework is proposed for DG. Also in FACT, the teacher receives as input Fourier-transformed images, while the student their original version, and a consistency loss is minimized between their extracted features. While Section 2.3 describes FACT,  it is not mentioned the similarity between FACT and DIFEX, and the former is not reported in the experimental analysis. Section 2.3 should better highlight the differences between DIFEX and FACT (as well as existing approaches using Fourier transform), and the experiments should include this baseline to clearly show the advantages and disadvantages of these two related frameworks. Note that FACT reports an average accuracy on PACS of 84.51, which is slightly higher than the one reported for DIFEX (i.e. 83.38, Table 2).\n2. According to Fig.7c, the lower the weight is given to the exploration and the higher the performance of the model, reducing its practical impact. This contrasts with the message of the title (stressing this component). It would be helpful to report the performance for negative values of $\\lambda_3$, to show whether or not exploring actually may even improve the performance (something suggested by the trend of Fig.7c).\n3. According to section 4.1, the experimental comparisons with previous approaches to image classification are mainly held out by re-training each model with the same set of shared hyperparameters. While this is a good effort toward decoupling optimization from model choices, still each of the previous approaches might need specific hyperparameters due to, e.g. different convergence times, etc. This may lead a reader to wonder whether the experimental comparison is actually fairer in this way or whether the performance of the other approaches may be reduced by this. Given that the purpose of DomainBed (used to implement previous models, as of Section 4) is exactly to provide a fair benchmark of comparison, I believe that a fairer evaluation would have been performed by either i) reporting the same results of previous works as reported in their respective papers (or DomainBed, if available), since this would have ensured tuned hyperparameters per approach; or ii) evaluating DIFEX on the DomainBed benchmark (to ensure the same evaluation ground). While the second option might be expensive in terms of resources, the first one (tuning the hyperparameters per DIFEX using the available validation splits) would still be viable.\n4. In Fig. 7, the numbers are higher than what is reported in the tables (i.e. over 85% vs 83.38% in Table 9). Why is this the case? \n5. From the abstract, it is not trivial to get the meaning of internal and mutual invariance. I would consider revising it to clarify these concepts.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The article presents a new framework for domain generalization (DG) based on feature alignment, named DIFEX. DIFEX is a teacher-student framework, where the teacher receives as input the phase of Fourier transformed images (preserving semantic rather than style information). The student is then trained to extract the same, style-agnostic features, but from non-transformed images.  To overcome a possible deterioration of the extracted features and exploit cross-domain information, an additional alignment loss is applied to the second-order statistics of features across domains (i.e. the CORAL loss of Sun & Saenko 2016), and the distance between features used in the first and second loss terms is maximized to avoid redundancy of the two terms. Experimental results show that the model achieves good results across different DG benchmarks on images and time-series data.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Following on the previous section, I deem as important points for revision:\n1. Improving the clarity of the manuscript for what concerns the definitions (point 6 above) and methodological components (point 1 above).\n2. Expand the comparison with related works (point 2 above) and provide the performance of other approaches when hyperparameters are tuned for their specific design choices, at least when available already in the literature (see point 4 above).\n3. Provide more details/clarifications (point 5) and expand the ablation analysis (point 3) on Fig.7.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TPXvExvUYt", "reviewer": "Reviewer_gsxo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n- This paper tackles an important problem of domain generalization, in which target data is not available for training.\n- The manuscript is well organized.\n- The idea of learning invariant features for both internal and mutual-side is interesting.\n- The method is evaluated on the time-series data.\n\nCons:\n\n- Some methodology details need to be further clarified (see below requested changes)\n- The comparison lacks some strong DG methods, e.g., Mixstyle[1], SWAD[2], etc. Besides, this paper[3] applies Fourier transformation for domain generalization tasks, comparison and analysis with this method would make the experiments more convincing.\n- The experiment results are much lower than the results reported in the DomainBed. And the performance gaps for different methods are inconsistent. Is there any explanation? \n- It is not clear how the proposed method can perform single domain generalization. How is the proposed cross-domain correlation alignment applied? Besides, there are methods specifically designed for the single domain generalization task; the current comparison using methods not specifically designed for the single domain may not be fair.\n\n[1] Zhou, Kaiyang, Yongxin Yang, Yu Qiao, and Tao Xiang. \"Domain generalization with mixstyle.\" ICLR (2021).\n\n[2] Cha, Junbum, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. \"Swad: Domain generalization by seeking flat minima.\" Advances in Neural Information Processing Systems 34 (2021).\n\n[3] Xu, Qinwei, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, and Qi Tian. \"A fourier-based framework for domain generalization.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14383-14392. 2021.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims to solve the domain generalization problem by learning domain-invariant features from two aspects, including using the knowledge distillation with Fourier transform to capture internal invariant features and using correlation alignment to capture mutual invariant features. The proposed method is validated on a time-series dataset and three DG benchmark datasets.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Some more detailed comments are as follows:\n\n- Figure1 gives a toy example on 1-d data, how’s the quality of using phase to reconstruct data on 2-d images? Is there any information loss by solely using the phase data? Besides, the restored data in Fig.6 have many checkboard artifacts, which may introduce noisy patterns. \n\n- The motivation for aligning correlations to learn mutually invariant features are not clear. Why can aligning correlations make the feature mutually invariant? Here needs to be further clarified.\n\n- In Eq. (5), it is not clear how to calculate the covariance matrix C^i.\n\n- In the parameter sensitivity study, the accuracy is high with a small value for \\lambda_3. What is the performance if the \\lambda_3 is smaller than 0.01? ", "broader_impact_concerns": "This paper has no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new domain generalization method that explores internal invariance together with domain invariance. To this end, the proposed method relies on a knowledge distillation framework to capture the high-level Fourier phase as the internally-invariant features and learn cross-domain correlation alignment as the mutually-invariant features. 

The reviewers agree that this paper makes novel contributions to domain generalization. Especially, the internal invariance idea has not been explored in previous works. Reviewers has some concerns on the novelty of some components, such as the Fourier method and the domain-shared and specific separation, and the authors have addressed these concerns in the discussion. Also, the authors also added lots of new experimental studies as suggested by the reviewers. Finally, all the reviewers vote for acceptance of this paper.

I would recommend acceptance of this paper given its novel contribution and solid empirical results (after revision). 
Decision: Accept (poster)