Title: Generalizing to Unseen Domains for Regression
Venue: NeurIPS 2023
Accepted: no
Keywords: domain generalization

regression

meta-learning
Abstract: In the context of classification, domain generalization (DG) aims to predict the labels of unseen target-domain data  using only labeled source-domain data, where the source and target domains usually share the same label set. However, in the context of regression, DG is not well studied in the literature, and the main reason is that the ranges of response variables in two domains are often different, even disjoint under some extreme conditions. In this paper, we systematically investigate domain generalization in the regression setting and propose a weighted meta-learning strategy to obtain optimal initialization across domains to tackle the challenge. Unlike classification, the labels  (responding values) in regression naturally have ordinal relatedness.  The relatedness brings a core challenge in meta-learning for regression: the hard meta-tasks with less ordinal relatedness are under-sampled from training domains. To further address the hard meta-tasks, we adopt the feature discrepancy to calculate the discrepancy between any two domains and take the discrepancy as the importance of meta-tasks in the meta-learning framework.  Extensive regression experiments on the standard benchmark DomainBed demonstrate the superiority of the proposed method.

Abstract

In the context of classification, domain generalization (DG) aims to predict the labels of unseen target-domain data using only labeled source-domain data, where the source and target domains usually share the same label set . However, in the context of regression, DG is not well studied in the literature, and the main reason is that the ranges of response variables in two domains are often different , even disjoint under some extreme conditions. In this paper, we systematically investigate domain generalization in the regression setting and propose a weighted meta-learning strategy to obtain optimal initialization across domains to tackle the challenge. Unlike classification, the labels (responding values) in regression naturally have ordinal relatedness. The relatedness brings a core challenge in meta-learning for regression: the hard meta-tasks with less ordinal relatedness are under-sampled from training domains. To further address the hard meta-tasks, we adopt the feature discrepancy to calculate the discrepancy between any two domains and take the discrepancy as the importance of meta-tasks in the metalearning framework. Extensive regression experiments on the standard benchmark DomainBed demonstrate the superiority of the proposed method.

1 Introduction

Domain generalization (DG) receives increasing attention due to its challenging setting: learning models on source domains and inferring on unseen but related target domains [1, 2]. However, most existing approaches focus on semantically invariant representations for classification, limiting their practical applications to regression tasks. For example, real-world applications often involve predicting the recovery/survival time of patients in clinic or estimating the ages/skeleton joints/gaze direction of humans [3, 4, 5]. These tasks can be grouped into cross-domain regression problems.

In cross-domain regression, the label's marginal distribution shift can differ significantly compared to DG for classification. In DG classification, the shift typically represents variations in class probability densities across domains [6]. In regression, the shift can take on a specific form, e.g., when the responding (regression) interval of the source domain is [0 , 0 . 7] , the shifted responding interval of the target domain can be [0 . 5 , 1] . This type of shift often occurs in regression settings such as predicting unseen ages, depths and rentals. In some cases, these regression intervals even have no overlap. We refer to this particular regression scenario as domain generalization in regression (DGR). Fig. 1 illustrates the differences between imbalanced domain regression and the DGR. Unlike imbalanced regression [7], DGR focuses on exploration or interpolation for regression.

Comparisons to traditional DG. From the perspective of domain generalization, DGR can be 33 viewed as a special generalization case where the target labels are continuous. However, most domain 34 generalization methods are suboptimal for addressing the DGR problem due to the ordinal relatedness 35 of regression labels. For example, feature alignment [8] might be unnecessary and even harmful in our 36 DGRsetting. Assuming that a closer feature discrepancy implies closer predictions, feature alignment 37 methods may cause the model to exclusively map all predictions into one source interval, which 38

(a) Imbalanced Regression

(b) Generalized Regression

Y

∈

[0, 1]

P(Y)

Source

Target

P(Y)

Y

∈

[0, 1]

P(Y)

Source

Y

∈

[0, 0.7]

Target

P(Y)

Y

∈

[0.5, 1]

does not reduce total generalization risks. In addition to feature alignment, feature disentanglement 39 usually disentangles semantically related discriminant representation for classification [9], while 40 overlooking the ordinal relatedness of the target domain. Furthermore, semantic-related discriminant 41 representation might be unnecessary for regression tasks like age estimation. Robust optimization 42 methods [10] can perform moderately distributional exploration, but also lack the ability to tackle 43 ordinal relatedness in regression. 44

45

46

47

48

Comparisons to open-set DG [1, 11]. Open-set DG primarily focuses on classification applications and the ability to detect unknown classes. If open-set DG methods are used to address our problem, they can only identify these samples whose response intervals differ from that of the source domain but cannot obtain their response values.

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

To effectively capture ordinal relations and facilitate modest extrapolation in the DGR problem, we propose a robust optimization algorithm via meta-learning. Meta-learning algorithms, e.g., model agnostic meta-learning (MAML, [12]) have been extensively utilized in traditional domain generalization [13, 14, 15]. In each meta-task, these methods usually sample a support and a query classification task from two distinct domains and optimize the meta-model by a bi-level paradigm. However, this paradigm alone falls short for addressing the complexities of DGR. The task sampling strategy employed in these methods typically follows an implicit assumption, assuming that all training meta-tasks have equal importance [16, 17]. We argue that this implicit assumption no longer holds in our regression setting.

In contrast to classification, regression tasks exhibit ordinal relations between each pair of labels [18]. When considering the label discrepancy between the support and query domains, it is observed that meta-tasks with a larger regression margin are sampled less frequently compared to those with a smaller margin. Additionally, meta-tasks with a larger regression margin tend to be more challenging to optimize within the meta-learning framework. These key factors bring a sampling bias that harder meta-tasks are less sampled from training data. Consequently, the sampling bias makes harder meta-tasks underrepresented in the training data, i.e., the meta-model tends to choose the easier meta-tasks, limiting the exploration and interpolation capabilities of the model. To mitigate this sampling bias, we propose a simple yet effective strategy: assigning higher weights to harder meta-tasks. These weights are computed based on the feature discrepancy between the query and support examples of each meta-task.

In conclusion, we have developed a DGR benchmark that encompasses both overlapping and nonoverlapping labels between the source and target domains. We conduct experiments on three regression tasks, including causality exploration with a toy logic dataset, predicting unseen ages according to face images, and forecasting rental prices across different regions. Our proposed method, named margin-aware meta regression (MAMR), makes the following main contributions:

We investigate generalized regression from the perspective of domain generalization, a previously understudied area with significant practical implications.

To enhance exploration and interpolation capabilities, we introduce a margin-aware metalearning framework that mitigates sampling bias and encourages the model to recognize long-range ordinal relations.

Although our solution achieves considerable improvements regarding baselines, our empirical analyses demonstrate that generalizing to unseen responses is still challenging.

2 Related Work 81

In this section, two related research areas are briefly introduced. One is domain adaptation for ordinal 82 regression and classification, and the other one is generalization for regression. 83

84

2.1 Domain Adaptation for Ordinal Regression and Classification

Domain adaptation aims to migrate the knowledge from a source domain to a target domain, where 85 there may exist a distribution shift between them. Typical domain adaptation methods try to get confi86 dent decision boundaries for classification tasks based on clustering assumption [19]. However, when 87 it comes to cross-domain regression (also known as ordinal classification [18]), these assumptions 88 are not satisfied, posing challenges for existing domain adaptation methods. Some pioneer works 89 like [20] try to provide regression discrepancy in reproducing kernel Hilbert space. Most recent 90 works address cross-domain regression in specific application scenarios, such as estimating object 91 boxes in cross-domain/few-shot object detection [21, 17], regressing human skeleton key-points in 92 cross-domain gesture estimation [4] and calculating the gaze direction in cross-domain gaze tracing 93 [22]. Furthermore, [3] proposes a general cross-domain regression method via subspace alignment, 94 which reduces domain gap by minimizing representation subspace distance (RSD) with the principal 95 angles of representation matrices. [23] proposes an adversarial dual regressor to achieve a direct 96 alignment between two domains. 97

98

99

100

101

102

103

104

105

106

107

108

109

However, nearly all cross-domain regression methods inherently assume there only exists covariate shift in input examples, i.e., p ( x s ) = p ( x t ) , where p ( · ) is the probability density function and x s , x t denote the source and target examples. This assumption implies that these methods may not be capable of handling label shift across domains. The label shift in cross-domain regression can arise as interval shift of responding values, e.g., the source interval y s ∈ [0 . 3 , 0 . 5] while the target interval y t ∈ [0 . 6 , 0 . 7] . The responding values in the real world can be gasoline consumption data and vary significantly across developed and developing countries [24]. [25] also considers the interval shift problem and tries to learn a ranking on the target domain, followed by mapping the ranking to responding values. This method assumes the availability of the responding interval on the target domain at the adaptation stage, which might be contradictory to the setting of unavailable labels. In contrast, we assume all target domain data are not available at the training stage, which is more practical and challenging in real-world scenarios.

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

̸

2.2 Generalization/Causality for Regression

Domain generalization introduces a more challenging setting where the model can only access the labeled source data at the training stage [1, 2, 26, 27, 28, 29, 30, 31]. A thorough discussion of domain generalization might exceed the scope of our paper. We focus on potential methods that can be applied to regression settings. Among existing generalization methods, some works try to generalize to continuous outputs by capturing causal relations [32, 33]. Recent works like DDG [9] concentrate on capturing invariant semantic features, which might overlook the variational features for continuous predictions. In contrast, the meta-learning paradigm holds potential for regression settings due to its model-agnostic property and strong generalization ability.

The spearhead work MLDG [13] introduces MAML [12] into the domain generalization framework. [14] leverages class relationships and local sample clustering to capture the semantic features of different classes. These two operations are hard to be migrated to regression settings because the clustering assumption is usually not reasonable for regression. Moreover, in many regression tasks like age estimation, the semantic features might be unimportant, e.g., distinguishing each face might be useless for age regression. Instead, the style features, like the texture of the faces might be important information for age regression. Moreover, [30] proposes an implicit gradient to get stable meta-learning loss, which may provide orthogonal solution compared to our method.

3 Problem Setting and Notations

In this section, we introduce the formal definition of the DGR problem. We denote the input 128 space and the label space by X and Y , where Y has a continuous range from 0 to 1 and can 129 include two sub-spaces, e.g., Y source and Y target . D s = { ( x , y ) ∈ {X × Y source }} and D t = 130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

{ ( x , y ) ∈ {X × Y target }} respectively denote the source and target domain data. The model can only utilize D s at the training stage, and then predicts labels in D t without further adaptation. The above settings are very similar to the classification tasks of domain generalization. But the label spaces across domains are different in our regression setting. A prediction ˆ y from regression model R can be denoted with ˆ y = R ( x ) = G ( F ( x )) . We use F : X → Z to denote a feature encoder, where Z is a feature space. After the encoder, we use a linear regressor with sigmoid activation to map the range of predictions into [0 , 1] , i.e., G : Z → Y .

4 Margin-Aware Meta Regression

4.1 Distribution Alignment Produces Regression Margin

Following the typical setting of domain generalization that domain labels are available. We split D s into K source domains { D 1 , D 2 , · · · , D K } and simulate the generalization setting between D s and D t . As we know, feature alignment is the core idea of many typical domain alignment solutions for domain adaptation [34] as well as domain generalization [8]. For domain generalization, the alignment is usually performed among multiple source domains to find domain-invariant semantic features. This alignment can be formalized using a general discrepancy measure, i.e., integral probability metric (IPM, [35]). Let X 1 , X 2 denote two independent random variables from domain distributions P i and P j . The domain discrepancy can be defined with:

where E denotes the expectation, f denotes the transformation function in function space H . Applying specific condition on H , IPM can be transformed into many popular measures, such as maximum mean discrepancy (MMD, [36]) and wasserstein distance (WD, [37]).

Incorporating the domain discrepancy between P i and P j , the objective of the regressor can be formulated as:

where Θ is model parameter, L Θ ( x , y ) = || R Θ ( x ) -y ) || is the empirical risk and can be the squared loss, ̂ IPM is the estimator from two batch examples x 1 and x 2 . For example, ̂ IPM can be the unbiased U-statistic estimator ̂ MMD 2 u ( x 1 , x 2 ) [36]. In general domain generalization for classification tasks, all terms in the above objective could be minimized. However, our regression setting is like open domain generalization, which learns a model from the source domain and inferences in unseen target domains with novel classes [11]. To regress unseen target values, one strategy is to simulate the setting in the training stage. That means the labels in D i and D j have few or no overlaps. Therefore, when the domain discrepancy ̂ IPM is minimized, there might be only one term minimized between L Θ ( x 1 , y 1 ) and L Θ ( x 2 , y 2 ) . This problem can be formally introduced with the following definition:

Proposition 1 (Regression Margin) . Let ( X 1 , Y 1 ) and ( X 2 , Y 2 ) be the random variables corresponding to two source domains D i , D j , the [ a, b ] and [ c, d ] be the regression interval of Y 1 , Y 2 . When ̂ IPM is reduced to 0 for a function f , we have

The regression margin represents the minimal margin (or difference) between errors in the two 165 domains (i.e., Eq. (3)). Eq. (4) is the rearrangement of Eq. (3). In Eq. (4), because ̂ IPM is reduced 166 to 0 for the function f , E [ f ( X 1 )] -E [ f ( X 2 )] = 0 , then obtaining the Eq. (5). The above analysis 167 suggests that a large domain margin M i,j can lead to a divergent optimization when simultaneously 168 minimizing the domain discrepancy and the empirical risks. One strategy is to bypass explicit feature 169 alignment. For example, in the meta-learning paradigm towards domain generalization, one can learn 170 a meta-model by a bi-level optimization. In the inner optimization, the model learns on a support 171 (source) domain. In the outer optimization, the learned model tries to generalize to a query (target) 172

domain. This training strategy naturally avoids explicit feature alignment. Moreover, the bi-level 173 optimization emphasizes the importance of query loss, which might alleviate the above regression 174 margin because the inner model and outer model can be viewed as different sampling instances in 175 parameter space. 176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

4.2 Regression Margin Leads to Sampling Bias in Meta-Learning

Existing meta-learning domain generalization methods are sub-optimal for the DGR problem. In the classification, each meta-task consisting of support tasks and query tasks is assumed to have the same sampling probability. However, the responding intervals of the support and query have ordinal relations in regression. When the regression margin between the support and query tasks is larger, the sampling probability is smaller. The left part of Fig. 2 depicts the relationship between the regression margin and the sampling strategies of meta-tasks. Intuitively thinking about the extreme case that when the regression margin is close to 1 , the corresponding sampling probability of meta-tasks is close to 0 . We formalize this using a simple theorem:

Theorem 1 (Sampling Bias in Meta-Learning) . Given a support domain i , let S ( j | i ) denote the number of available query domain j that can be sampled. Let M 1 i,j , M 2 i,j denote the regression margin of the meta-task 1 and meta-task 2. if M 1 i,j > M 2 i,j , then S 1 ( j | i ) < S 2 ( j | i ) .

The intuitive explanation is: the number of sampling strategies of a larger regression margin meta-task is always less than a small margin meta-task. We will provide a simple and intuitive proof below.

Proof. Following the previous description, the source data D s can be sorted into K disjoint source domains { D 1 , D 2 , · · · , D K } according to their regression interval. The query and support tasks are sampled from D i , D j with regression interval [ a, b ] and [ c, d ] respectively. Let ∆ denote the length of single regression interval, n = M i,j ∆ denote the number of spanning intervals of regression margin M i,j . Given a support task on domain index i , the query tasks on j -th domain have S ( j | i ) choices:

From the above equation, when the regression margin M i,j increases (i.e., n is increasing), the number of available-to-sample query tasks decreases, leading to a smaller number of eligible meta-tasks.

4.3 Margin-Aware Meta-Training

As illustrated by the left part of Fig. 2, a larger regression margin between the support and query tasks usually means a harder meta-task. Therefore, without any specialized sampling strategy, the meta model is prone to be biased towards the small margin tasks. To alleviate this issue, we want the large margin meta-task to have a larger weight in the meta-learning process. One direct strategy is to calculate the weight using the domain discrepancy, i.e., a larger regression margin means a larger meta-task weight. The learning objective can be redefined with:

where D i , D j respectively denote the query domain and the support domain, d is discrepancy functions like ̂ MMD 2 u ( · , · ) or simple Euclidean metric, and β is the inner loop learning rate on the support domain { x s , y s } .

The graphical training process of one meta-task can be seen in the right part of Fig. 2. Different from existing meta-learning models, our MAMR model considers the domain discrepancy by discrepancy function d ( · ) , but the data node in d ( x s , x q ) does not have gradients. The reason is directly minimizing this domain discrepancy might harm the generalization ability of our MAMR model. Our task weighting method is similar to recent sharpness-aware minimization [38], which simultaneously minimizes loss value and loss sharpness. The related topic can also have an extension to penalizing gradient norm [39] and independence-driven importance weighting [40]. With Euclidean distance d ( · ) , we describe the detailed method in Algorithm 1.

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

Query Set

Support Set

Query Set

Meta-Task 2

Meta-Task 1

Y

Inner

Outer Gradients

Task Model

Data Flow

Algorithm 1 Training Algorithm of MAMR

Input : The source domains data D s , the inner loop learning rate β , the out-loop learning rate α , the domain number K to split D s , model parameters Θ .

Output : The learned Θ .

1: Split the source data D s into sub-domains { D 1 , D 2 , · · · D K } .

2: while not convergence do

3: Sample T = K ( K -1) / 2 domain pairs { ( D i , D j ) } that i = j .

4: for index = 0 → T do

5: Sample a batch of support data ( x s , y s ) ∈ D j and query data ( x q , y q ) ∈ D i ;

6: Compute task discrepancies: d ( x s , x q ) = || F ( x s ) -F ( x q ) || 2 ;

7: Get task-specific model parameters: Θ ′ = Θ -β ∇ Θ [ L Θ ( x s , y s ] ;

8: Compute the weighted regression error: L Θ ′ ( x q , y q ) · d ( x s , x q ) ;

10: end for

9: Update Θ : Θ = Θ -α ∇ Θ [ L Θ ′ ( x q , y q ) · d ( x s , x q )] ;

11: end while

5 Experiments

In this section, we will empirically explore what MAMR can learn and compare it to related works from the view of performance and methodology, including introductions to baselines and experimental details, results on three datasets, and detailed analyses.

5.1 Baselines

We use multiple domain generalization and the variants of domain adaptation methods as baselines, including: (1) risk minimization methods ( ERM [41], IRM [42]); (2) feature alignments and robust optimization ( MMD [8], CORAL [43], DANN [34], SD [44], Transfer [45]), MODE [10]; (3) subspace alignments ( RSD [3]); (4) self-supervised and data augmentation methods ( SelfReg [46], CAD [47], MTL [48]) (5) meta-learning ( MLDG [13]) and (6) disentanglement and causality method ( DDG [9], CausIRL [49]). All the introductions of baselines can be seen in Appendix A.

5.2 Training and Evaluation

To ensure fairness and comparability, we put all the baselines into a public evaluation benchmark DomainBed [50]. For age regression, we uniformly use ResNet12 as the backbone encoder F for all methods. ResNet12 is a popular encoder in meta-learning for few-shot learning. For rental regression, we uniformly use a 5-layer MLP as the backbone encoder F . For regressor G , we use a single linear neural network followed by a sigmoid function. Note that all labels are normalized from 0 to 1 . Including toy experiments, all methods are implemented with Pytorch and can be executed on an NVIDIA RTX 3090 GPU. Appendix B provides detailed settings of the hyper-parameters, such as the learning rates, the training seeds, etc.

5.3 Toy Causality Dataset and Results

To figure out what the MAMR model can learn in regression problems, we create a toy dataset in which the input examples and their responding values obey some causal mechanism. We assume the

̸

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

X

1

0.6

0.7

0.8

0.9

1.0

X

2

0.6

0.7

0.8

0.9

1.0

Y

0.7

0.8

0.9

1.0

(a) ERM (MSE: 0.0219)

X

1

0.6

0.7

0.8

0.9

1.0

X

2

0.6

0.7

0.8

0.9

1.0

Y

0.7

0.8

0.9

1.0

(b) RSD (MSE:0.008)

X

1

0.6

0.7

0.8

0.9

1.0

X

2

0.6

0.7

0.8

0.9

1.0

Y

0.7

0.8

0.9

1.0

(c) Ours (MSE:0.0004)

1-dimensional random variables X 1 and X 2 follow a uniform distribution in [0,1], and the responding values Y are under the control of X 1 and X 2 . The control mechanism can be complex as given in Appendix C. At training stage, regression models can only use X 1 ∈ [0 , 0 . 6] and X 2 ∈ [0 , 0 . 6] . At the test stage, we record the regression values when given X 1 ∈ [0 . 6 , 1] and X 2 ∈ [0 . 6 , 1] .

The toy experiments sample 15000 and 10000 regression tasks at the training and test stage, respectively. We use a 4-layers fully connected neural network for ERM, RSD and our MAMR. Fig. 3 provides the test time explorations results of the three methods. On 10000 test tasks, the ground-truth responding values and the predicted values respectively form a gray region and a blue region. When given unseen values of X 1 and X 2 , ERM fails to use the causal mechanism. The strong baseline method RSD captures a part of the causal mechanism. MAMR gets the best exploration performance by maximum causal discovery.

5.4 Cross-Domain Age Estimation Datasets

Perfect age estimation is based on the assumption that all age data are available, while many realworld datasets are not perfect and have partial ages due to privacy concerns. Hence age estimation has been introduced in cross-domain works [18, 51].

CACD 1 . Cross-Age Celebrity Dataset (CACD) contains 163,446 images from 2,000 celebrities collected from the Internet. The age of celebrities ranges from 16-62 and can be classified into 5 disjoint age intervals (domains), i.e., [15 -20) , [20 -30) , [30 -40) , [40 -50) , [50 -60] . The images of each celebrity are sampled by different devices across multiple years. Therefore each domain has different facial characteristics. To consider the overlapped intervals, we further create CACD-O dataset, where each interval has 3 ages of neighbors, e.g., [15 -20) includes 8 different ages from 15 to 22 and [20 -30) has 15 ages from 18 to 32 .

AFAD 2 . The Asian Face Age Dataset (AFAD) originally is an age estimation dataset containing more than 160K face images and aging labels. We split the dataset into 5 age intervals (domains), i.e., [15 -20) , [20 -25) , [25 -30) , [30 -35) , [35 -40] . Like CACD, each age interval has its own face characteristics and can be viewed as 5 related domains for regression.

In each task, only one domain is viewed as the target domain, and the left is viewed as sources. Please refer to Appendix E for more details on these age estimation datasets.

5.5 Cross-Domain Rental Prediction Dataset

The Rental dataset 3 was released by an online competition in 2019 to predict housing rental in Shang 268 Hai, China. The data categories include rental housing, regions, second-hand housing, supporting 269 facilities, new houses, land, population, customers, real rent, etc. We split 15 regions into 4 groups as 270 4 different domains. Each domain has different rentals due to its population and economic conditions. 271 Please refer to Appendix D for more introduction to this dataset. 272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

5.6 Quantitative Comparisons

Comparison to risk minimization methods. ERM and IRM are typical risk minimization methods. From Tab. 1, we find that ERM is better than IRM, which might imply that the gradient invariance in IRM is useless for our problem. Another result is that the naive ERM is surprisingly comparable with advanced methods, e.g., MMD, DANN and MLDG. Even on AFAD dataset, ERM is a very strong baseline. Previous works [50, 52] also find a similar phenomenon in classification tasks.

Comparison to the methods using feature alignments and robust optimization . As discussed in Sec. 4, directly using feature alignments, e.g., MMD, DANN and CORAL, may perform poorly due to the regression margin. Furthermore, DANN and Transfer try to apply adversarial robustness, and MODE uses style augmentation for distribution robustness. Our results demonstrate the robustness design in these methods might bring the opposite impact on ordinal predictions.

Comparison to subspace alignments , e.g., RSD. We find that RSD gets comparable performance with respect to feature alignment methods. With principal angle alignment between sub-spaces, the sub-space alignments effectively slack the traditional feature alignments. This might imply that the domain adaptation method RSD can also generalize to out-of-distribution data.

Comparison to self-supervised and data augmentation methods , e.g., SelfReg. The self-supervised methods, especially with contrastive learning, can be strong baselines for our problem. The reason might be that SelfReg uses strong data augmentation and mixup operation in their models. We find the follow-up work CAD does not surpass SelfReg. The reason might be that the part of marginal distribution alignment in CAD harms the generalization ability like DANN. MTL augments the original feature space with the marginal distribution of feature vectors. However, MTL performs poorly in our regression settings. The reason might be augmenting the original feature space destroys the ordinal information of features.

Comparison to meta-learning method . MLDG simultaneously optimizes the support risks and query risks. While in DGR, the support and the query tasks usually change a lot, which makes the MLDG hard to be optimized. Our method does not simultaneously optimize the two risks and is attentive to hard tasks. The experiments also demonstrate that our method outperforms MLDG.

Comparison to disentanglement/causality . DDGdisentangles the latent representations into semantic 300 features and variation features. DDG may capture the causal mechanism between the inputs and their 301 responding values. However, our further experiments with CausIRL method demonstrate that DDG 302 can collapse with generated variational samples. DDG is originally proposed to minimize the semantic 303 difference among generated samples from the same class while diversifying the variation across 304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

Methods

[15-20)

[20-30)

[30-40)

[40-50)

[50-60]

Avg

MAMR-

0

.

0348

±

0

.

01

0

.

0284

±

0

.

01

0

.

0015

±

0

.

00

0

.

0156

±

0

.

01

0

.

0235

±

0

.

01

0

.

0208

MAMR-G

0

.

0475

±

0

.

00

0

.

0505

±

0

.

03

0

.

0248

±

0

.

02

0

.

0431

±

0

.

02

0

.

0754

±

0

.

04

0.0483

MAMR-P

0

.

0331

±

0

.

01

0

.

0143

±

0

.

00

0

.

0021

±

0

.

00

0

.

0078

±

0

.

00

0

.

0371

±

0

.

01

0

.

0189

0.1

0.2

0.3

0.4

Regression Margin

0.015

0.020

0.025

0.030

0.035

0.040

MSE losses

MAMR-

MAMR

(a)

0.1

0.05

0.01

0.005

0.001

1

2

3

4

5

Inner Iterations

0.00741

0.00251

0.00325

0.00668

0.00746

0.00741

0.00178

0.00283

0.00665

0.00746

0.00741

0.00131

0.00517

0.00736

0.00746

0.00741

0.00207

0.00744

0.00746

0.00746

0.00741

0.00181

0.00651

0.00745

0.00746

0.0000

0.0005

0.0010

0.0015

0.0020

0.0025

0.0030

(b)

0.1

0.05

0.01

0.005

0.001

1

2

3

4

5

Inner Iterations

0.00188

0.00157

0.00121

0.00119

0.00139

0.00188

0.00492

0.00122

0.00119

0.00140

0.00188

0.00453

0.00128

0.00121

0.00142

0.00188

0.00146

0.00124

0.00121

0.00141

0.00188

0.00284

0.00130

0.00120

0.00139

0.0000

0.0005

0.0010

0.0015

0.0020

0.0025

0.0030

(c)

source domains. This design may let DDG overlook the variation features, which are coincidentally important in regression setting. Instead, CausIRL captures the style variables and finds sufficient conditions that do not rely on source domains.

5.7 Detailed Analyses

Tab. 2 provides 3 ablation models. MAMR- is our method without the margin-aware weighting mechanism. MAMR-G computes a mean weight for query tasks using the MMD with Gaussian kernel. MAMR-P computes the pair-wised Euclidean distances among the support and query tasks and provides a weight for each query task. We encourage MAMR-P to perform long range exploration by our proposed margin-aware weighting, which helps achieve better average regression performance. Besides that, the results demonstrate the averaged weight in MAMR-G may be invalid compared to pair-wised weights. The pair-wised Euclidean distances can be viewed as a special case of optimal transport distances [53] between the query data points and the support data points. Furthermore, Fig. 4(a) provides the regression performances of MAMR- and MAMR-P (MAMR). When manually enlarging the regression margin on the CACD dataset, MAMR consistently demonstrates better performance and smaller variance. Note that we set 0 . 1 as the start regression margin between the domain [20 , 30) and [30 , 40) in CACD.

The key hyper-parameters of the MAMR model include the inner loop learning rate β , the outer loop learning rate α and the iteration steps of the inner loop. To reduce the search of hyper-parameters, we set α = 0 . 1 ∗ β . We conduct a grid search for β and the iteration steps. Fig. 4(b) and Fig. 4(c) provide the MSE heatmaps on the CACD dataset using two generalization tasks. We find that more inner iteration steps do not have a significant influence on the generalization results. This phenomenon is consistent with our analysis of the method: different from 5 or 10 inner steps in meta-learning for few-shot learning, fast adaptation by multi-steps is not necessary for DGR.

6 Conclusion and Limitations

We investigate domain generalization for ordinal regression problems. A margin-aware meta-learning 329 regression method is proposed to achieve long-range exploration and interpolation. We build a 330 regression benchmark to systematically investigate the performance of existing domain generalization 331 methods for regression. Limitations: (1) Our empirical analyses demonstrate that domain generaliza332 tion for regression still has a large exploration space when dealing with high-dimensional data. (2) 333 Initial calculation of representation distance in meta-space is not reliable, one strategy is to consider 334 a suitable warm-up strategy. (3) Finally, most used datasets have balanced source labels, applying 335 MAMRto imbalanced source domains is also a more practical setting. 336

337

338

339

340

341

342

References

K. Zhou, Z. Liu, Y. Qiao, T. Xiang, and C. C. Loy, 'Domain generalization: A survey,' IEEE Transactions on Pattern Analysis and Machine Intelligence , pp. 1-20, 2022. 1, 2, 3

J. Wang, C. Lan, C. Liu, Y. Ouyang, T. Qin, W. Lu, Y. Chen, W. Zeng, and P. Yu, 'Generalizing to unseen domains: A survey on domain generalization,' IEEE Transactions on Knowledge and Data Engineering , pp. 1-1, 2022. 1, 3

X. Chen, S. Wang, J. Wang, and M. Long, 'Representation subspace distance for domain adaptation 343 regression,' in Proceedings of the 38th International Conference on Machine Learning (M. Meila and 344 T. Zhang, eds.), vol. 139 of Proceedings of Machine Learning Research , pp. 1749-1759, PMLR, 18-24 Jul 345 2021. 1, 3, 6, 8 346

J. Jiang, Y . Ji, X. Wang, Y . Liu, J. Wang, and M. Long, 'Regressive domain adaptation for unsupervised 347 keypoint detection,' in IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, 348 June 19-25, 2021 , pp. 6780-6789, Computer Vision Foundation / IEEE, 2021. 1, 3 349

350

351

352

Y. Wang, Y. Jiang, J. Li, B. Ni, W. Dai, C. Li, H. Xiong, and T. Li, 'Contrastive regression for domain adaptation on gaze estimation,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 19376-19385, June 2022. 1

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

X. Liu, Z. Guo, S. Li, F. Xing, J. You, C.-C. J. Kuo, G. El Fakhri, and J. Woo, 'Adversarial unsupervised domain adaptation with conditional and label shift: Infer, align and iterate,' in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pp. 10367-10376, October 2021. 1

Y. Yang, K. Zha, Y.-C. Chen, H. Wang, and D. Katabi, 'Delving into deep imbalanced regression,' in International Conference on Machine Learning (ICML) , 2021. 1

H. Li, S. J. Pan, S. Wang, and A. C. Kot, 'Domain generalization with adversarial feature learning,' in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 5400-5409, 2018. 1, 4, 6, 8

H. Zhang, Y.-F. Zhang, W. Liu, A. Weller, B. Schölkopf, and E. P. Xing, 'Towards principled disentanglement for domain generalization,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 8024-8034, 2022. 2, 3, 6, 8

R. Dai, Y. Zhang, Z. Fang, B. Han, and X. Tian, 'Moderately distributional exploration for domain generalization,' in Proceedings of the 40th International Conference on Machine Learning , 2023. 2, 6, 8

Y. Shu, Z. Cao, C. Wang, J. Wang, and M. Long, 'Open domain generalization with domain-augmented meta-learning,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 9624-9633, 2021. 2, 4

C. Finn, P. Abbeel, and S. Levine, 'Model-agnostic meta-learning for fast adaptation of deep networks,' in Proceedings of the 34th International Conference on Machine Learning - Volume 70 , ICML'17, p. 1126-1135, JMLR.org, 2017. 2, 3

D. Li, Y. Yang, Y.-Z. Song, and T. Hospedales, 'Learning to generalize: Meta-learning for domain generalization,' in AAAI Conference on Artificial Intelligence , 2018. 2, 3, 6, 8

Q. Dou, D. Coelho de Castro, K. Kamnitsas, and B. Glocker, 'Domain generalization via model-agnostic learning of semantic features,' in Advances in Neural Information Processing Systems (H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, eds.), vol. 32, Curran Associates, Inc., 2019. 2, 3

Y. Du, X. Zhen, L. Shao, and C. G. M. Snoek, 'Metanorm: Learning to normalize few-shot batches across domains,' in International Conference on Learning Representations , 2021. 2

H. Yao, Y. Wang, Y. Wei, P. Zhao, M. Mahdavi, D. Lian, and C. Finn, 'Meta-learning with an adaptive task scheduler,' in Advances in Neural Information Processing Systems (M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, eds.), vol. 34, pp. 7497-7509, Curran Associates, Inc., 2021. 2

N. Gao, H. Ziesche, N. A. Vien, M. Volpp, and G. Neumann, 'What matters for meta-learning vision regression tasks?,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 14776-14786, June 2022. 2, 3

X. Liu, S. Li, Y. Ge, P. Ye, J. You, and J. Lu, 'Recursively conditional gaussian for ordinal unsupervised domain adaptation,' in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pp. 764-773, October 2021. 2, 3, 7

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

J. Liang, D. Hu, and J. Feng, 'Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation,' in International Conference on Machine Learning (ICML) , pp. 60286039, 2020. 3

C. Cortes and M. Mohri, 'Domain adaptation in regression,' in Algorithmic Learning Theory , (Berlin, Heidelberg), pp. 308-323, Springer Berlin Heidelberg, 2011. 3

Y. Zheng, D. Huang, S. Liu, and Y. Wang, 'Cross-domain object detection through coarse-to-fine feature adaptation,' in 2020 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2020. 3

Y. Bao, Y. Liu, H. Wang, and F. Lu, 'Generalizing gaze estimation with rotation consistency,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 4207-4216, June 2022. 3

H. Xia, P. Wang, T. Koike-Akino, Y. Wang, P. Orlik, and Z. Ding, 'Adversarial bi-regressor network for domain adaptive regression,' in Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22 (L. D. Raedt, ed.), pp. 3608-3614, International Joint Conferences on Artificial Intelligence Organization, 7 2022. Main Track. 3

T. Teshima, I. Sato, and M. Sugiyama, 'Few-shot domain adaptation by causal mechanism transfer,' in Proceedings of the 37th International Conference on Machine Learning , ICML'20, JMLR.org, 2020. 3

B. Chidlovskii, A. Sadek, and C. Wolf, 'Universal domain adaptation in ordinal regression,' 2021. 3

S. Saengkyongam, L. Henckel, N. Pfister, and J. Peters, 'Exploiting independent instruments: Identification and distribution generalization,' in Proceedings of the 39th International Conference on Machine Learning (K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, eds.), vol. 162 of Proceedings of Machine Learning Research , pp. 18935-18958, PMLR, 17-23 Jul 2022. 3

A. Rame, C. Dancette, and M. Cord, 'Fishr: Invariant gradient variances for out-of-distribution generalization,' in Proceedings of the 39th International Conference on Machine Learning (K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, eds.), vol. 162 of Proceedings of Machine Learning Research , pp. 18347-18377, PMLR, 17-23 Jul 2022. 3

M. G. Weber, L. Li, B. Wang, Z. Zhao, B. Li, and C. Zhang, 'Certifying out-of-domain generalization for blackbox functions,' in Proceedings of the 39th International Conference on Machine Learning (K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, eds.), vol. 162 of Proceedings of Machine Learning Research , pp. 23527-23548, PMLR, 17-23 Jul 2022. 3

H. Wang, H. Si, B. Li, and H. Zhao, 'Provable domain generalization via invariant-feature subspace recovery,' in Proceedings of the 39th International Conference on Machine Learning (K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, eds.), vol. 162 of Proceedings of Machine Learning Research , pp. 23018-23033, PMLR, 17-23 Jul 2022. 3

B. Gao, H. Gouk, Y. Yang, and T. Hospedales, 'Loss function learning for domain generalization by implicit gradient,' in Proceedings of the 39th International Conference on Machine Learning (K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, eds.), vol. 162 of Proceedings of Machine Learning Research , pp. 7002-7016, PMLR, 17-23 Jul 2022. 3

X. Chu, Y. Jin, W. Zhu, Y. Wang, X. Wang, S. Zhang, and H. Mei, 'DNA: Domain generalization with diversified neural averaging,' in Proceedings of the 39th International Conference on Machine Learning (K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, eds.), vol. 162 of Proceedings of Machine Learning Research , pp. 4010-4034, PMLR, 17-23 Jul 2022. 3

J. Peters, P. Bühlmann, and N. Meinshausen, 'Causal inference using invariant prediction: identification and confidence intervals,' Journal of the Royal Statistical Society, Series B (Statistical Methodology) , vol. 78, no. 5, pp. 947-1012, 2016. (with discussion). 3

D. Rothenhäusler, N. Meinshausen, P. Bühlmann, and J. Peters, 'Anchor regression: Heterogeneous data meet causality,' Journal of the Royal Statistical Society Series B , vol. 83, pp. 215-246, April 2021. 3

Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. March, and V. Lempitsky, 'Domain-adversarial training of neural networks,' Journal of Machine Learning Research , vol. 17, no. 59, pp. 1-35, 2016. 4, 6, 8

A. Müller, 'Integral probability metrics and their generating classes of functions,' Advances in Applied Probability , vol. 29, no. 2, pp. 429-443, 1997. 4

F. Liu, W. Xu, J. Lu, and D. J. Sutherland, 'Meta two-sample testing: Learning kernels for testing with 439 limited data,' in NeurIPS , 2021. 4 440

J. Shen, Y. Qu, W. Zhang, and Y. Yu, 'Wasserstein distance guided representation learning for domain 441 adaptation,' Proceedings of the AAAI Conference on Artificial Intelligence , vol. 32, Apr. 2018. 4 442

P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur, 'Sharpness-aware minimization for efficiently improv443 ing generalization,' in International Conference on Learning Representations , 2021. 5 444

445

446

447

448

Y. Zhao, H. Zhang, and X. Hu, 'Penalizing gradient norm for efficiently improving generalization in deep learning,' in Proceedings of the 39th International Conference on Machine Learning (K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, eds.), vol. 162 of Proceedings of Machine Learning Research , pp. 26982-26992, PMLR, 17-23 Jul 2022. 5

R. Xu, X. Zhang, Z. Shen, T. Zhang, and P. Cui, 'A theoretical analysis on independence-driven importance 449 weighting for covariate-shift generalization,' in Proceedings of the 39th International Conference on 450 Machine Learning (K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, eds.), vol. 162 451 of Proceedings of Machine Learning Research , pp. 24803-24829, PMLR, 17-23 Jul 2022. 5 452

V. Vapnik., The nature of statistical learning theory . Springer science business media, 1999. 6, 8 453

M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz, 'Invariant risk minimization,' 2019. 6, 8 454

B. Sun and K. Saenko, 'Deep coral: Correlation alignment for deep domain adaptation,' in ECCV 2016 455 Workshops , 2016. 6, 8 456

M. Pezeshki, S.-O. Kaba, Y. Bengio, A. Courville, D. Precup, and G. Lajoie, 'Gradient starvation: A learn457 ing proclivity in neural networks,' in Advances in Neural Information Processing Systems (A. Beygelzimer, 458 Y. Dauphin, P. Liang, and J. W. Vaughan, eds.), 2021. 6, 8 459

G. Zhang, H. Zhao, Y. Yu, and P. Poupart, 'Quantifying and improving transferability in domain general460 ization,' Advances in Neural Information Processing Systems , 2021. 6, 8 461

D. Kim, Y. Yoo, S. Park, J. Kim, and J. Lee, 'Selfreg: Self-supervised contrastive regularization for 462 domain generalization,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 463 pp. 9619-9628, 2021. 6, 8 464

Y. Ruan, Y. Dubois, and C. J. Maddison, 'Optimal representations for covariate shift,' in International 465 Conference on Learning Representations , 2022. 6, 8 466

G. Blanchard, A. A. Deshmukh, U. Dogan, G. Lee, and C. Scott, 'Domain generalization by marginal 467 transfer learning,' J. Mach. Learn. Res. , vol. 22, jan 2021. 6, 8 468

M. Chevalley, C. Bunne, A. Krause, and S. Bauer, 'Invariant causal mechanisms through distribution 469 matching,' 2022. 6, 8 470

I. Gulrajani and D. Lopez-Paz, 'In search of lost domain generalization,' in International Conference on 471 Learning Representations , 2021. 6, 8 472

X. Liu, S. Li, Y. Ge, P. Ye, J. You, and J. Lu, 'Ordinal unsupervised domain adaptation with recursively 473

474

conditional gaussian imposed variational disentanglement,'

Machine Intelligence , pp. 1-14, 2022. 7 475

E. Rosenfeld, P. Ravikumar, and A. Risteski, 'Domain-adjusted regression or: Erm may already learn 476 features sufficient for out-of-distribution generalization,' 2022. 8 477

G. Peyré and M. Cuturi, 'Computational optimal transport: With applications to data science,' 2019. 9 478

IEEE Transactions on Pattern Analysis and

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors tackle domain generalization for regression tasks, or DGR. DGR is often different from classification tasks in DG as well as imbalanced regression tasks due to shift in labels.\n\nPopular feature alignment type of algorithms using IPM may not be applicable to DGR because the labels of different domains may not align, which may cause a model to fit to only one domain.\nThe authors propose margin-aware meta learning framework to solve DGR by weighting query based on the distance to support, which mitigates sampling bias.\n\nThe authors provide experiment results on multiple datasets compared to several feature alignment methods in domain adaptation as well as self-supervised learning and meta-learning methods.\n\nSTRENGTHS:\n1. The authors clearly show the relationship between the margin between support and query, and hardness of query tasks.\n2. From this analysis, the authors propose a simple weighting strategy for meta-learning in regression tasks.\n3. Experiments are conducted on many datasets - toy, age estimation and cross-domain rental prediction.\n\nWEAKNESSES:\n1. The relationship between the margin and hardness that the authors demonstrated is not something new. It is somewhat obvious. For example, in Gaussian Processes, the uncertainty increases as a query point gets far away from support points.\n2. Although the authors apply the proposed MAMR to meta learning framework, only one baseline model (MLDG) is used as a meta learning model. Without comparing it with multiple meta learning models e.g. ANIL, Neural Processes, it is hard to see if the improvement is coming from the proposed weighting or different framework (feature alignment vs. episodic meta training).\n3. I believe MAMR is applied to MAML framework given that there is inner and outer loop updates (needs to be clarify unless it has already been specified). Related to 2, I would like to see if MAMR improves the performance on top of other meta learning algorithms. If my understanding is correct, MAMR should be applicable to any meta learning algorithms for regression tasks.\n4. It could’ve been if different backbones were used for experiments.\n5. In Line 324-327, the authors mentioned “different from 5 or 10 inner steps in meta-learning for few-shot learning, fast adaptation by multi-steps is not necessary for DGR.” It is not really true even for MAML. With some frameworks e.g. learn2learn, MAML performs as good with 1 step adaptation as 5 or 10 step adaptations.\n\nQUESTIONS:\n1. In line 168, what do you mean by divergent optimization?\n\nLIMITATIONS:\nPlease refer to the weakness and questions above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addressed the problem of domain generalization for regression, where the authors assume that different domains may have disjoint labeling space. The authors proposed a meta-learning-based method to learn the parameter initialization for DGR. In particular, the choosing frequency of different support and query domains relates to the label discrepancy. To evaluate the proposed method, the authors have also developed a DGR benchmark, considering both overlapping and nonoverlapping labels between the source and target domains.\n\nSTRENGTHS:\n1. Domain generalization in regression is a novel and interesting problem.\n\n2. Leveraging task discrepancy to tackle domain generalization by meta-learning is novel.\n\n3. The experimental results show effectiveness on a range of datasets.\n\nWEAKNESSES:\n1. It is unclear why using meta-learning to tackle the DGR problem in this paper. Although a range of works on DG has been proposed to use meta-learning, what are the significance and novelty of using meta-learning for domain generalization in regression?\n\n2. The authors claim that the proposed method can enhance the exploration and interpolation capabilities. But it is unclear in the paper why the problem of domain generalization benefits from the interpolation capabilities.\n\n3. The theoretical results in this paper are weak to me. Thm. 1 states that a larger regression margin of meta-tasks leads to more available domains. But it neglects the probability of meta-task over the set of labeling spaces. \n\n4. the objective of meta-learning is a bi-level optimization problem.  It is unclear why not use the task discrepancy as one of the objectives of out-loop optimization, directly. Besides, how many iterations for the inner-loop optimization and outer-loop optimizations?\n\nQUESTIONS:\nPlease answer my questions about the weakness during the rebuttal period. It is hopeful that my concerns can be addressed by the author's responses.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
This work investigates and analyzes domain generalization for regression tasks and propose a margin-aware meta-learning method for DG regression, which can help learn long-range exploration and interpolation. Also, a benchmark for evaluating regression dg is built.

STRENGTHS:
- the paper is presented and derived well to make it easier to understand
- the difference between DG for regression and previous classification tasks is well analyzed

WEAKNESSES:
- is there any possibility to apply the proposed meta-learning based DG to more general regression tasks like object detection?

QUESTIONS:
Please refer to the weakness part

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper describes an extension of domain generalization to the regression domain - traditionally, most of the research in Domain Generalization has occurred in image classification and to some degree in combined tasks such as object detection, depth estimation, and semantic segmentation and there has been a lack of research purely tackling domain generalization for regression. The authors specifically focus on the label shift aspect of domain shift, where the same input features might lead to two very different responses/regression values depending on the domain that they are in. The authors describe how the extra structure in regression problems (e.g. ordinal relatedness) is something that is mostly ignored by existing methods, and they propose a modification to the common meta learning methods for domain generalization to weight the meta-tasks appropriately based on this extra structure. The proposed method achieves SOTA among a variety of datasets in DomainBed.\n\nSTRENGTHS:\n### originality\n* the paper is quite original. As the authors note, this problem has not been well studied, and so most of the results and discussion surrounding how to take advantage of domain generalization for regression is relatively novel. \n* The technique proposed in the paper is fairly novel, even though it does not contain any individually novel pieces. Weighting a meta-learning strategy based off of the \"difficulty\" of the meta-task pair, defined by how related the two tasks are is certainly not something that I have seen before and is a neat strategy that could work in a variety of use-cases. \n### quality\n* The paper has mostly sound experiments, and does a good job providing both empirical and theoretical evidence to support its claims. \n* The toy experiment (figure 3) is a great way to motivate why these kinds of techniques are needed and how existing methods may fail when they are presented with regression problems.\n### clarity\n* the paper is clear and easy to read, with both pseudo code, good diagrams/figures, and propositions that help clarify its points.\n### significance\n* domain generalization for regression is highly under-studied, and this work is a nice step forward to, at the very least, bring attention to this problem and how the underlying nature of it is quite different from classification.\n* The technique of weighting the meta learning tasks based on some objective function feels like it could be used in a variety of different contexts, and I encourage the authors to explore this more in their conclusion/future work section.\n\nWEAKNESSES:\nThe paper would benefit from a more nuanced discussion of the methods limitations, as well as more justification for some of the claims that are made. In particular,\n\n* The limitations section is currently limited to 3 bullets, none of which have any justification - for example, \"most used datasets have balanced source labels, applying to imbalanced is a more practical setting\" - for this claim, what would be an example of an imbalanced dataset? Do we expect this kind of technique to still work there, or does the method fail on imbalanced source labels? Similar comments hold for the other limitations.\n\n* Lines 36 - 38 talk about a potential outcome where feature alignment may be harmful for DGR tasks, but this would be much stronger if there was a toy experiment or example that the authors could talk about here that shows that this kind of feature alignment is a particular failure mode for popular feature alignment techniques. \n\n* The paper specifically focuses on experiments that are mostly single value regression tasks in nature, e.g. rental prediction - however, a large portion of regression tasks have image features as input - for example, depth estimation or bounding box regression. The paper would benefit from a discussion on whether or not these techniques could extend to those more common regression tasks, or what would need to change in order to make that happen.\n\nQUESTIONS:\n* What is the compute required to run the meta learning algorithms? Does computing the margins cause significant extra weight?\n\n* I know the limitations touch on \"unreliable calculations of representation distances in meta space\" - how sensitive is this technique to ensuring that these calculations are correct?\n\nLIMITATIONS:\nAddressed in the weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focused on the generalization problem in the regression context and proposed an interesting setting, Domain Generalization in Regression (DGR). Compared to domain generalization for classification, DGR is a more challenging task due to its different domain shift forms between source and target domains. Compared with the traditional domain generalization for classification where the source and target domains usually shared the same label set, the range of response variables between the source and target domains in the context of regression can be very different. From this perspective, this setting proposed in this paper is novel and worthy to be explored. This paper proposed to solve the DGR problem through the meta-learning philosophy. However, the authors pointed out that when applying the meta-learning method for the DGR problem, the harder meta-tasks with larger regression margins on the label discrepancy can be less sampled and this sampling bias made these harder tasks difficult to optimize. Then, the authors proposed to address this sampling issue by measuring the feature discrepancy between the query and support samples and assigning higher weights to harder meta-tasks. Extensive experiments showed that the proposed MAMR can solve the DGR problem in an effective manner.\n\nSTRENGTHS:\n1. This paper proposed a novel and practical setting, Domain Generalization in Regression. The generalization problem in the regression context was less explored in the community. This paper may inspire more explorations into this topic.\n2. This paper analyzed the differences between the domain generalization tasks in the classification and the regression and showed the specific features of this problem, e.g., the distribution shift between the source and target domains can be under a different form, the sampling bias due to the ordinal relatedness in the regression. Based on these analyses, we can see that some existing methods cannot be directly applied to this field.\n3. The motivation for the proposed methods is clear. Due to the sampling bias, the author proposed a weighted meta-learning method. The solution is reasonable.\n4. The extensive experiments on some synthetic and real-world datasets demonstrated the effectiveness of the proposed MAMR.\n5. This paper is well-written and easy to follow.\n\nWEAKNESSES:\n1. Some assumptions in the theoretical analysis can be further discussed. For example, it seems that Theorem needs to assume that the data from a meta-task distribute uniformly with respect to the response variable $y$.\n\nQUESTIONS:\n- In order to make Theorem 1 hold, do we need to assume the label distribution within a meta-task need to be uniform? If so, is it possible to extend the conclusion into imbalanced scenarios?\n- In Section 5.3, it is interesting to see that MAMR performed better for the causality discovery in this toy example. Could the author explain more about the reasons why MAMR had better performances under this kind of scenario?\n- In Table 2, it seems that the MAMR-G using MMD with Gaussian kernel usually had poorer performances compared to MAMR- (without any margin-aware weight strange). Could the author explain more about this interesting observation?\n\nLIMITATIONS:\nNo severe limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}