Robust Transfer Learning Based on Minimax Principle
Authors: Xinyi Tong
Xiangxiang Xu
Shao-Lun Huang
Lizhong Zheng
Venue: Submitted to ICLR 2023 (2023)
Keywords: Transfer Learning
Minimax Principle
Robustness

Abstract: The similarity between target and source tasks is a crucial quantity for theoretical analyses and algorithm designs in transfer learning studies. However, this quantity is often difficult to be precisely captured. To address this issue, we make a boundedness assumption on the task similarity and then propose a mathematical framework based on the minimax principle, which minimizes the worst-case expected population risk under this assumption. Furthermore, our proposed minimax problem can be solved analytically, which provides a  guideline for designing robust transfer learning models. According to the analytical expression, we interpret the influences of sample sizes, task distances, and the model dimensionality in knowledge transferring. Then, practical algorithms are developed based on the theoretical results. Finally, experiments conducted on image classification tasks show that our approaches can achieve robust and competitive accuracies under random selections of training sets.

Under review as a conference paper at ICLR 2023
ROBUST TRANSFER LEARNING BASED ON MINIMAX PRINCIPLE
Anonymous authors Paper under double-blind review
ABSTRACT
The similarity between target and source tasks is a crucial quantity for theoretical analyses and algorithm designs in transfer learning studies. However, this quan- tity is often difficult to be precisely captured. To address this issue, we make a boundedness assumption on the task similarity and then propose a mathematical framework based on the minimax principle, which minimizes the worst-case ex- pected population risk under this assumption. Furthermore, our proposed minimax problem can be solved analytically, which provides guideline for designing robust transfer learning models. According to the analytical expression, we interpret the influences of sample sizes, task distances, and the model dimensionality in knowledge transferring. Then, practical algorithms are developed based on the theoretical results. Finally, experiments conducted image classification tasks show that our approaches can achieve robust and competitive accuracies under random selections of training sets.
INTRODUCTION
The goal of the transfer learning to solve target tasks by the learning results from some source tasks. In order to study the fundamental aspects of the transfer learning problems, it is important to define and quantify the similarity between source and target tasks (Pan & Yang 2009). While it assumed that the source and target tasks are kind of similar in transfer learning problems (Weiss et al. (016)) the joint structures and similarity between the tasks can only be learned from the training data, which is challenging to be practically computed due to the limited availability of the labeled target samples. Therefore, in order to conduct meaningful theoretical analyses, it is often necessary to make extra assumptions, such the linear combination of learning results (Ben-David al. 2010) and linear regression transferring (Kuzborskij & Orabona 2013), which could be limited in many applications.
As such, in this paper, attempt to theoretically study the transfer learning by only assuming that the similarity between the source and target tasks is bounded, which is a weaker assumption, and is often valid in transfer learning problems. Under such an assumption, the minimax principle can be applied (Verdu & Poor 1984) for estimating the target distribution. Based on this principle, the estimator minimizes the worst-case expected population risk (EPR) (Jin et al.) 2018) under the bounded task distance constraint, which maintains robustness against the weak assumption. Practi- cally, many empirical works have also followed the minimax setting and verify its validness (Zhang et al. 2019), while the theoretical analyses appear to be rather behind. The main challenge of analyzing general minimax problems in transfer learning is due to the difficulty of computing the expectations of the population risk under popular distance measures, such as the (K-L.) divergence (Thomas & Joy 2006).
To deal with this difficulty, we adopt the widely used x2 2ddistance and Hellinger distance (Csiszár & Shields (004) as the distance measure between data distributions of the tasks, and present a minimax formulation of transfer learning. By adopting such measures, the proposed minimax problems can be analytically solved. In particular, we show that the optimal estimation to linearly combine the learning results of two tasks, where the combining coefficient can be computed from the training data. This provides a theoretical justification for many existing analyzing framework and algorithms (Ben) David et al.) 2010); Garcke & Vanck, 2014). Note that the recent work (Tong et al. (021 also analytically evaluates the combining coefficients, which rely on the underlying task distributions that not available for real applications. Our work essentially provides the combining coefficient that

---

Under review as a conference paper at ICLR 2023
both theoretical optimal and computable from data, which can be more appealing in practical applications.
Moreover, the analyses of the minimax transfer learning problem discrete data be extended to the continuous data for real applications. In the continuous case, we consider similar transfer learning scheme as in (Nguyen al. 2020), which transfers the topmost layer of the neural networks between source and target tasks. In particular, we show the analytical solution optimal weights in the topmost layer, which is again linear combination of the weights of the source and target problem. Furthermore, we propose the transfer learning algorithm guided by the theoretical results, where the robustness and performances of the algorithms are validated by several experiments on real datasets.
The contribution of this paper can be summarized as follows:
We make mild assumptions of the task distance and propose a minimax framework for analyzing transfer learning. Additionally, we establish the analytical solutions of the minimax problems in the discrete data space.
We extend the analyses continuous data and establish similar results for the learning models with neural networks. Furthermore, we apply our theoretical results to develop robust transfer learning algorithms.
The experiments in real datasets validate our proposed algorithms, where our approaches can have higher robustness and competitive accuracy.
Due to the space limitations, the proof' of theorems are presented in the supplemental materials.
PROBLEM FORMULATION
2.1 NOTATIONS AND DEFINITIONS
We denote X and Y as the random variables of data and label with domains X' and V, respectively. For ease of illustration, the data X is set as discrete random variable in section and section 3
We consider the transfer learning problem that has a target task and source task, denoted as task and S, respectively. For each task = S, there are n, training samples {(=] ,ee generated from the underlying joint distributions p(i w (x. v) > 0, for all (z,y) E x y. The empirical distributions p(1) (i T,S) of the samples are defined as Xy
} i.i.d.
60) XY (5,
1
x. Vi (1) y):
where 11.} denotes the indicator function (Feleer 2008 and let Pn. be the set of all the possible empirical distributions supported by X' ) with n samples.
In this paper, we employ the following two distance measures for probability distributions, which are also widely used in statistics (Csiszár Shields 2004), and more convenient in our analyses.
Definition 2.1 (Referenced x2 -diistance). Let R(z), P(z), and Q(2) be the distributions supported by Z. The x2 --dittance between P(z) and Q(z) referenced by R(2) is defined as follows,
(P(z)- - 2())))
(P.Q)) Σ
R(2)
ZEZ
Definition 2.2 (Hellinger Distance). Let P(z) and Q(z) be the distributions supported by Z, The Hellinger distance between P(=) and Qzz) is defined as follows.
(P)).Q) A 1 Σ (VP())-V((:)
and H(P,Q) 2 ())...Q)
=EZ
'This assumption comes from the fact that in practice such joint distributions are typically modeled by some positive parameterized families, e.g., the softmax function.
2

---

Under review as a conference paper at ICLR 2023
2.2 MINIMAX FORMULATION
Since estimating the similarity between target and source tasks from data is challenging, we attempt only make the assumption that the distance between two tasks is bounded by constant D under a distance measure d(; )) i.e.
d (PXY. P/ S.D
(3)
Based on this assumption, we consider a minimax formulation for estimating the target distribution with estimator Qxy (PX) Pxy), where the notation represents that the estimator QXY is function of the empirical distributions P(D) and ((5) XY
Minimax Formulation:
IE d (P( (Xr. Qxr)
min
max p(T) p(a))
QXY POT) ((B)
plS)
where the expectation taken over all possible p00 and p(s) in Png and Png.
This formulation can be divided into two parts: (1) for given estimator Qxv. we consider the largest expected population risk (EPR) of the distance between the underlying target distribution and the estimator; (2) we find the best estimator QxY the function of training data that could minimize the worst risk. Note that the empirical distributions are sufficient statistics for the underlying distributions (Van der Vaart 2000). We therefore consider Qxy as the function of both empirical distributions. Accordingly, the EPR of the derived estimator under the true similarity is always smaller than the result of formulation (4). In other word, we design an estimator that has upper-bounded EPR and thus leads to robustness.
Notice that the formulation (4) is generally difficult be solved analytically due (i) the distance measure d(;;)) can cause difficulty in computation, e.g., the logarithm function the K-L. divergence; (ii) the expectations over Xx and X' follow the multinomial distribution (Csiszár 1998) i.e.,
the probability of the empirical distribution P(P p(0) x exp(-n, Xy | complicated to analyze. To address the issue (i), we choose the x2-distance and Hellinger distance, which are more convenient to be analyzed in minimax problems. Moreover, for the issue (ii,
XY which is
propose to study the surrogate problem which replaces the expectation computation in by the integral
[f Qxy)
IIx (-nid (PSr. dxx.- XY -=S
(5)
Note that (5) is the asymptotic approximation of the expectation over multinomial distributions with the additional surrogation that the exponent in (5) can be chosen different from the K-L. divergence. Such asymptotic approximation is also applied for theoretical analyses in high dimensional statis- ties (Morris) 1975). Then, the goal of this paper to study the following minimax problems for transfer learning.
Formulation (referenced x" -diistnnc::
(P(
(xx))xxx]
min (PT( P P X P() 52
max
(6)
where the expectation is the integral over
P
(00) (() Xvi XY
exp (- (-2
n4 2 XR Pxx: Pkx) ;
Note that the referenced- x distance can be recognized as an asymptotic approximation of K-L divergence by Lemma A.I in Appendix A Here the reference distribution /t selected as XY
p(s)
Formulation (Hellinger distance):
[[HH2
H (PXP.
min QXy H(P (())
max
(8)
"Since the souree samples sufficient, with high probability, all the entries of ((9) positive.
3

---

Under review as a conference paper at ICLR 2023
where the expectation is the integral over
( p() XY: XY
x exp (-2n,H2
H2 ( ( i
Note that Hellinger distance provides a lower bound of the K-L divergence with Lemma A.2 in
Appendix A and thus Formulation 2 computes a lower bound of the population risk
ANALYSES FOR DISCRETE DATA
In this section, we provide the analytical solutions of the formulations (6) and (8). Similar minimax
estimation problems have been studied in early works (Trybula, 1958 Berry 1990). We now directly
give the detailed expressions and the proof is provided in the supplementary material.
3.1 ANALYTICAL SOLUTION OF FORMULATION 1
Theorem 3.1. Let QXYy be the estimator that achieves the minimax solution of problem 6 and ther ((T)
(xy (1 a1) ( (I,z) + ai a (x, y).
((5)
(10)
for all (x,y)= E EX y/ where
n2g
I [xjvi
n a DD ny +ig
D
01
Rr + ng
Prnn DD1 12-+-15
D1
(11)
P(S)). Specifically, (x)- denotes the modified Bessel function of the first and D1 & ( xy,
kind with order v (Abramowitz et al.
1988, whose definition is in Appendix B
In the remaining parts of this paper, we denote for ease of presentation (,) A Tx())//Ts
Theorem 3.1 implies that linearly combining the learning results of different tasks is preferable method for robust transfer learning, which also widely used in existing algorithms and theoretical frameworks (Ben-David et al. 2010). Moreover, these works intuitively assume it, whereas we provide a theoretic support that could help explain the rationality.
Remark 3.2. To help understand the related factors contained in (77), we consider a special regime that x, where Ju( ~ x 2/v.. Then the expression of (I7) can be approximated by
n.
(1
mrng
O1 TTT + ns
(ns + nt )
(12)
This result is consistent with (6) ) in (Tong et al. (021) under this special regime as explained in Appendix C The coefficient og, which represents the requirement of source samples, is positively associated with the model dimensionality |X||), which comes from that we learn all the |X||]] entries of the target distribution, and negatively associated with the target sample size and task distance These relationships are examined in the experimental part. We also provide an interesting geometric explanation of this pattern, which is shown in Appendix C
3.2 ANALYTICAL SOLUTION OF FORMULATION 2
In the following, we provide the solution of problem (8) based on Hellinger distance.
Theorem 3.3. Let Q XY the estimator that achieves the minimax solution of problem and then
(x, (L (1-a2)V
o2) ((T) 02 (
((S) (x,z)
for all (xr E X x V/ where
ns
(1 (1-
D D2
AngnnrDD ToT ng
0.
Tor + ng
and D2 A (P(T) ((9)
Accordingly, we can also achieve a similar interpretation of the affecting factors as in section 3.1
TTii solution actually requests D/ / //nr + 1/ns S VPT which can be casily guaranteed.
'I requests D/ V1/4nT 1/Ans S / x.D
4

---

Under review as a conference paper at ICLR 2023
4 CONTINUOUS CASE AND ALGORITHM
In this section, we extend the previous analyses of discrete data continuous data, which conform to the practical setting. In such cases, the previously adopted empirical distributions not be seen as valid observations due to the infinite cardinality |.8], where most of the possible data are not sampled.
In order to apply the previous analyzing framework, we consider the retrain-head method (guyenn et al. 2020), which is a commonly used transfer learning technique. With this method, a pre-trained network is prepared for extracting the feature of the data and then the topmost layer (known as "head") can be retrained with observed samples. Under such a setting, we can recogmize the weights in retrained topmost layers the observations from the corresponding tasks. Note that compared with the cardinality of the large data space, the topmost layer has much fewer parameters.
In detail, pre-trained network is composed of two parts of networks: (a) the previous layers whose input is data x and output is the d-dimensional features f(x) ) E R and (b) the topmost layer for linear classification, with weights g(y) E IR of each label y. Each task provides the learned weights to estimate the optimal weights for the target task. Moreover, the learned weights are decided by the model proposed for describing the data. For thcoretical analyses, we provide the discriminative models for x distancce and Hellinger distance, where thereupon For convenience, we use notation h to represent the topmost layer in section
design practical algorithms.
4.1 REVISED FORMULATION AND MM-X2 ALGORITHM
When d---tttance is chosen as the distance measure, we consider the discriminative model for the target distribution in the factorization form
(f.g) (ulz) & (1+"
(1
(13)
which provides the probability of each label y y for any data x. Such a model has been introduced in factorization machines (Rendle, 2010) and is commonly used in natural language processing problems (Levy & Goldberg 2014).
Under the pre- pre-trained feature extractor f*).). the learned weights of topmost layers can be derived by minimizing the distance between the empirical distribution and the model. For computation, we avoid using the joint distribution the reference and define the -isttancce measure referenced by the product marginal distribution, i.e., (.--)
ppe pST) (,, ). Then, the learned weights 9: of
each task i T, can be defined as
g. ف arg ming Xhs (ph). xY, P(T) (
(14)
Now År and ğs are the observations to generate the minimax solution, where the expectations can be defined as gi (v) = (z)). Note that the parameters of gr are just the topmost weights we hope to achieve. Then, the minimax problem can defined as follows [ef. (6)):
g arg min max E (P
Yrx P.P (TP)))())) (0T
(r))))
(15)
g 93E9
where g (9: x (P
of
(r))) Y
P
(T)
(f S De)
We can directly apply Theorem
3.1 and obtain the following theorem. Theorem 4.1. When the empirical distributions P(T) and ((5) follow the density function 7 the minimax solution as defined in
15 is
g
(1 āl)rr + āigs,
(16)
where
ng
1 - D-Di"n --Daa
D DTRSDD
ā
ThT ngg
1oT + ngg
and D9 A (P of Q
.r)
((T)
(T) Q .9))
In practice, when the assumption D/ V1/Tr + 1/ns Vdly does not hold, the estimator can still provide sub-optimal solution.
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Minimax X² AAlgorithmm (MM-x²)
1: Input: target and source data samples {(0) (6) (0) ) 1 (i = T, S), learning rate 7)
31 B
2: Randomly initialize "
3: repeat
4: (f g"g + (f*. g") (1) g")
5:
ng
D Jayı
ar+ 6: until fg converge
D1
artas
7: return g9
Despite the theoretical analyses where the feature extractor is fixed, our algorithm jointly optimizes the feature extraction f and the topmost layers g, which a typical retraining procedure. It is proved that the linearly combined weights (T6) can be achieved by minimizing the linearly combined training loss with the same coefficient. We therefore define
Accordingly,
design algorithm based Theorem 4.1
(ax _ - a)xas (PAP) P ( + ax²s (PXP P(T P ((f-))) Y1xx
(17)
Then, the MM-x² algorithm given in Algorithm|
practice, the loss L, (a, .). the related quantities D, and Dj in Theorem 4.1 can be estimated by the empirical means of the features of samples. Detailed implementations are provided in the supplementary material. With the f and g computed by Algorithm [ the predicted label ((x)) for sample x is given by the maximum a posterior (MAP) decision rule with f(z) arg max Y
8²) 3") (y/æ))
4.2 REVISED FORMULATION AND MM-HEL ALGORITHM
When Hellinger distance is chosen the distance measure, consider the discriminative model for the target distribution in the following form. For each = S,
f. (y)x) ( (v) (1 f'())((y)) yix
(18)
This model a deformation of 13 which makes the model trainable under Hellinger distance. Similarly, under the pre-trained feature extractor f", the learned weights h; of each task i T, can be defined as
h 2 arg min H² (PLr. Plo 00/F")).
(19)
h
Now hr and hss are the observations generate the minimax solution, where the expectations be defined as (v) (v), T.S. The minimax problem can be defined follows [ef. (8)
h arg min max E 2 (T) QT. (rr), h(hr.hg) hiEH
(T.S
p.),p( ( (T.
(20)
where H A (h 1 Ever | (T)
V P (
(m) S D2) and A,
E Pro (X) for S.
We can directly apply Theorem 3.3 and obtain the following theorem.
Theorem 4.2. When the empirical distrbutions (TT XY and p(.) XX follow the density function 9). the minimax solution as defined in (20) is
h h²(y)
0z hr 02 (1)P (M)A +
for all y E y, where
Anrrs DD2 DDz
ā2 nT + nt
ns
1 -
Japi Jayi
(21)
D2
TET + ns
and D3 2 1 EnEE> | p( ((T
(
hes

---

Under review as a conference paper at ICLR 2023
Algorithm 2 Minimax Hellinger-Algoriihe (MM-Hel)
1: Input: target and source data samples {(x]
) (i T,S)
(0)
9i
2: (f", hj. h2) t- arg min f.hr Lf h1, h2)
3: a f oT (1 Jayı
Anyns DD2
(v)
4: hy) (1 -1-a))h(uv) + vv (1)
A3)((0)
5: return f",h"
Similarly, we can design an algorithm based on Theorem Note that we cannot apply the linearly combined training loss the Hellinger distance setting due the design of two different distribution models ( and y|x f... We choose jointly training the shared feature extractor f and individual topmost layers of the target and source task. The training loss is defined as
(f, h1, h22 a H² (P P QVIX + H (PlS) 1
Ovix ((S.f-aa))
(22)
Then, the MM-Hel algorithm is given in Algorithm Similarly, we also provide the estimation the related quantities in the supplementary material. With the computed f" and h", the predicted label
(ar) for sample is given by the MAP decision rule with j(x) arg max v VOYIx" (T.f19')) 9¹) (vlar)
5 EXPERIMENTS
To validate the theoretical analyses in Theorem 3.1 and Theorem 3.3 and the robustness of our algorithms, we conduct series experiments on common datasets for image recognition, including CIFAR-10 (Krizhevsky et al. 2009), Office-31 and Office-aaleech (Gong et al. 2012b) datasets. For convenience, different transfer settings are denoted by "source->target".
5.1 CIFAR-10
We conduct transfer learning experiments on CIFAR- 10 datasct in order to verify the theoretical interpretations of the related factors in Remark 3.2 which mainly cover the sample size and task distance. Specifically, CIFAR-10 dataset contains 50 000 training images and 10 000 testing images in 10 classes. We first construct the source tasks and target task by dividing the original CIFAR- 10 dataset into five disjoint sub-datasets, each containing two classes the original data, which corresponds to a binary classification task. Then, choose one our target task (task 1), and the other four as source tasks referred to as task 2, 3, 4, 5, where four corresponding transfer learning tasks are established.
In each transfer learning task, we use 2000 source images, with 1000 images per binary class. Target sample size n is set feature if is of dimensionality d = 10, generated by GoogLeNet (Szegedy et al.) 2015)) pre-trained by ImageNet (Russakovsky et al. 2015). and followed by a fully connected layer.
12, 20, 60, 200 for four sub-tasks. Throughout this experiment, the
The accuracies on the target testing images of MM-x² and MM-Hel algorithms are summarized in Table Table 2 In each task, target samples are randomly picked from the target training set. All the accuracies and standard deviations are reported over five random selections of target samples. In detail, we analyze the effect of target sample sizes in figure T figure T shows the changes of the accuracics and coefficient áj as defined [16 (averaged over tests) of different target sample sizes. The coefficient ãj represents how much the final model relies on the source task. It corresponds to Remark 3.2 that larger target sample size could lead to less dependency on
our interpretation the source task, and meanwhile the accuracies become higher and more stable. We also analyze the effect of task distances as in figure 2 figure 2jsoows the changes the accuracies and task distance D2 as defined in (1) (averaged over 5 tests) of different source tasks, where a larger task distance can lead to worse accuracy and stability.
7

---

Under review as a conference paper at ICLR 2023
Table 1: Accuracies (%) of CIFAR-10 trans- fer learning tasks based on MM-x² algorithm, where ny represents the target sample size. The baseline is trained with merely target samples.
Table 2: Accuracies (%) of CIFAR-10 trans- fer learning tasks based on MM-Hel algorithm, where nr represents the target sample size and shares the same baseline with Table
Tasks
12
PV 20
60
200
Tasks
By 12
Bp 20
By 60
By 200
Baselino
77.8=2.7
83,4+170
90.0=1.0
93.4+0.7
2
81.5++1.2
88.5+1111
91.2440.8
94.94-0.5
82.6=14
80.24-1.7
87.==12 1.2
90.9=0.7
90.6++1.2
94.2+00.6
93.94-0.7
3
81.3+17
87.5+12
91.24008
94.24:0.6
4
79.1.22.2
85.1-1.7
90.0.11.3
93.4.10.8
4-+1
77.5+18
83.9
90.211.0
93.511..
15.6++1.5
89.7-113
91.140.77
94.34-0.5
84.2+1.
90.2+:111
92.24+077
94.74:0.5
Accuracles on target tack?'
41-
3.2
Accuracies on target tooket
0-0
17
1.6
3.28
20.0
1.5
distance..
Coefficient
27.5
Acorrscias
Costiicntt
3.4
3.12
25.0
1.3
1.2
30
Target sample size
Source Task
Figure 1: The accuracies and coefficient og in transferring task > based on MM-x2 algorithm under different target sample sizes.
Figure 2: The accuracies and distance measure
D: of target sample size ny 12 based on MM-Hel algorithm under different source tasks.
5.2 OFFICE-31
Office-31 dataset contains images of 31 categories with sub-datasets, including Amazon (A, 2817 images), Dslr (D, 498 images), and Webcam (W, 795 images). Six transferring tasks can be established as A-D, A D-W, D->,, A, W-A, and W-D. We adopt the transfer learning setting in (Tzeng al. 2015), illustrated follows. Specifically, 3 target samples per category are used for training, and the training sample size (per category) for source task is set to 20 or 8, depending on whether the source task is Amazon not.
In this experiment, the feature f is extracted by the VGG-16 (Simonyan & Zisserman 2014 network pre-trained on the ImageNet, succeeded by fully connected layers, and the output 64-dimensional. We introduce the UDDA (Motiann et al. 2017) algorithm as the typical baseline and the iterative linear combination method (Tong et al. 2021) (ILCM) for comparison, which employs similar linear combination method.
Table summarizes test accuracies under different transfer settings, where all reported accuracies and standard deviations are averaged over five train-test splits. The results indicate that our algorithms generally have higher robustness and competitive accuracies.
5.3 OFFICE-CALTECH
Mlfice-Caltech dataset is composed of categories, divided as four sub-datasets; Amazon (A. 958 images), Caltech (C, 1123 images), Webcam (W, 295 images), and Dslr (D. 157 images). We focus on the six transfer settings depending on C, i... A-C, WCC, D-c. C-*A, C-WW and C-D. The train-test split is introduced in (Gong et al. 2012a The feature is based on the pre-trained DeCAF network (Donahue et al. 2014), succeeded by fully connected layers, and the output dimension is 10.
Table shows the performances in comparison with CPNN (Ding et al. 2018) and ILCM (Tong et al. 2021 algorithms, where CPNN is chosen the baseline to be consistent with ILCMM.
8

---

Under review as a conference paper at ICLR 2023
Table 3: Test accuracies for target tasks under different transfer settings on Office-31.
Method
A-+
A-W
D-W
D-A
W-A
W-+
UDDA
89.0:1122
88.2:1100
96.4100.8
71.810.5
72.111100
97.6:10.44
ILCM
9..0
87.3.1.1
96.511.0
72.411.
72.1110.9
97.2:10.4
MM-x² (Ours)
90.2.10.6 89.
87.510.5 88.1+004
9..7100.5
72.910.8
72.2:00.6
97.610.3
MM-Hel (Ours)
96.0+0.5
72.2+00.5
72.0+0.4
97.5+0.4
Table 4: Test accuracies for target tasks under different transfer settings Office-Caltech.
Method
A-C
W-C
D-+C
C-AA
C-W
C+D
CPNN ILCM
7..310.6
72.1-0.8
66.61008
86.2:00.5
86.0.10.5
79.910.77
80.3+0.7
72.9+0.7
72.2+0.99
88.4
85.9
83.5+0.9
MM-x² 2 (Ours)
79.9=0.77
73.5+005
71.7+0.7
90.1+0.5
86,8+0.5
84.0+0.7
MM-IHel (Ours)
7.3
72.710.55
72.110.6
88.5.10.4
85.5.10.4
83.510.55
6
RELATED WORKS
6.1 MINIMAX ESTIMATOR OF BOUNDED NORMAL MEAN
Minimax estimator a significant theme in statistical decision theory, which deals with the problem of estimating deterministic parameter in certain family (Hodges & Lehmann 2012). Under the special setting of bounded normal mean, many works study the analytical solution when the centers of Gaussian observations are restricted, including analytical solution for 1-dimensional observations (Casella & Strawderman 1981). high-dimensional observations (Berry) 1990, Marchand & Perron 2002). Moreover, the objective function also be measured by norms other than mean square error (Bischoff et al.) 1995), which allows the applications in machine learning scenarios and help derive the solutions of this paper's formulations.
6.2 SELECTION OF DISTANCE MEASURE
In this paper, we select x2 disttance and Hellinger distance as the distance measure in which help analytically solve the minimax problem. These two measurements are both from the family of f-divergence (Csiszár & Shields, 2004) and are widely-used in machine learning. Specifically, >--distance lead the typical alternating conditional expectation algorithm (Xu & Huang 2020 Hellinger distance is also used to evaluate the domain adaptation transfer learning (Baktashmotlagh
al., 2014; 20166). Moreover, most existing measurements with non-linear functions, e.g. K-L divergence containing the logarithm function, could be ill-defined, as explained in section 2.2
6.3 MINIMAX TRANSFER LEARNING AND ROBUSTNESS
Minimax principle has been widely-used in transfer learning to promote the robustness of algo- rithms (Verdu Poor 1984p. The most common empirical method is connected to the adversarial learning methods (Shafahi al. 2019), including maximizing the training loss of adversarial classi- fier (Tzeng et al. 2017) and maximizing the discrepancy between classifiers' outputs (Saito et al.) 2018). Meanwhile, transfer learning settings naturally imply minimax optimization problems in view of the relationship between target and source tasks (Zhang et al. 2019). Recent researches reveal that the maximization
succeed over the constraints the distribution shift (Lei al.) 2021). the similarity between neaural network parameters (Kalan et al., 2020), and optional source
tasks CCai & Wei 2021)
CONCLUSION
This paper introduces minimax framework for transfer learning based the assumption of task distance. We provide the analytical solution of the minimax problem and characterize the roles of sample sizes, task distance, and model dimensionality in knowledge transferring. In addition, develop robust transfer learning algorithms based on theoretical analyses. Experiments on practical tasks show the robustness and effectiveness of our proposed algorithms.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Milton Abramowitz, Irene A Stegun, and Robert Romer. Handbook of mathematical functions with formulas, graphs, and mathematical tables, 1988.
Mahsa Baktashmotlagh, Mehrtash T Harandi, Brian C Lovell, and Mathieu Salzmann. Domain adaptation on the statistical manifold. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2481-2488. 2014.
Mahsa Baktashmotlagh, Mehrtash Harandi, and Mathieu Salzmann. ceerriutiio-mmachiinn embed- ding for visual domain adaptation. Journal of Machine Learning Research, 17:Article-number 2016.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79(1)=111-175, 2010.
J Calvin Berry. Minimax estimation of bounded normal mean vector. Journal of Multivariate Analysis, 35(11:130-133) 1990.
Wolfgang Bischoff, Werner Fieger, and Stefanie Wulfert. Minimax-and 7y-minimxx estimation of a bounded normal mean under linex loss. Statistics & Risk Modeling, 13(3):287-298, 1995.
Tony Cai and Hongji Wei. Transfer learning for nonparametric classification: Minimax and adaptive classifier. The Annals of Statistics, 49(1):100-128, 2021.
George Casella and William E Strawderman. Estimating a bounded normal mean. The Annals of Statistics, 94))8870-878 1981.
Imre Csiszár. The method of types jinformation theory]. IEEE Transactions on Information Theory, 2466))2505-2523, 1998.
Imre Csiszár and Paul C Shields. Information theory and statistics: A tutorial. 2004.
Zhengming Ding, Nasser M Nasrabadi, and Yun Fu. Semi-supervised deep domain adaptation via coupled neural networks. IEEE Transactions on Image Processing, 27(11):5214-522 2018.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In International conference on machine learning, pp. 647-655, PMLR, 2014.
Willliam Feller. An introduction to probability theory and its applications, vol 2. John Wiley & Sons, 2008.
Jochen Garcke and Thomas Vanck. Importance weighted inductive transfer learning for regression. In Joint European conference on machine learning and knowledge discovery in databases, pp. 466-481. Springer, 2014.
Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE conference on computer vision and pattern recognition, pp. 2066-2073. IEEE, 2012a.
Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE conference on computer vision and pattern recognition, pp. 2066 2063. IEEE, 2012b.
JL Hodges and Eric Lehmann. Some problems in minimax point estimation. Selected Works of EL Lehmann, pp. 15-30. Springer, 2012.
Chi Jin, Lydia T. Liu, Rong Gc, and Michael I Jordan. On the local minima of the empirical risk. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
and R. Garnett (ods.),
Seyed Mohammadreza Mousavi Kalan, Zalan Fabian, A Salman Avestimehr, and Mahdi Soltanolkotabi. Minimax lower bounds for transfer learning with linear and one-hidden layer neural networks. arXiv preprint ar0iv:2006. 10581, 2020.
10

---

Under review as a conference paper at ICLR 2023
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Ilja Kuzborskij and Francesco Orabona. Stability and hypothesis transfer learning. In International PMLR, 2013.
Conference Machine Learning, pp.
Qi Lei, Wei Hu, and Jason Lee. Near-optimal linear regression under distribution shift. In Interna- fional Conference on Machine Learning, pp. 6164-6174, PMLR, 2021.
Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. Advances in neural information processing systems, 27:2177-2185, 2014.
Eric Marchand and François Perron. On the minimax estimator of bounded normal mean. Statistics & probability letters, 58(4))327-333, 2002.
Carl Morris. Central limit theorems for multinomial sums. The Annals of Statistics, pp. 165-188, 1975.
Saeid Motiian, Marco Piccirilli, Donald A Adjeroh, and Gianfranco Doretto. Unified deep supervised domain adaptation and generalization. Proceedings of the IEEE international conference on computer vision, pp. 5715-5725, 2017.
Cuong Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. Leep: A new measure to evaluate transferability of learned representations. In International Conference on Machine Learning, pp.
PMLR, 2020.
Sinno Jialin Pan and Qiang Yang. survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10))1345-1333 1359 2009.
Steffen Rendle, Factorization machines. 2010 IEEE International Conference on Data Mining, pp- 995-1000. IEEE, 2010.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115
1)::2112552, 2015. doi: 10.1007/s11233-015-00
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3723-3732, 2018.
Ali Shafahi, Parsa Saadatpanah, Chen Zhu, Amin Ghiasi, Christoph Studer, David Jacobs, and Tom Goldstein. Adversarially robust transfer learning. arXiv preprint
2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint
1556, 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
MTCAJ Thomas and A Thomas Joy. Elements of information theory, 2006.
Xinyi Tong, Xiangxiang Xu, Shao-Lun Huang, and Lizhong Zheng. mathematical framework for quantifying transferability in multi-source transfer learning. Advances in Neural Information Processing Systems, 34, 2021.
Stanislaw Trybula. Some problems of simultaneous minimax estimation. The Annals of Mathematical Statistics, 29(1):245-25 253, 1958.
Eric Tzeng, Judy Hoffman, Trevor Darrell, and Kate Saenko. Simultancous deep transfer across domains and tasks. In Proceedings of the IEEE international conference on computer vision, pp. 4068-40 1076.4076, 2015.
11

---

Under review as a conference paper at ICLR 2023
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7167-7176, 2017.
Aad W Van der Vaart. Asymptotic statistics, volume Cambridge university press, 2000.
Sergio Verdu and H Poor. On minimax robustness: A general approach and applications. IEEE Iransactions on Information Theory, (002))328-340) 1984.
Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big data, 3(1): 3(1):1-40, 2016.
Xiangxiang Xu and Shao-Lun Huang. Maximal correlation regression. IEEE Access, 8:26591-26601, 2020. doi: 10. ACC//ACCCS.22......
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learming, pp. 7404-7413, PMLR, 2019.
A APPROXIMATION OF K-L DIVERGENCE
Firstly, we provide the following approximation of K-L. divergence, under the assumption that R(z), P(z), and Q()) are close to each other.
Lemma A.1. Suppose that [P(z) R(2) < c and IQ(z) R(z) < c for each 2, where c/ z 1, K-L diverreeee between P(z) and Q(2) can have the following approximation
1 D(II - x 0(6))
(23)
Proof.
D (PIIQ)) = Σ - PP(2) Q(2)
P(z)
2EZ
1+ Q(2) - Q(2)-((2)) P(z)
Poz)log
P(z)
2
Σ P(()) (Q(2) -P(2) (Q(2)-P(x))) P(=))²
+ +
P(=)
2P2(z)
2EZ
(Q(2)
- P(2))2 0(2))
Σ
2P(2)
ZEZ
(Q(=) - -((2))2
Σ
2(R(2) + P(z) - R(z)) +0(2))
#EZ
1
(Q(z) - P()))
+ +0(E).
2
R(2)
ZEZ
Secondly, we consider Hellinger distance is closely connected with K-L divergence as the lower bound, which is explained in the following lemma.
Lemma A.2. Let P(z), and Q(z) be the distribution supported by 2. We have
D (PI()) 2 2H²(P.Q).
(24)
The proof of this lemma is easy find and omitted here.
12

---

Under review as a conference paper at ICLR 2023
B
PROOF OF THEOREM 3.1 AND THEOREM 3.3
First, the regular definition of the modified Bessel functions of the first kind is
(2m+t)
[x,) ml[ (m + v + 1) 2
where r()) denotes the gamma function.
To solve the minimax problem (6) and (8, we apply the minimax estimation of bounded normal mean vector |eerr (1990). The following lemma is direct extension of the bounded normal mean results.
Lemma B.1. Given two observations y ~ (x, xx and w N2, IR where 13 denotes the k x k identity matrix, their centers satisfy || - = I| S D, and D/vof + of S Vk. Let it be the
minimax estimator for the following minimax problem, e.g.,
x =ag min
max
E Ell w) x xx1²]
(25)
(y x =>
Then, the expression of a is
o of + 02
of
2 ( ojto
w|l
4"
W +
y
+
(y - w)
(26)
of + 0
of
+
o 1 ooto
yy
w|l)
ly - wi
Proof. First, we derive the posterior MMSE estimator for under the uniform prior distribution on the surface of the sphere.
Lemma B.2 (MMSE estimator). When the means and variances are finite, the MMSE estimator for parameter a of observation is uniquely defined and is given by
3(v) R[mr/]]-
(27)
The likelihood of the observation y. u is
(yy x)² 2
(z w)
IP (y, w|2 x) P (y)zz) P (w|z) exp
exp
(28)
20
Let (2] be the uniform prior distribution on the surface of the sphere in k dimensions with center at and radius D. Let t x - z, and we denote the prior as π (t.) = 11 [[e]= Under such prior distribution,
Ply, w/xx & / exp (- (U [WV-
(y)=r)² 20
w))² (z (=-w)²
exp
1 (110- - z)d2.
(29)
203
Then, the posterior distribution is
y x)²
(x (1-w) w)2
PP (Ily w) x
exp
20
exp
203
1 [||-- (1l==2||D)((2dd
Rt
(x
y) 2
(x:
(w + t))²
exp
exp
1 [[|e|= (t)dt
20
20
Rk
1
+ of (w +t)
exp
2 o
of + 0
(y
(w t))²
exp
1 [||)= (t)dt.
(30)
of
+ oz
13

---

Under review as a conference paper at ICLR 2023
Therefore, the Bayes estimator of the posterior is
x²
y w|
1
1
oiy +  +
A
JR
R4
exp
2
of
02
of 03
1 (y
(w
+ +)))²)
exp
1 |=]|||=D]( D (t)dtdæ
2
of + 02
x By t
0 +t)
=A
exxp
JRL RR
of
03
of + 0
y
(w + +1)2
.exp
2
of + o2
11 [||t|= [[]] (t)dadt
D2 y
+
w +t)
(
(w++ + t))²
gh
of
+
02
exp
2
of + 02
(111) DD) (t)dt
B2 + ofw
o
(y
(w + t))
A'
texp
[(It)=D)(t)dt (t)dt
of +
1 + 2
Rk
2
of + 0z
By +
of
1
of
11
w|l)
D
(y wy-w), w)
(31)
of + 2
of
+
l
y
w
where A and A' are the normalization constants.
Then, will prove that #' is the minimax estimator. Specifically, prove that oLy + of is independent of y - w. Since both two r.v.s are normal, we only need to prove Cov + of w) 0, i.e.,
Cov Cov(ozv + oi w) = of Var(() + Var(w) =
= 0.
(32)
We then define the risk function
02y + of
I
y w
27
RE. (t) 2
(y - w) - x
of +
of +
2 I -1
11y w [] 11 - wil
o +
2090z
of
2
+
of + +o²
of o
1g T lyy w/l
D
E, N(.ooft))))
(3)
w
(33)
| by
w|))
y
w
Lemma B.3 (Minimax Theorem Marchand & Perron (002 the unique minimax estimator when
The unique Bayes estimator o" is also
max Rs- (t) = Re (t)dT(t).
(
(o7 11 -
-1) 27 w (g) w) -1) )
Let (L) A Ey ()-N((.o+
Tg- 118 wil)
Lemma B.4 (Berry 1990 When D/ D/ oz < VT.
max R'(t) = ('(()dm(t)
(34)
Based Lemma B.3 and Lemma B.4 o² is the minimax estimator.
14

---

Under review as a conference paper at ICLR 2023
Lemma B.1 reveals that when two Gaussian observations contain prior knowledge that their centers have a maximum distance, the optimal estimator is a linear combination of the observations, where the combining coefficient is related to the variances of the two observations, the maximum center distance, and the dimension of observations.
For Formulation
6
define random vector u, RR where for all (a. E X y,
((2) XY (y
u(x, -
(35)
VR(a.y)
and
((5)
XY (x, v)
ozz v) -
(36)
VR(z,8) y)
P (a.a) and wo vo y)
P (mar)
Their centers are uo(2,1) v)
According (7
have
/la.y)
R(z.y)
NNuo: L | (x)s)
and v N (No (xII) and problem 6
can be re-defined
min w(u,)) vo Iv - 3
max
[[ w
(37)
With Lemma
B.1 we derive Theorem
3.1
For Formulation 2 ( we can define a random vector u, v E RI where for all (r,z) E X' y.
u(x v) - v (x)
(38)
and
v(2, v) - ((.) (I,
(39)
Their centers u1 v) u ~
P
(T) XY (x.v) v) and vo v)
v
P (x. v). According to 99. we have
N(uo Zney
I I | 2o)) and o ~ N N(vo 2ng (xp))) and problem g can be re-defined as
min ((( oo Il -otoo-olll un P2DD2
max
E
u4 w |
(40)
With Lemma B.1 we derive Theorem
3.3
C INTERPRETATION OF THEOREM 3.1
Firstly, Remark 3.2 leads to that 01 ir+n
ay g D
Considering that |>||>) < D2
(ns+ar)) |
we have
RIDI
11.
Wyy
(41)
11 + ng 1+ (ng + XT|I
Bra
+ D
which is close to (6) in Tong et al. (2021).
A geometric explanation for Theorem 3.1 can be depicted in figure 3 where the entire space represents
all the distributions supported by X' x y. Two balls centered at the target and source distributions represent the empirical distribution sets P and Pag. In particular, the radii of the balls show the variances of the empirical distributions, which are inversely proportional to the sample sizes. The area with deeper color contains those paracentral empirical distributions, which have higher probability according to 7 In addition, the distance between the centers is determined by the task distance assumption. The minimax problem (6) is meant to find estimator Q which is closest the target distribution average, where the dash line implies the linear family of P(T) and P(S) Xy- When the target sample size increases, the blue ball will shrink. When distance D increases, two balls will be farther away. Then, ((T) would be closer to the target distribution, which implies a smaller coefficient on.
15

---

Under review as a conference paper at ICLR 2023
Q
p18)
Pxy
p
PXP)
S D2
Figure 3: A geometrical explanation of the minimax setting
6
Two balls centered at the underlying
distributions represent all possible empirical distributions.
D PROOF OF THEOREM 4.1 AND THEOREM
Without loss of generality, we assume E p [f(X) = 0. It can be easily guaranteed by deducting the average feature when computing the feature of each
We denote A1 ( and As - (f
D.1 REVISED FORMULATION
For revised Formulation
15 first, we give the expression of ÿ: as defined in 14 Tong et al. 2202
Σ 5(0 P(0) (
(v) (v) AT
(42)
Then, we can define a random vector us v E RdD where
u a WP (y)too)) (2T"()1)
(43)
and
(1) (2
v 2
(44)
Their centers are
V (1 (0..)/.))...VV//(/)0).. (
uo
A
(45)
and
Uy
(46)
With have
1 ~ N(u, - Laira Novo. nT
1 Lapa).
(47)
ns
Problem (15) can be re-defined
min w(u,r) mo:
max uo 222
[[ wi w|
(48)
and thus Theorem 4.1 is proved.
16

---

Under review as a conference paper at ICLR 2023
D.2 REVISED FORMULATION 2
For revised Formulation 2 15 we give the expression of h, as defined in 19
h,(
(P AT'
Σ W
(49)
Then, can define random vector u, RdV where
- V P (1)7))))))0))))-....)))))))))))) (BRFF/)).../////0A00///0.. (P
(50)
and
v
2
v (( (1)A3
(51)
Their centers are
V (1T)())))))))( T
uo v (1)
(52)
and
Pl (PDA v p( (1)A51())))
vo
(53)
With 7 we have
1
~ N(uo 2nT
Lliir NVo,
dapy
(54)
2ng
Problem
15
be re-defined
min w(u.o) uot 11 uall D2
max
E
u40 - w/l
(55)
and the optimal estimator is
a1- + 22.
(56)
Note that w here refers to the vector
VP (IA VV
Thus, Theorem 4.1 is proved.
E DETAILS OF ALGORITHM IMPLEMENTATIONS
E. ALGORITHM
Here we provide the details of the loss function (a, f.g) in line 4 of Algorithm the quantity D
and the quantity D2 in line 5. Our main procedures follow the results in Xu & Huang (020))
As for the zero-mean assumption of f", which results in zero-mean weights g, first define f(X) & f(X) Bgr [f(X) and g(Y) A g(Y) per (Y)).
Then, we define Â and Âg as the covariance matrices features on target samples:
A E pm f(X)fT (X)),
(57)
Ag - pcr) (Y)
(58)
17

---

Under review as a conference paper at ICLR 2023
In our implementations, all the computations in terms of the underlying distribution replaced by the corresponding empirical distributions. As proved in Xu & Huang (2020), minimizing
xd X (P()) ((7) 0 (( 9)
xv.
( (f
YIX (ular)
p(T) (v) (1 + (x)g(()) is equivalent to maximizing
H n (f.g) g)
-
E
te (X)g - tr())
(59)
XY
where: = T, S. Then, line in Algorithm || can be implemented by
(i f. Meanwhile, the distance bound D is also estimated from samples. As the simplest way, we let
- H + (H
(60)
D = D2 (D can actually be adjusted), where
D3 = (P @ ΦT)
Σ IPPT(v)) (v)
E Bpeny You
[f(x)]
E pro)
(f(X))
A7'
E Epery [f(x)] X1Y==
E Epcs: [f(x)]
X/Y==
x
vEy
(61)
E.2 ALGORITHM
Here we provide the details of the loss function L2(f, h1, h2) in line 2 of Algorithm 2 the quantity D and the quantity D2 in line 3.
Similarly, we define fi (X) A S(X) - E Eggn [f(X)) fx(X) & f(X) E pcs [X(]], h. (Y) a
h1 (Y)
E pm 19 (Y)), and h (Y) a hy( 1 (h2
Then the covariance matrices are
Á - E pgm f(
(62)
Å, A E prs) [f2(X) f (X)],
(63)
Åg. - E pm
(g (Y (r]g' (Y),
(64)
Ag A E B3o
و (Y (Y).
(65)
Still, all the computations in terms of the underlying distribution are replaced by the corresponding empirical distributions. Considering the local approximation that p(t (z. P P() (v) the
Hellinger distance loss function 22 (PXY- (f (.))) can be implemented by maximizing
(f,))
Σ
1 ((T) (z, (1) tr(( (r(AA
(66)
NEXWNE
Similarly,
(T
Σ NE
((S) (x) ur((As
(67)
Line 2 in Algorithm
2 can be implemented by
Le(f, h1, h22 (f, + (1(s) (f, 122).
(68)
We let D = =D² (D can actually be adjusted), where
D3
Σ
V P ( (v) (
pp(5) (v)
( (
NEY
Σ
V p(T) (3) A hi()) - v (A (v) he(
(69)
2
vEy
18

---

Under review as a conference paper at ICLR 2023
and
AT + Af
(70)
As t Afa-
(71)
Specifically, to compute the value of Bessel functions, we make the following approximations when needed. When < v. J2(a) ~ /vv When >> (x( ~ (2x (2x)
F INSTRUCTION FOR CODES
We provide code examples in "supplementary Suupleementrry.mattriilllppp In the folder "./cifar10", we provide the code examples on CIFAR-10 dataset for feature extraction (construct feature vectors.py .ypto, MM-x² algorithm (mmchi2.py) and MM-Hel algorithm (mmhel.py). Folder "'/office 31 (aticee--//feattue 31/feature" contains the features of Office-31 dataset. Folder /ooffce-31/miiiiie 31/mnmmmmmiiiix"""" contains the code examples for Office-31 dataset, including the feature extraction code (feature centureeexttatttcoooe 4096..t. MM-x2 algorithm (OC (oC_atod.vgg16.400 1ttbo__vgg6.44006.ch22,,,, 4096.chii..pp), and MM-Hel algorithm (oc.atol.vgg6664466hel)l))
19


Reviews:
[{"review_id": "wssSt8eRXd8", "reviewer": "Reviewer_snNr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper discussed the robust transfer learning through min-max principle. Specifically, this paper adopted chi^2/Hellinger distance to measure the joint distribution. Through min-max formula, this paper focused on the task reweighting approach such that to learn the source target weight, embedding and its downstreaming tasks. This paper is further empirically validated in standard computer vision benchmarks.\n", "summary_of_the_review": "This study investigates an important and challenging topic of estimating joint distribution similarities across heterogeneous tasks in the context of high-dimensional and complex data, the related theoretical analysis is further provided. However, there are several major flaws (such as significance, clarity) in the manuscript that prevent it from being accepted.\n\n### Update after rolling discussions\n\nBased on rolling discussions and checking others’ reviews, I am still not convinced. Thus I would keep my current score.  \n", "strength_and_weaknesses": "### Summary \n\nIn general, this study investigates an important and challenging topic of estimating joint distribution similarities across heterogeneous tasks in the context of High-dimensional data, the related theoretical analysis is further provided. However, there are several major flaws (such as significance, clarity) in the manuscript that prevent it from being accepted.\n\n### Pros\n\n1. This paper considered robust transfer learning through min-max principle, through distributional robust viewpoint (w.r.t. joint distribution), the key technical difficulty is to measure the joint distribution similarity in both raw data space and representation space. \n2. A novel task reweighting based approach is proposed and further validated on standard computer vision benchmark. (even tasks with different label spaces).\n\n### Cons\n\n1. In general there is a significant mismatch between claimed contributions and actual contribution, making this reviewer quite confused. \n2. The notations within the paper are seemingly rather unclear, making some parts quite difficult to follow.\n3. [Significance] Despite the intensive analysis, this paper still did not theoretically explain why this could ensure a transfer. I mean in terms of **sample complexity**. \n\n### Detailed comments on cons \n\n1. The title “robust transfer” is designed as the main goal. Throughout the whole paper, except the minmax formula, I could not find any sort of clear justification (either in theory or practice) to show the robust transfer is achieved.  Indeed, minmax formula can be regarded as a nature interpretation of robust, while this paper did not clearly present how robustness is learned. \n\n2. In abstract, several key points are highlighted: task similarity, boundness, worst-case expectation. I do have concerns and confusions on all these concepts \n\n- **Task similarity** I have no doubt that task similarity on the joint distribution is quite difficult, while I do think that chi^2 and Hellinger Distance are incorrect metrics in measuring task similarity in high-dimensional and complex data regimes. In experiments, the source and target task could have different distribution support, which clearly violates the assumption in Sec 2, and chi^2 distance will be arbitrarily large in this case. Thus I would think the whole analysis does not exactly match the proposed scenarios. To this end, I would recommend paper [1] through Wasserstein distance (this is more reasonable!) to properly measure the task similarity by considering the parameter and data similarity. In contrast, this paper did not consider (I mean theoretically) the influence of parameters in estimating the task similarity. This gap enables a clear mismatch between the theory and practice in the representation learning setting. Overall, compared with paper [1], this reviewer feels the solutions are not convincing in both theory and practice. \n\n- **boundness** It is quite natural that the source and target should be similar within certain levels of bounded distance. While in practice,the source and target are fixed, this assumption does not make strong sense for me. \n\n- **worst-case expectation** This concept makes me quite confused, the nature of min-max surely ensures the worst-case performance. However, I feel like this paper does not clearly prove/demonstrate the improvement of worst-case loss.\n\n3. Notation/concept clarifications\n\n- In Eq(3), how D is determined?\n- Eq(5) seems quite unclear for me because I could not understand why is the exponential form $exp(-d(.,.))$.\n- Eq(12) does not make sufficient sense for me, since source samples are much larger than target, and $|X|$ is also quite large, which will enable $\\alpha_1 \\to 1$. \n- Eq(13), $g(y)$ is quite wired, it is essentially the downstreaming predictor?\n- Eq(17), I feel like the disconnection between the analyzed theory and proposed loss. Why here the distribution could be decomposed as $P(x)Q_{Y|X}$? How is this related to the analysis?\n\n4. Comments on experiments. I would strongly suggest more analysis rather than numeric accuracy to justify the robust transfer is ensured. \n\n5. About general theoretical analysis in transfer learning. I would strongly suggest the analysis on the sample complexity, which is the key in understanding transfer learning. Unfortunately, the population level loss (and analysis within the paper) could not illustrate why the sample complexity in the target domain is improved. \n\n[1] An Information-Geometric Distance on the Space of Tasks. ICML 2021\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "q27Q-J0S57", "reviewer": "Reviewer_cYFo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies transfer learning from the aspect of the minimax principle. Accordingly, the strategies of minimizing the worst-case EPR based on $\\chi^2$-distance and Hellinger distance are proposed for transfer learning. The effectiveness of the proposed algorithms is supported by empirical results on several benchmark data sets.", "summary_of_the_review": "While applying the minimax principle to transfer learning is an interesting idea, I encourage the authors to address my concerns and comments before it is ready to publish.", "strength_and_weaknesses": "Strengths:\n\n1.  This paper investigates transfer learning as a minimax problem, which is interesting and novel. \n\n2. The proposed robust transfer learning algorithms are theoretically justified.\n\n3. The paper is well-organized\n\n\nWeaknesses:\n\n1. The claims made in the paper are not clearly verified. For example, in the abstract, it is said the similarity between the source and target domains is “difficult to be precisely captured”. However, after reading the paper, it is still not clear to me compared to existing quantities, how and why the $\\chi^2$-distance and Hellinger distance can “precisely” capture the similarity. Moreover, it is unclear to me why the boundedness assumption is weaker than existing assumptions and notions of similarity (e.g., $\\lambda$ and $H$-divergence in [1]). More specifically, I didn't any analysis of the generalization bound of the proposed method. Given that, how can we conclude whether the assumption is \\emph{milder } or the proposed estimator can \\emph{precisely} capture the similarity?\n\n2. The problem setting studied in this paper is supervised transfer learning, where the label information is available in the target domain. However, it seems that the related work in this field is not reviewed (i.e., theoretical analysis of supervised transfer learning).\n\n3. Another major concern comes from the empirical results, which are quite weak from my aspect. In particular, Office-Caltech and Office-31 are relatively easy tasks compared with more realistic datasets such as Office-Home, VisDA, and mention DomainNet. I believe these (at least the first two) are standard benchmarks for transfer learning and domain adaptation in this field. In addition, the baselines adopted in this paper are not strong enough. For example, in [2], the average accuracies on Office-Caltech and Office-31 are 93% and 89.6%, much higher than the accuracies reported in this paper. Lastly, it is not clear to me how the robustness is empirically verified in the paper\n\n\n[1] Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., & Vaughan, J. W. (2010). A theory of learning from different domains. Machine learning, 79(1), 151-175.\n\n[2] Wang, Q., & Breckon, T. (2020). Unsupervised domain adaptation via structured prediction based selective pseudo-labeling. In Proceedings of the AAAI conference on artificial intelligence.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "oHkQVWdcS2_", "reviewer": "Reviewer_Vs9B", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies transfer learning based on the min-max principle under the Chi-square distance and Hellinger distance. Compared with  KL-divergence distance, the paper uses two distance metrics to fix the difficulty of computing the expectations of the population risk. Under mild assumptions, the paper shows that the optimal estimation is to linearly combine the learning results of the source task and target task, which is consistent with previous work. Based on the theoretical results, the authors proposed an algorithm and evaluated its efficacy on CIFAR-10, Office-31, and Office-Caltech datasets.", "summary_of_the_review": "Although there are some questions I mentioned in the Weakness part that blocked me, the paper has its novelty and I tend to accept it. ", "strength_and_weaknesses": "Strength:\n1. As far as I know, this is the first paper to consider Chi-square distance and Hellinger distance in transfer learning, and it can efficiently compute the expectations of the population risk.\n2. The paper includes the analysis of the continuous data. Also, the paper provides the combining coefficient which is both theoretically optimal and computable from data.\n\nWeakness and Questions:\n1. I did not get the sense of robustness mentioned in the title. How to understand the robustness of this work?\n2. For equation (13), there is no intuition about using factorization. What is the insight here?\n3. The theoretical analysis considers a linear probing setting, while the algorithm uses fine-tuning. Here is a gap between the theory and experiments. I wonder whether we can fix the gap here.\n4. It seems the experiment results in Table 3 and Table 4 only gain marginal improvements compared with other state-of-the-art. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "R2RMEVOg80", "reviewer": "Reviewer_Axde", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors provided a general minimax framework to tackle the unknown similarity between the target and source in transfer learning problems. Some specific f-divergences were considered as the similarity measure and population-level best minimax estimators were derived. The best minimax estimators turned out to be a weighted average of the target estimator and source estimator, which is very intuitive. The numerical results demonstrated the effectiveness of the new method.", "summary_of_the_review": "In general, no matter judging from the problem, the method, or the results, this is an interesting paper. But there are indeed some places that are not well-explained in the paper, as I noted above. These issues need to be fixed and better clarified. It is hard for me to recommend the current version for acceptance. If all the concerns can be properly addressed, I may change my mind, but it's hard to say.", "strength_and_weaknesses": "Strengths:\n1) The minimax framework is quite general and explicit forms of the best estimators are derived;\n2) Good numerical performance of the new method;\n3) Clear writing;\n\nWeaknesses: \n1) The authors claimed that the bounded similarity is a weaker assumption, compared to the usual assumptions in literature. But I think the similarity condition (3) is not well explained. For example, parameter D seems to play a key role in both theory and algorithms. How large should D be to ensure the improvement on the target model compared to the target baseline (say, training models only using target data)? How should users pick D in practice? Your algorithms seem to require the input of D and some other unknown parameters to calculate the best weight, but there are no discussions on this. What are the choices of these unknown parameters in your numerical studies? More clarifications and discussions might be necessary. \n2) Some of the mathematical presentations are not very rigorous. For example, before equation (5), what's the meaning of \"the probability of the empirical distribution\"?  And the following $D$ represents the KL divergence? What does the notation $\\mathbb{P}(\\hat{P}; P)$ mean? Similar notations appear in other places too. Also, in Theorem 3.1 and some other places, what if $D \\rightarrow \\infty$ (i.e. the case of negative transfer)? The weight becomes negative?\n3) Some model settings and results are a little bit ad-hoc without enough interpretations. For example, the authors considered different underlying models ($(X,Y)$ distributions) when using different f-divergences. Is this just for the simplicity of theoretical analysis? Also, in Section 4, it seems that the neural network (NN) is only used as a dimension reduction tool. If this is the case, can the theoretical results be more general? \n4) There are only theories on the best population-level estimators, but no theories on the empirical estimators (which is actually what you used in practice).\n\nOther comments: \n1) Is it possible to extend the framework to the case of multiple sources?\n2) Can you provide more interpretations of how the results are connected to \"robustness\"? What is the benefit of using the minimax framework? As you mentioned, the minimax framework considers the worst scenario so it can be more robust to sources with different similarities. But I cannot get this message from your theoretical results and numerical studies. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject