Distribution-Informed Neural Networks for Domain Adaptation Regression
Authors: Jun Wu
Jingrui He
Sheng Wang
Kaiyu Guan
Elizabeth Ainsworth
Venue: NeurIPS 2022 Accept (2022)

Abstract: In this paper, we study the problem of domain adaptation regression, which learns a regressor for a target domain by leveraging the knowledge from a relevant source domain. We start by proposing a distribution-informed neural network, which aims to build distribution-aware relationship of inputs and outputs from different domains. This allows us to develop a simple domain adaptation regression framework, which subsumes popular domain adaptation approaches based on domain invariant representation learning, reweighting, and adaptive Gaussian process. The resulting findings not only explain the connections of existing domain adaptation approaches, but also motivate the efficient training of domain adaptation approaches with overparameterized neural networks. We also analyze the convergence and generalization error bound of our framework based on the distribution-informed neural network. Specifically, our generalization bound focuses explicitly on the maximum mean discrepancy in the RKHS induced by the neural tangent kernel of distribution-informed neural network. This is in sharp contrast to the existing work which relies on domain discrepancy in the latent feature space heuristically formed by one or several hidden neural layers. The efficacy of our framework is also empirically verified on a variety of domain adaptation regression benchmarks.

Distribution-Informed Neural Networks for Domain Adaptation Regression
Jun Wu, Jingrui He, Sheng Wang, Kaiyu Guan, Elizabeth Ainsworth University of Illinois Urbana- Trbana-Champaigh
jinguu,, jiengri, kaiyug, winswort)@illlnooss.............
Abstract
In this paper, we study the problem of domain adaptation regression, which learns regressor for target domain by leveraging the knowledge from relevant source domain. We start by proposing a distribution-inforee aims to build distribution-ave -wwabe relationship of inputs and outputs from different domains. This allows us to develop a simple domain adaptation regression frame- work, which subsumes popular domain adaptation approaches based domain invariant representation learning, reweighting, and adaptive Gaussian process. The resulting findings only explain the connections of existing domain adaptation approaches, but also motivate the efficient training of domain adaptation approaches with overparameterized neural networks. We also analyze the convergence and generalization error bound of our framework based on the distribution-infoo neural network. Specifically, our generalization bound focuses explicitly on the maximum mean discrepancy in the RKHS induced by the neural tangent kernel of iistributioniifoomme
neural network, which
neural network. This is in sharp contrast the existing work which relies on domain discrepancy in the latent feature space heuristically formed by one or several hidden neural layers. The efficacy of our framework also empirically verified on variety of domain adaptation regression benchmarks.
1 Introduction
Domain adaptation tackles the knowledge transfer from a source domain with adequate label informa- tion relevant target domain with little or no label information. It is shown 39 II 3
that the generalization performance on the target domain can be improved by leveraging the source domain
knowledge both classification [60] and regression [1]] settings.
In this paper, we focus on studying the domain adaptation regression problem, as classification can be naturally formulated regression K4] In the past decades, most domain adaptation approaches were developed using the following paradigms: learning domain invariant representation 42] 43 reweighting the source examples [T
deriving adaptive transfer kernel for Gaussian process 7 44]. More recently, modern practice for domain adaptation repeatedly demonstrates the benefit of considering overparameterized neural networks as the backbones for feature extraction [1 In this case, L-layer neural network would be manually divided into two parts: the feature extraction function (e.g., the first layers) and the prediction function (e.g., the left L layers). Then, the core idea of domain adaptation with overparameterized neural networks is to enhance the adaptation source and target domains in the hidden feature space learned by the feature extraction layers. Nevertheless, it is unclear how those overparameterized neural networks affect the convergence and generalization of domain adaptation approaches during model training. Moreover, the heuristic selection of feature extraction layers might lead to sub-optimal performance of domain adaptation approaches in practice, as the lower and higher layers an overparameterized neural network might encode different information [58]
37
20
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Compared with standard supervised learning, the key challenge of domain adaptation lies in the distribution shift between source and target domains. This indicates that the relationship between inputs and outputs might be different across domains. As illustrated in Figure 1 due to the distribution shift across domains, two similar input data points (e.g., I2 R 2z)) might have significantly different outputs (i.e., 1/2 * V3). This is also observed in recent work [52] To solve this problem, we propose a distribution- cerriuttion-iffrmeed neural network to build the unified relationship of inputs and outputs from different domains. This neural network can be explained as the integration of standard feature representation learning and input-oriented distribution representation learning. Then, a simple domain adaptation regression framework named DIND is proposed based on the distribution informed neural network, followed by the theoretical analysis on its convergence and generalization performance.
istributioo-informeed
In particular, we show that the distribution discrep- ancy of source and target domains can be defined by the evolution of intributiioninformen work during training. That is, we measure the distri- bution shift using the maximum mean discrepancy (MMD) 24 in the reproducing kernel Hilbert space (RKHS) induced by the neural tangent kernel of distribution-infone
Labeled source
neural net-
Unlabeled target
Local model uvvalation
Labeled target
neural network. Compared to previous works [37 20 MI our domain discrepancy measure has the following benefits. First, it can be estimated from the entire neural network, whereas previous works usually estimate the domain discrep- ancy in the pre-defined feature extraction layers. For example, (37] extracts features of input source and target examples and then estimates the discrepancy using MMD with standard kernels (e.g.. RBF ker- nel). Intuitively, it is a two-fold composition kernel of the neural kernel induced by the feature extractor and the standard kernel. Second, our domain discrep- ancy measure focuses on the training dynamics of the neural network (e.g., fo+so in sharp contrast to the previous works which mea- sure the distribution shift using the static state of the neural network (e.g.. fe(x) (x)). As shown in Figure D a source data point 23 might have similar output with both æ and 24 from target domain, when simply considering the trained neural network fa().. But we see that the neural network has similar evolution patterns at I3 and x44 but different patterns at 2i and æ1. That is, 23 is more likely be aligned with 24 after distribution minimization across domains, That explained why the training dynamics of the neural network can better identify the distribution shift across domains.
X1
X2 X3
X
Illustration of the domain adap- tation regression, where I I3 denotes two source data points and æ2, 244 denotes two tar- get data points. Here solid and dotted lines denote two consecutive states of the neural network around data points during training. (1) Similar data points (c.g., 22 and 23 from different domains) might have different out- puts. (2) Source data point T3 is more likely to be aligned with target data point x4, as they have similar local model evolution.
Figure 1:
fo fz(z)). This is
In addition, we show that in special cases, the training dynamic of the distribution informmo neural network can also explain the rationale of the existing domain adaptation techniques. To be more specific, we have the following observations. (1) At random initialization, our distribution-inforae neural network is equivalent to the adaptive Gaussian process 44 (2) Under gradient descent training, our domain adaptation framework based on the distributtonn-informee neural network can recover the prediction function of the reweighting domain adaptation approaches (0 13] (3)
When the distribution representation is shared by all the examples, our framework degenerates into
standard domain- invariant representation learning 37 20 contributions can be summarized as follows.
Compared to previous works, our
Unified Framework: We propose simple domain adaptation regression framework DINO based on the distribution distrbuttin-inffrmmed informed neural network, and then identify its connections to previous techniques, including domain invariant representation learning, reweighting and adaptive Gaussian process.
is shown that for the distribution-inforredd neural network with sufficient width, the convergence and generalization of our framework can be theoretically guaranteed. Different from previous deep domain adaptation theories, we propose to measure the distribution shift across domains using the training dynamics of the neural network.
Theoretical Results:

---

Empirical Performance: We experimentally investigate the performance of our DINO frame- work on a variety of domain adaptation regression benchmarks, and show its effectiveness over sfate-of-the-ar baselines.
The rest of this paper is organized as follows. We introduce the related work in Section and summarize the preliminaries of previous domain adaptation techniques in Section Section 4 shows the proposed domain adaptation regression framework, followed by its theoretical analysis. The experimental results are presented in Section 5 Finally, conclude the paper in Section C
2
Related Work
2.1 Domain Adaptation
Domain adaptation 5 52 53 63
improves the generalization performance of a learning algorithm on the target domain, by leveraging the knowledge from relevant source domain with adequate labeled data. There are three solutions to bridge the distribution gap between source and target domains. (1) Domain invariant representation: It maps the source and target examples into new feature space such that the distribution discrepancy between source and target domains can be explicitly minimized 442 37
61 60, [T]. (2) Reweighting: The core idea is to correct the difference between distributions by multiplying the prediction loss of each source example by a non-negative weight [4
43
20
IR ! (3) Gaussian process: It generalized the standard Gaussian process regression to the domain adaptation setting by learning an adaptive transfer kernel A 44 38 [50 By using the neural networks as the backbones, modern domain adaptation techniques have achieved state-of-te tite-offthhearr performance in a variety of real-world tasks 228 3. Nevertheless, little effort has been devoted to theoretically analyzing the domain adaptation techniques with overparameterized neural networks.
More recently, it has been revealed 1T] that there is gap between domain adaptation regression and domain adaptation classification problems. That is, one common loss function of domain adapta- tion regression approaches is mean square error (MSE), whereas domain adaptation classification approaches 37 20) often uses cross-entropy loss with softmax. Specifically, softmax changes the feature scales (i.e., Frobenius norm of feature matrix || where И is the hidden feature learned by the neural network), and the change of feature scales might lead to the performance degradation of domain adaptation [IO. Therefore, in this paper, we focus on the neural network with MSE loss for domain adaptation regression. Different from the existing domain adaptation regression approaches. e.g., domain invariant representation learning [10. reweighting [16], and adaptive Gaussian process Z. we develop the domain adaptation regression algorithms based on novel distribution-informeed neural network and derive the theoretical analysis regarding the convergence and generalization bound for our algorithms.
2.2 Overparameterized Neural Networks
It has been revealed from the perspective bias-variance decomposition [56] that the generalization етrоr of neural networks decreases with respect to the model complexity (e.g., number of neurons). The benefit of increasing the number of neurons has also been empirically confirmed in modern neural networks
52 59 Moreover, the convergence and generalization of overparameterized neural networks have been studied in the neural tangent kernel (NTK) regime [1
R 4 35
g 33
However, those works focus on the standard supervised learning with the assumption that training and testing examples follow the same distribution. In real scenarios, this assumption often fails due to the distribution shift (8 More recently. it is found that previous theoretical results be generalized to both federated learning 29 and multi-task learning [55. Our work fundamentally differs from them in that little label information is available in the target domain for our studied domain adaptation problem. This motivates us to explicitly analyze the domain discrepancy when using neural networks for domain adaptation.
3 Preliminaries
In this section, introduce the notations and background of different domain adaptation techniques.
3

---

3.1 Notation and Neural Network Architecture
Let X' and y denote the input space and output space. Following (6. we assume that the joint proba- bility distribution PP of any domain is drawn from probability distribution space 30 over X' x V. For domain adaptation regression, we have source domain with labeled examples {a.Th y2 1. J==i drawn
from joint probability distribution IPtc, and target domain with both label examples {2j tot 1st ni from Ipat and unlabeled examples (x"")) "M
Uj f==1
1j=1 (nigt < nigd drawn from marginal probability distri- bution pett The goal is to predict the outputs of unlabeled target examples by leveraging the label information from the source domain. In this paper, we consider an L-layer
neural X' C IR into the output E 3 C R. Here denotes
network fo(-), which maps the input all the parameters of f(-) (e.g.. 9o denotes the initialized parameters time stamp 0) and denotes the dimensionality of input data. In next section, we generalize the standard fully-connected neural network f-)-) to the distribution-infrreed neural network f(-) for domain adaptation. For notation simplicity, we let X {-]t Rie and yse {y; (re Meee for source examples. We use ;i=1
similar notations X and 2 for target examples. Then we use X = x U X/ denote all the labeled training inputs, and = y U y be the corresponding outputs.
3.2 Domain Adaptation Techniques
We first review the existing domain adaptation techniques. In Subsection show that these techniques can be explained in unified framework.
3.2.1 Domain Invariant Representation
The key idea domain invariant representation learning is to map the source and target examples into a new feature space where the distribution discrepancy across domains is explicitly minimized [20) 60) The objective function of this framework can be summarized as follows.
min E (1. - [L(fo(*), v)] + d (Irare p
where d (Ieree puet) denotes the domain discrepancy measure the hidden feature space learned by fo()), and L(.,)) )) is the loss function.
3.2.2 Reweighting
Reweighting aims to correct the difference between source and target distributions by reweighting the source examples, i.c., multiplying the prediction loss of every source example by non-negative weight 14
min (xx))) (w5"((z, v) L(fe(z), y)]
where T0 (a. v) pe (ur.y)
0 denotes the importance every source example. The impor-
p (a..))
tance ratio w w (x. y) can be estimated using kernel mean matching (KMM) 60 57 or adversarial learning [6].
3.2.3 Adaptive Gaussian Process
Following 77) the domain adaptation regression problem can also be formulated as the Gaussian process with an adaptive transfer kernel. Specifically, the adaptive transfer kernel can defined as
K K(X X T K(X X
T K(X X²" K(X X
K'
where 7 E [0, is trainable parameter explicitly indicating the relatedness of source and target domains, and K( ((..)) is a base kernel, e.g., RBF kernel. Then following the standard Gaussian process regression [9), the predictions of this adaptive Gaussian process over testing target data can be analytically derived.
4 A Unified Framework
In this section, we propose the distribution-informee neural network, and then present a simple domain adaptation regression framework.

---

4.1 cenibbuttioIInfommee Neural Network
An I--layer fully-connected neural network f()) can be written as fo(x) = (x)) where 0-L is the vector of parameters in the first I- layers and w is the parameter of the output layer (we assume the output layer has no bias). For single task, the neural network f(-) can model the relationship between input data and output label. But it might suffer in the domain adaptation scenarios due to the distribution shift. Thus, given a domain distribution IP E P and an input example x P. we propose to explicitly learn the iestribution-relate output gu, (Ip)x) = (IP) wa where (IP) the input-oriented distribution feature representation, and Wi is the parameter. Given finite basis examples ø1. I. ~ P. (x can be formally defined as follows.
(P) Σ Su 1,xx
(4)
where Bn E R is the coefficient (P) in the space spanned by (,)x))xx and indicates the similarity of input example x and basis example ã, Here (-, xx)xx denotes the feature map of I; in the NNGP kernel space induced by -nfe f(-). Therefore, o (IP) represents the distribution representation of P when example is observed in the domain associated with sampling distribution P.
One special case is when 3r.a, 1/n, it degenerates into the mean mapping of P in the kernel space, which commonly used for measuring the distribution discrepancy/similariyy [24]. Compared with this plain distribution representation E (', k.)x,, 42 (P) pays more attention to the region around æ with large By.s. for ā; E X. For example, for data points æ and x4 in Figure [ they would they are sampled from the same target domain.
share the same plain distribution representation, But it feasible differentiate them using the parametric distribution representation (P).
Generally, by taking both example x and its associated probability P as random variables, Bayes' theorem tells us that P(x, P) P(z) P(Plx) where P(-) denotes the probability of an observed event. The event involving can be described by observing its representation f(z); given æ, the event involving the irput-oriented distribution representation T (TP). This motivates us to develop a distribution-informeee neural network as follows.
(5) The intuition behind Eq. ( can be explained as follows. For domain adaptation, the feature representation of an input example is fomain-dependent. as two similar examples (2 x from different domains might have distinctive outputs (y²T * ) The input-oriented distribution representation _ (P) allows us to identify the source and target examples with similar inputs but distinctive outputs. By analyzing the distribution-infornee neural network at initialization and under gradient descent training, we propose two domain adaptation algorithms: DINO-INIT (see Subsection and DINO-TRAIN (see Subsection respectively.
f(x, IPP) fo( Iw, x) (deer (w) (D, w))) w (doce (2) Wig
4.2 Initialization
We start by studying the ceeribution-informen neural network initialization, e.g., all the parameters of distribution-informe neural network f(-) are initialized as standard normal variables. Inspired by
the connections between standard neural networks and Gaussian processes 15|
34
22
54] we show
that the distribution-infol informed neural network f(z PP) in Eq ( is equivalent to a domain adaptive Gaussian process, which explains the existing domain adaption regression algorithm 7
Lemma 4.1. Assume that all the parameters of Dnstributioniinforee neural network f(:) are initialized as standard normal variables, when the network width goes to infinity, the output function of fxx) in Eq at initialization is iid centered Gaussian process, ie., f(.) ~ N (O,KDA) with
DA (I,, (1', P')) = Kx(x, K
where Kx((s)) .) is the NNGP kernel induced by the neural network f(-) over input space X, and (-((--) -) is a distribution kernel identifying the similarity of two input-oriented distributions P and I' over distribution space P:
K x (I (
[x,x''''
B a. Br a Kx(
(6)
5

---

where n (I'') ) is the number of basis examples sampled from the distribution IP (P').
This lemma motivates us propose an adaptive Gaussian process regression algorithm named DINO-INIT with adaptive transfer kernel defined as K.DA, For notation simplicity, we represent the ceaributioo-informeed kernel K ((z, IP), (a', P)) as KCPA (x, x') in the following. In this case, we consider a noisy model yf - [(,, PT) + (r E {src, igtH), where the noise of follows zero-mean Gaussian (N0, of). Given labeled training examples (X,Y), we assume a prior Gaussian process
K
P(Y) = (N (DA) where K:DA(X, X) is a block matrix. That is, (X, = K
K
where KR == 1 (X ( (K22
== K (X,et xx denotes the kernel matrix of source (target) data, and KDD (K KX x!"))
denotes the kernel matrix across domains.
Then for testing target examples X their output can be inferred using the predictive distribution p( X x = (a, Σ). Similar to standard Gaussian process regression
40 the mean
and variance of the predictive distribution can be exactly calculated as follows.
p = K (X.', X
Σ = PA (X. x!=) - ICPA(x (X." ,XC xc (x."
2
T 0
where C' C= TK= (X, +
Here II denotes the identity matrix.
iig
This adaptive Gaussian process involves the following parameters: coefficient 32,z, in the kernel function KA and noise variance Tsre Otg. Following 77 we optimize these parameters by maximiz- ing the conditional likelihood p( Xare y=s)). It can be seen that p(),"'X X yst)
is also Gaussian process, i.e., p(y/5" X yss) (KDA (KRA + o mm yscc
12 + oight ) .CBA (KCR + In KPA)
Instantition of the coefficient B,,a;; It is notable that it can only optimize the coefficient 3z.ü, of Eq. 4 for training (source or target) examples. The coefficient of testing target examples X is unknown. To solve this problem, in this paper, we formulate the estimate of B,-,,; (r E {src, tgt)) as B2.z: = [x" a 2. "", where w" is a domain-specific parameter vector and denotes vector concatenation. Note that domain adaptation assumes that the domain labels of all input examples are known. As shown in Eq. 4. the input-oriented distribution representation can be learned from the domain-specific vector w" and the basis examples 27, ... , 3h from domain Here is determined by the domain label of the input example x. As illustrated in Algorithm D. training source (target) examples as the basis source (target) examples "r, ... Fr. As result, the input-oriented distribution representation learning of testing target examples X can be inferred by w and the pre- pre-defined basis target examples.
we use all the
4.3 Gradient Descent Training
Here we assume that the model parameters of the distribution-infommed neural network can be updated by gradient descent during training. Based on the distribution-infii neural network, we propose a novel DINO-TRAIN algorithm. Its objective function using mean square error is formulated as follows.
ripg
[())
27esro (flo", prt)
Ne.
1 2nigg
(fla)) pm)
)²0²² 2
"MMDapA (Par
where a E (0, 1) and 2 0 are hyperparameters to balance different terms. The first two terms are standard supervised learning losses over labeled source and target examples, and the third one is the empirical maximum mean discrepancy (MMD) [4 in the RKHS HDA induced by the neural tangent kernel of our cnaribution-informee neural network, i.e..
MMD2pA DA
(p pplt)
Vol(a?)
Vol
tst pli) plet)
(8)
Thane
Thgs
RDA
where Negs og te is the total number of target training examples. As shown in Theorem below, our framework empirically minimizes the upper error bound of the expected prediction enoor in the target domain.
4.5

---

Remark. In our framework of Eq. we measure the distribution shift using the training dynamics of the distribution-inforne informed neural network over the source and target examples. This is funda- mentally different from previous works 220 60 I associated with a two-stage domain discrepancy measurement. That is, those works would first learn the feature representation in hidden neural layers and then measure the domain discrepancy in the learned feature space. In the second stage, they usually require additional modules, c.g., auxiliary neural networks 10 21 [ or pre-defined distribution distances 337 Therefore, compared to previous works, our discrepancy measure Eq. (B) can not only unify the domain adaptation regression framework with neural networks, but also enable the theoretical convergence and generalization analysis in the following.
Before analyzing the dynamics of the distribution-informee neural network under Eq. we first introduce the following assumption.
Assumption 4.2. Given labeled data X X U X we assume Amin (0(X,X)) 0 and
Amin (K x)) > O, where (X, = Vof(X)
is standard neural tangent ker-
nel
31 induced by ((-) with infinite width, and K X) is the input-oriented distribution
kerne
(see Eq. 6 Here, Amin(-) denotes the smallest eigenvalue.
The assumption Amin (0(X, > 0 has been studied in previous works [3] 18 holds as long as no two inputs z; and Iy are parallel in real scenarios. The implication of this assumption is that the input data (e.g., images) would not be linearly changed by single factor, i.c., there is no two inputs such that x; = S 2 for some & E R.
Lemma 4.3. Given the labeled training data X = = U X and the basis examples { l==1 (fm Rud) denote the coefficient matrix of Kapi XX, (r E {src, Hgt we let Y E [R + mly)
where for Th
E X (k= 1,... Mar + mhggh ) its 1ht row (T), = pT " 3 ,,.
when 2k
E X [[], = [0.... 0., 2a P may
otherwise. Then, the assumption
Amin X)) > holds when T is positive definite.
Lemma 4.3 shows that in real scenarios, the assumption Amio (X, > can be guaranteed by the positive definiteness of the coefficient matrix T. We also empirically evaluate the smallest eigenvalue Amin (Kor (X,X) in Subsection 5.2
Theorem 4.4. For any coefficient 32,z. of the input-oriented distribution representation in Eq. F there exists 11" E R2 such thut for the infinitely-wide distribution-infuree neural network f(-)
trained under gradient flow with learning rate 17 < nÿ", the test prediction fo, (x!*)) of the domain adaptation regression in Eq. 0 over test target data X is
fo,(X!")) = (x?) - ODA(X", (000ax(XX"))00xxxxXx) (1-e (1-
where (DA(.))) is the distribution NTK, i.e., (D)(x, = 0(x,x') Kappx (IP, P Pa, x') and Č = diag aa/nsee........................ ".see
.aaeee/...////a aa/nnne (1 (1-a)/nl a)//i/n. ... (1-a)/mm is a diagonal matrix. Moreover.
Teom
4.2 when the network width goes to infinity, lim, Fo, (X!*)) converges to Gaussian process with mean p(X""") and variance E(X'"' x as follows.
under the assumption
)(x!")) = DDA (xx",) 0D)(XXX)-1x
"XX", = DA (x x + ODa (x.',, x)) (XX -(ODA (X.. , X) (x, X.) + h.c.)
(x,X.)
where " "+h.c." means "plus the Hermitian conjugate".
In addition, our result on generalization bound is given in the following theorem.
Theorem 4.5. Assume for any training example (x. y) E X x Y, we have (f(x,P) - y)² < Mo for some constant Mn 2 0. Let F be the hypothesis space induced by infinitely-wide neural networks f. Then, for any f E F and > O, with probability at least 1 - 8, the expected error in the target
Here we let (x, Kaai (B, F. x') brevity.
7

---

domain can be bounded as follows.
Efr( - a Σ (aut", y1 )
a whe
2 (Malf, pear) 1130)
+ Ma MMDeDA (P"T ,prrr) +
where 92 Rm (Her) + 2(1 Rin (Hagi) + My V (mer + mn/e) (opl1//) H, {(.
(f(x",P") - y²) FF is a set of functions (r E {src, (gr)), Ry. (H,, the Rademacher complexity of of Hr given ny examples, and M max nire. Mo)
Though domain adaptation theories 1 39 have been generalized to deep learning scenarios in recent years, neural networks are simply considered as an expressive feature extractor in modern generalization errors [60]. Nevertheless, recent works 66 36 reveal that the feature space learned by the neural networks might worsen the adaptation between source and target domains. As a comparison, Theorem 4.3 provides unified generalization error for neural network based domain adaptation, as it directly measures the domain discrepancy over input source and target examples in the RKHS induced by the distribution-informed NTK O
4.4 Discussion
4.4.1
Gaussian Process at Initialization
One special case of distribution-informeed neural network initialization is that when the coef- and 2x = 1 Σi= (+, xx)xx. it can be seen that 2x = i= (,, xx is the normalized mean mapping 22! of data distribution P in the NNGP kernel space. Then, the distribution kernel K x (IP, 1P,, x") in Eq. (6 can be explained as the inner product of the mean mappings of and IP. Moreover, using the definition of maximum
ficient Se.,,
(P)
mean discrepancy (MMD) 24]], it can be shown that Kaix (P, x') C-MMD (P.P')
(P)..)) where (-) denotes the empirical MMD in the NNGP kernel space, and n Σi Kx Kx (z)
MMDK,
7 Eij=KK' J.y). It implies that Ka (IP, (o, x') is negatively correlated with the popular MMD estimator. MMD measures the distribution distance using the difference of mean mappings of distributions, whereas Xa (P, PP||,r, x²) measures the distribution similarity via the inner product of the mean mappings of the distributions IP and in the NNGP kernel space.
The following corollary shows that the existing domain adaptive Gaussian process (7] can be explained as implicitly learning the distribution-infomme
representation in the kernel space.
Corollary 4.6. With the assumptions in Lemma 4.1 when 8z.s. 1/ll Ei (.))x,l the Gaus-
sian process induced by f(x) at the initialization would be equivalent to the adaptive Gaussian process in [7] over NNKP kernel.
4.4.2 Fully Trained Model
When there exist some labled target examples, the objective function of the reweighting domain adaptation techniques (see Subsection 3.2.2p can be rewritten as follows.
min
W
(f(x²)
vi )(
(f(x))) 5
(9)
27a
2
The following corollary holds that our framework Eq. in the function space.
can recover the reweighting approach Eq.
Corollary 4.7. With the assumption in Theorem our framework Eq. 7 with iistribbutioninnforrer
neural network i(.) can recover the popular reweighting domain adaptation approach Eq. in the function space.
Besides, we show that when the distribution representation is shared by all the examples, our framework can be naturally degenerated into the Nomain-invariaal representation learning 37

---

Methods
C-N
C S
N -
N
-
- N
Avg.
NNGP 30
2.041 10001
1.823
0.445 4:0002
0.624
0.197 400002
0.459.0.000
0.932
NTKGP
2
1.34540.0e
1.227 +00000
0.323 4-0.02
0.529 0.0004
0.248 +00001
0.425 0.002
0.683
AT-GP
F
0.194
20.002
0.104::0.01
0.252-20.005
0.
0.189
0.186
TL-NTK
33
0. 0.164
0.124 20.00
0.24220.002
0. 0.125
0.197 0.004
0.181
DINO-INIT (ours)
0.128
0.233
0.114::0000
0.227
0.112 0.0022
0.181
0.166
DINO-TRAIN (ours)
0.127 20.002
0.240
0.127 +00000
0.243
0. 4:0001
0.194
0.177
Table 1: Results of domain adaptation regression on dSprites
Methods
RL - RC
RL T
RC + RL
RC + T
T RL
T RC
Avg.
NNGP
1.000
2..08
0.515 0.356-10005
0.367 1:0001
0.324=20.004
0.386
NTKGP
0.396:0000
0.365 20.00
0.200
0.390
0.390
0.354_o.0u
0.349
AT-GP
7
0.214
0.209
0.227 +0010
0.198
0.2364.0000
0.249
0.222
TL-NTK
38
0.200 0.206
=0.000
0.213
0.197 n.000
0.2264:0.0m
0.218
n..0n
0.210
DINO-INIE INIT (ours)
0.204-+0.001
0.185_20.me 200006
0.185
0.207
00000
0.182
n..001
0,212 0.000
0.201
DINO-TRAIN (ours)
0.193 0.000
0.1 0,003
0.207 0.000
0.188 0,002
0.226
0.21
0,001
0.204
Table 2: Results of domain adaptation regression on MP13D
IGA DAY
DIAO-RIT
0:10
Sayer
6.3
0.74
609
5
RC
Number Layes
Damahs Acupoation Taska
(a) dSprites
(b) MPI3D
(a) Number of layers
(b) Discrepancy
Figure Model comparison
Figure 3: Model Analysis
Corollary 4.8. Under mild conditions, our framework Eq. with distribution distribution-informeed neural network f(.) can recover the standard domain invariant representation learning in Eq. 4. where the domain discrepancy measure dl(,-) is instantiated with MMD in RKHS induced by standard NTK θ
5
Experiments
Data Sets: We use three domain adaptation regression benchmarks. Following 10.,
we use two image data sets: dSprites 40 and MP13D 233 Specifically, dSprites has 737,280 images from three domains: Color (C), Noisy (N) and Scream (S). For each image, it has three regression tasks, i.e., predicting three factors of variations (scale, position X and Y). MPI3D contains over 3M images from three domains: Toy (T), Realistic (RC) and Real (RL). It is shown I0 that for each image, involves two regression tasks, i.c., predicting three factors of variations (position and Y). In addition, we also use a plant phenotyping data set. It predicts diverse traits (e.g., Nitrogen) of plants related the plants' growth using leaf hyperspeetral reflectance. Here we consider the following two domains 46 Maize (M) and Maize Maize_UNL _NL (MU). In our case, the task is to predict the Nitrogen content of maize using the leaf hyperspectral reflectance (see Appendix A.11 for more details).
Baselines: In the experiments, we consider the following baseline methods. (1) Plain Gaussian
process:
NNGP 34 and NTKGP (25) (2) Domain adaptive Gaussian processes: AT-GP 7 and
TL-NTK 3X (3) Deep domain adaptation methods: DAN
37 WANN [6 and RSD
10
Implementations: In the experiments, our algorithms are implemented using L-layer (L 6) fully-comnected neural network with ReLU (see Appendix A. for more details). The induced NNGP and neural tangent kernels induced can be estimated using the Neural Tangents package [1] In addition, we set a = 0.5 and μ = 0.1 for -INO-TRAIN.
5.1 Results
Table I Table Table 3 provide the domain adaptation regression results on dSprites, MPI3D and Plant Phenotyping data sets. Following [10]. we report the Mean Absolute Error (MAE) between the predicted outputs and the ground-truth outputs the target test set (the best results
indicated in bold). It is observed that our method DINO achieves competitive performance over

---

the Gaussian process baselines. The weakly- trained DINO-INIT slightly outperforms DINO-TRAII, in some cases. This observation is consistent with previous work [33] The relatively unstable
performance of DINO-TRAIN might be caused by the
large covariance of Gaussian process with NTK [25]. In addition, we also compare DINO with domain adap- tation methods instantiated by overparameterized neu- ral networks. WANN 16 reweights the source exam- ples, and DAN d and RSD [0A learn the domain invariant representation. The performance compar- ison in Figure 2 confirms the effectiveness of our DIND approach over those state-of-the-ar baselines.
Methods
M + MU
MU - M
NNGP F41
0.562.10001
0.672:10010
NTKGP 255
0.562.10.04
0.702:10.000
AT-GP 7
0.308 20,00
0.593
TL-NTK 38
0.31 6
0.488
PINO-INIT (ours)
0.3 1.316.am 16
0.645
DINO-TRAIN (ours)
0.3
0.443
Table 3: Results on Plant Phenotyping
5.2 Analysis
Ablation Study: We investigate the im- pact of input-oriented distribution represen- tation in our DINO framework. We consider the following variants. DINO-INIT (DINO- TRAIN) w/o distribution feature: it would not utilize the input-oriented distribution representation, i.c., 9w, (1|x)) 1 for all X. DINO-INIT (DINO-TRAIN) w uni- form distribution feature: the coefficient $2.2. 1/n a constant. The results are given in Table E We observe that the distribution representation improves the domain adaptation performance. Compared globally shared distribution representation with Bg.e, = 1//n, our para- metric Φ (P) in Eq. 4 encourages to learn the local distribution representation around the input data point x. The empinical results in Table confirmed the efficacy of the input-oriented distribution representation.
Methods
C-N
DINO-INEE w/o distribution feature
0.189
DINO-INIT w uniform distribution feature
+000!
DINO-INIT
0.128
DINO-TRIAN w/o distribution feature
0.166
DINO-TRIAN uniform distribution feature
0.147
DINO-TRIAN
0.127
Table 4: Ablation study on dSprites
Impact of Model Architecture and Discrepancy Measure: Figure Ba shows the results of DINO with different number of neural layers L on dSprites (C - N). The results indicate that the performance of DINO positively related to the expressiveness of NTK and NNGP kernel induced by the neural network. In addition, we compare the proposed MMDepa (:--) with MMD-RBF (i.e., MMD with standard RBF kerne)) in domain adaptation regression, where MMD-RBB is estimated in the feature space learned by different neural layers (see Appendix I for more details). As shown in Figure 3b the results on Plant Phenotyping data set reveal that the selection of feature space affects the model performance when using MMD-RBF, and our proposed MMDepu (---) outperforms the MMD-RBF
Positive Definiteness of X XX; We report the smallest eigenvalue of the distribution kernel K x (X, X) over the training examples on Plant Phenotyping data set. The smallest eigenvalues of Kap (X, X) in DINO TRAIN after training are 3e 6 and 8e 7 on M -> MU and MU + M, respectively. This empirically confirms the positive definiteness of Ka x (X, X) in DINO-TRAIN.
6 Conclusion
In this paper, we study the domain adaptation regression by proposing a distribution-inforree neural network. Our domain adaptation framework with iestrbution-infoomee neural network subsumes the existing adaptation approaches based on domain invariant representation, reweighting, and adaptive Gaussian process. We also analyze the convergence and generalization bound of our framework. The experiments demonstrate the effectiveness of our DIND framework over state-of oftte--f-thhann the-art baselines.
Acknowledgments and Disclosure of Funding
This work is supported by National Science Foundation under Award No. 194-194,,03, IIS- 2117902, IIS-2137468, and Agriculture and Food Research Initiative (AFRI) grant no. 2020-670211 32799/project accession no. 1024178 from the USDA National Institute of Food and Agriculture. The views and conclusions are those of the authors and should not be interpreted as representing the official policies of the funding agencies or the government.
10

---

References
[1] David Acuna, Guojun Zhang, Marc T. Law, and Sanja Fidler. f-domain adversarial learning: Theory and algorithms. In International Conference on Machine Learning, pages 66-75. PMLR. 2021.
[2] Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. convergence theory for deep learning via over-parameterizaaion. In International Conference on Machine Learning, pages 242-252 PMLR, 2019.
[3] Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of op- timization and generalization for overparameterized two-layer neural networks. In International Conference on Muchine Learning, pages 322-332. PMLR, 2019.
[4] Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in Neural Information Processing Systems, 32, 2019.
[5] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jen- nifer Wortman Vaughan. A theory of learning from different domains. Machine Learning. 79(1):151-175, 2010.
[6] Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classifi- cation tasks to new unlabeled sample. Advances in Neural Information Processing Systems, 14:2178-21188 2011.
[7] Bin Cao, Sinno Jialin Pan. Yu Zhang, Dit- Yan Yeung, and Qiang Yang. Adaptive transfer learning. In proceedings of the AAAI Conference on Artificial Intelligence, volume 24, 2010.
[8] Yu Cao, Meng Fang, Baosheng Yu, and Joey Tianyi Zhou Unsupervised domain adaptation on reading comprchension. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7480-7487, 2020.
(9] Yuan Cao and Quanquan Gu. Generalization bounds of stochastic gradient descent for wide and deep neural networks. Advances in Neural Information Processing Systems, 12:10836-10846. 2019.
[10] Xinyang Chen, Sinan Wang, Jianmin Wang, and Mingsheng Long. Representation subspace distance for domain adaptation regression. In International Conference on Machine Learning, pages 1749-1759. PMLR, 2021.
[11] Corinna Cortes, Yishay Mansour, and Mehryar Mohri. Learning bounds for importance weighting. Advances in Neural Information Processing Systems, 10:442-450, 2010.
[12] Corinna Cortes and Mehryar Mohri. Domain adaptation and sample bias correction theory and algorithm for regression. Theoretical Computer Science, 519:103-126, 2014.
[13] Corinna Cortes, Mehryar Mohri, and Andrés Munoz Medina. Adaptation based on generalized discrepancy. The Journal of Machine Learning Research, 20(1):1-30, 2019.
[14] Corinna Cortes, Mehryar Mohri, Michael Riley, and Afshin Rostamizadeh. Sample selection bias correction theory. In International Conference on Algorithmic Learning Theory, pages 38-53. Springer, 2008
[15] Alexander G. de G. Matthews, Jiri Hron, Mark Rowland, Richard E. Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. International Conference Learning Representations, 2018.
[16] Antoine de Mathelin, Guillaume Richard, François Deheeger, Mathilde Mougeot, and Nicolas Vayatis. Adversarial weighting for domain adaptation in regression. In IEEE 33rd International Conference Tools with Artificial Intelligence (ICTAI), pages 49-56. IEEE, 2021.
[17] Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International Conference on Machine Learning, pages 1675-1685. PMLR, 2019.
11

---

[18] Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. In International Conference on Learning Representations, 2019.
[19] Zhekai Du, Jingjing Li, Hongzu Su, Lei Zhu, and Ke Lu. Cross-domain gradient discrepancy minimization for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3937-3946, 2021.
[20] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The Journal of Machine Leurning Reseurch, 17(1):209--0030, 2016.
[21] Zhiqiang Gao. Shufei Zhang, Kaizhu Huang, Qiufeng Wang, and Chaoliang Zhong. Gradient distribution alignment certificates better adversarial domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8937-8946, 2021.
[22] Adrià Barriga-AAonso, Carl Edward Rasmussen, and Laurence Aitchison. Deep convolutional networks as shallow gaussian processes. In International Conference on Learning Representa- lions, 2019.
[23] Muhammad Waleed Gondal, Manuel Wuthrich, Djordje Miladinovie, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. On the transfer of inductive bias from simulation to the real world: new disentanglement dataset. Advances in Neural Information Processing Systems, 32:15740-155
2019.
[24] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
[25] Bobby He, Balaji Lakshminarayanan, and Yee Whye Teh. Bayesian deep ensembles via the neural tangent kernel. In Advances in Neural Information Processing Systems, volume 33, pages 1010-1022, 2020.
[26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[27] Fumio Hiai and Minghua Lin. On an eigenvalue inequality involving the hadamard product. Linear Algebra and its Applications, 515:313-320 2017.
[28] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International Conference on Machine Learning, pages 1989-1998. PMLR, 2018.
[29] Baihe Huang, Xiaoxiao Li, Zhao Song, and Xin Yang. Fl-ntk: A neural tangent kernel-base framework for federated learning analysis. In International Conference on Machine Learning. pages 4423 4434. PMLR, 2021.
[30] Jiayuan Huang, Arthur Gretton, Karsten Borgwardt, Bernhard Schölkopf, and Alex Smola. Correcting sample selection bias by unlabeled data. Advances in Neural Information Processing Systems, pages 601-608, 2006.
[31] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: convergence and generalization in neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 8580-8589, 2018
[32] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25:1097- 1 1105, 2012.
[33] Jaehoon Lee, Samuel Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein. Finite versus infinite neural networks: an empirical study. Advances in Neural Information Processing Systems, 13:15156-15172 2020.
12

---

[34] Jachoon Lee, Jascha Sohl-dickstein, Jeffrey Pennington, Roman Novak, Sam Schoenbolz, and Yasaman Bahri. Deep neural networks gaussian processes. International Conference on Learning Representations, 2018.
[35] Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in Neural Information Processing Systems, 32:8572-8583, 2019.
[36] Hong Liu, Mingsheng Long, Jianmin Wang, and Michael Jordan. Transferable adversarial training: A general approach to adapting deep classifiers. In International Conference on Machine Learning, pages 4013-4022. PMLR, 2019.
[37] Mingsheng Long, Yue Cao, Jianmin Wang. and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pages 97-105, PMI.R, 2015.
[38] Wesley Maddox, Shuai Tang, Pablo Moreno, Andrew Gordon Wilson, and Andreas Damianou. Fast adaptation with linearized neural networks. In International Conference on Artificial Intelligence and Statistics, pages 2737-2745. PMLR, 2021.
[39] Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. In 22nd Conference on Learning Theory. 2009.
[40] Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentangle- ment testing sprites dataset. http://gghhbccoommlemmmmll//llth 2017.
[41] Roman Novak, Lechao Xiao, Jiri Hron, Jaehoon Lee, Alexander A. Alemi, Jascha Sohl- Dickstein, and Samuel S. Schoenholz. Neural tangents: Fast and easy infinite neural networks in python. In International Conference on Learning Representations, 2020.
[42] Sinno Jialin Pan, Ivor W Tsang, James T Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE Trunsactions on Neural Networks, 22(2):199-210, 2010.
[43] Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
[44] Neeti Wagle and Eric W Frew. Forward adaptive transfer gaussian process regression. Journal of Aerospace Information Systems, 1444):244-231 2017.
[45] Haoxiang Wang. Han Zhao, and Bo Li. Bridging multi-task learning and meta- meta-learning: Towards efficient training and effective adaptation. In International Conference on Machine Learning, pages 10991-11002 PMLR, 2021.
[46] Sheng Wang, Kaiyu Guan, Zhihui Wang, Elizabeth A Ainsworth, Ting Zheng, Philip A Townsend, Kaiyuan Li, Christopher Moller, Genghong Wu, and Chongya Jiang. Unique contributions of chlorophyll and nitrogen to predict crop photosynthetic capacity from leaf spectroscopy. Journal of experimental botany, 72(2):3413354 2021.
[47] Zirui Wang, Zihang Dai, Barnabás Póczos, and Jaime Carbonell. Characterizing and avoiding negative transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1129-11132. 2019.
[48] Colin Wei, Jason D Lec, Qiang Liu, and Tengyu Ma. Regularization matters: Generalization and optimization of neural nets their induced kernel. Advances in Neural Information Processing Systems, 32, 2019.
[49] Christopher K Williams and Carl Edward Rasmussen. Gaussian processes for machine learning, volume 2. MIT press Cambridge, MA, 2006.
[50] Jun Wu, Elizabeth A Ainsworth, Sheng Wang, Kaiyu Guan, and Jingrui He. Adaptive transfer learning for plant phenotyping. In Third International Workshop on Machine Learning for Cyber-Agricultural Systems, 2021.
13

---

[51] Jun Wu and Jingrui He. Continuous transfer learning with label-informed distribution alignment. arXiv preprint arXiv:2006.0322 2020.
[52] Jun Wu and Jingrui He. Indirect invisible poisoning attacks on domain adaptation. In Proceed- ings of the 27th ACM SIGKDD Conference Knowledge Discovery & Data Mining, pages 1862-1862, 2021.
[53] Jun Wu and Jingrui He. Domain adaptation with dynamic open-set targets. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2039-2049, 2022.
[54] Greg Yang. Wide feedforward or recurrent neural networks of any architecture are gaussian processes. In Advances Neural Information Processing Systems, volume 32, 2019.
[55] Greg Yang. Tensor programs ii: Neural tangent kernel for any architecture. arXiv preprint arXiv:2006.14548, 2020.
[56] Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. Rethinking bias-varianes variance trade-off for generalization of neural networks. In International Conference on Machine Learning, pages 10767-10777. PMIR, 2020.
[57] Yao-Liang Yu and Csaba Szepesvári. Analysis of kernel mean matching under covariate shift. In International Conference on Machine Learning, pages 1 1147-1154, 2012.
[58] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818-833. Springer, 2014.
[59] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference Learning Representations, 2017.
[60] Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. International Conference on Machine Learning, pages 7404-7413. PMLR, 2019.
[61] Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. In International Conference on Machine Learning, pages 7523-7532. PMLR, 2019,
[62] Yao Zhou, Lei Ying, and Jingrui He. MultiC?: an optimization framework for learning from task and worker dual betcrogeneity. In Proceedings of the 2017 SIAM International Conference on Data Mining, pages 579-587. SIAM, 2017.
[63] Yao Zhou, Lei Ying, and Jingrui He. Multi-task crowdsourcing via an optimization framework. A ACM Transactions on Knowledge Discovery from Data (TKDD), 1-26: 2019.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information how to answer these questions. For each question, change the default [TODO] to [Yes]. [No], or [N/A]. You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data proprictary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
14

---

1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Abstract and Section |
(b) Did you describe the limitations of your work? [Yes] See Appendix A.10
(c) Did you discuss any potential negative socictal impacts your work? [No] Our work does not involve any potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs all theoretical results? [Yes] See Appendix A.2
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material as a URL)? [Yes] See the supplemental material.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Section 5 and Appendix [A.IT]
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix AA.1
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "hkWzQP9g9yY", "reviewer": "Reviewer_ezF7", "summary": "This paper proposes a framework called DINO for domain adaptation regression, which is based on distribution-informed neural networks. This framework can be theoretically related to a) domain invariant learning, b) source data reweighting, and c) adpative gaussian processes.  Authors show that convergence of such networks can be guaranteed. Also, the framework uses the training dynamics of the network when measuring the distribution shift between domains. Experiments were conducted on dSprites, MPI3d, and Plant Phenotyping - two of which are image datasets. Overall, both DINO-INIT and DINO-TRAIN show strong performances in all three datasets.", "questions": "* In this paper, the authors use an L-layer fully-connected network for all experiments. Does the theoretical analysis and method presented in this paper naturally extend to diverse architectures, such as CNNs and/or transformers?\n* It seems like one underlying assumption is that the neural network should be over-parametrized. But can we say that a 6-layer fully-connected network is over-parametrized for dSprites or MPI3D, both of which are image datasets?", "limitations": "Limitations are mentioned in the Appendix. Authors note that the framework is based on single-source domain adaptation. ", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n* DINO presents a unified framework that is well connected with a few previous works. In other words, the authors show that methods proposed in other works may be interpreted as a special instantiation of the DINO framework proposed in this paper.\n* The authors theoretically verify that the proposed framework is able to converge.\n* Overall, the paper is well written.\n\n**Weaknesses**\n* not clear what the difference between DINO-INIT and DINO-TRAIN is. First mention of DINO is in the intro, while the second mention is in Section 5. Explanation of DINO-INIT and DINO-TRAIN are in the Appendix (A.10), but should really be part of the main paper (or at least, it should be mentioned that DINO-INIT/TRAIN are explained in the Appendix). \n* Why is it that DINO-INIT outperforms DINO-TRAIN in both dSprites and MPI3D? More insights could be provided for this.\n* I think the paper could be improved by reducing the amount of theoretical analysis in the main paper (could be moved to the Appendix), as I feel that it takes away from the main points of the paper. \n* It is not clear how DINO would be generalized to more practical datasets/network architectures. Overall, experiments that demonstrate practicality would make the submission much stronger.\n\n**Other**\n* L258 Typo: (8) can **not** only unify ...\n* Some equations leak into the margins. For example, equation after Lines 229, 285, and 291. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dL37CsTTGVX", "reviewer": "Reviewer_WMY4", "summary": "In this paper, the authors propose a distribution-informed neural network for domain adaptation regression, called DINO.  The proposed distribution-informed neural network can be explained by existing domain adaptative Gaussian process. The authors also analyze the convergence and generalization bound of DINO. Experiments are done to show the effectiveness of the proposed DINO framework.", "questions": "1.The paper is specifically written for domain adaptation regression. However, to this reviewer, the technical contribution is not specific to regression. The authors claim that classification is naturally formulated as regression. Thus, the reviewer wonders how this method performs in those classification tasks (there are quite a lot SOTA UDA papers for classification, the authors may want to compare with some recent ones.).\n\n2.To this reviewer, the related work section should be re-organized and further improved. Compared with discussing the big picture of domain adaptation, the authors should directly focus on the regression works on domain adaptation or transfer learning. The authors may want to add two more paragraphs, one discussing domain adaptation regression works and another one discussing the GP based adaptation setting, as these two research lines are closely related to the current paper.\n\nMore importantly, it is more convincing to discuss how those closely related works, e.g., [14] and [11], differ from the current paper, instead of just listing them out. \n\n3.To this reviewer, the setting studied in this paper is different from the settings of [14] and [11], where the former [14] only has unlabeled target data and the latter [11] only has limited labeled target data. This paper seems to be a mixed setting of the above two settings. In this sense, the reviewer has the following three concerns:\n\n(1).  with both labeled and unlabeled target data, existing semi-supervised learning methods (here I give a reference example [ref1]. The authors may want to check the more recent SOTA methods). The authors should consider to show the superiority of the current paper over this kind of methods.\n \n[ref1] Kostopoulos, Georgios, et al. \"Semi-supervised regression: A recent review.\" Journal of Intelligent & Fuzzy Systems 35.2 (2018): 1483-1500.\n\n(2). as the paper mixed the setting of [11] and [14], the reviewer expects the method should clearly outperforms [11] and [14] as it uses more target information. However, from Tabel 3, the methods is not as good as [11] at the task M -> Mu.\n\n(3). How the scale of labelled target data affects the final performance? Furthermore, how the ratio of labelled and unlabeled target data matter?\n\n4.The empirical evaluations are not very convincing. In Table 1 and 2, the DINO is only slightly better than those baselines. In table 3, DINO is not as good as [11]. Note that [11] is a 12-years ago paper, the authors should consider more recent adaptive GP methods for comparisons.\n\n\n\n######################### After rebuttal ##################\nI would like to thank the authors' detailed response, which well address my concerns. I am willing to increase my score accordingly. \n\n", "limitations": "The limiatation is stated in Appendix, mainly for the possible future working direction. It is more interesting to discuss the limitations on the technical part of the method. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the paper is well-written. The idea of using domain adaptative Gaussian process to explain the neural network feature extractor is interesting. The distribution shift measure using the training dynamics of the distribution-informed neural network is a novel and sound point. However, I also have the following comments as stated below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "V-mP79vWMCu", "reviewer": "Reviewer_cqod", "summary": "This paper proposes a domain adaptation (DA) regression framework based on the distribution-informed neural network, which connects to previous techniques, including domain invariant learning, reweighting, and adaptive Gaussian process. It also theoretically analyzes the convergence and generalization bound of the framework. The method is evaluated on three DA regression datasets and outperforms some state-of-the-art methods.", "questions": "Besides the points listed above, I have some questions:\n1. Can you clarify why the method proposed in section 4.2 is called DINO-Init? How does it relate to section 4.3? In experiments, I find that DINO-Init and DINO-Train may outperform each other in different cases. Can you explain this? How could we choose between these two algorithms in practice?\n2. Can you clarify what the ‘training dynamics’ mean in Line 251? Why is it called ‘training dynamics’? Why is it better than previous measurements?\n3. In Line 238, why do we use all examples as the basis examples? Would this prevent the method from scaling to larger datasets? How do the different choices of basis example sets influence the method?\n4. In practice, how should we choose hyper-parameters such as \\alpha and \\mu in Equation 7?", "limitations": "The authors addressed the limitation that the framework may not handle multi-source settings. I think another limitation is that it is unknown whether this paper can generalize to commonly-used deeper networks with diverse architectures.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. It is interesting and inspiring to explore the ideas of distribution-informed neural networks in the DA regression problem.\n2. This paper provides theoretical analyses on the convergence and generalization of the proposed DA regression framework. Different from previous theories, distribution shifts are measured using the training dynamics of the neural network.\n3. The proposed method outperforms some state-of-the-art DA regression methods on three benchmarks.\n\nWeaknesses:\n1. I think this paper is generally not clearly written and hard to follow, especially the most important sections 4.2 and 4.3. The titles of the subsections, such as ‘4.2 Initialization’, are confusing. I think it also needs more transitional sentences to lead or summarize the sections and to clarify the motivations.\n2. The proposed framework is mainly evaluated on a shallow fully-connected neural network. It is unknown whether it can be extended to other architectures (such as CNN or Transformers), deeper models, and models already pre-trained from large datasets since these models may be more commonly used in practice.\n3. The method is mainly evaluated on some toy datasets, and I think it needs some experiments on more realistic situations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HjY2zvtbpL2", "reviewer": "Reviewer_2Ab3", "summary": "The paper proposes to use a distribution-informed neural network for domain adaptation regression. The input-oriented feature representation is defined based on its distance to other examples in the same domain, thereby achieving distribution-conditioned representation of the input data. Theoretical analysis shows that the proposed distribution-informed neural network is equivalent to domain adaptive Gaussian Process. \nEmpirical results on dSprites and MPI3D demonstrate that the proposed approach outperforms Gaussian Process, domain adaptive Gaussian Process and deep domain adaptation methods.", "questions": "Q1-2: Please refer to my question in the weakness section.\n\nQ3. In Eq. 5, it is not clear how $f_\\theta(x)=\\phi_{\\theta<L}(x)^{T} w$ could represent $P(x)$. The probabilistic formation is not enforced in the training process.", "limitations": "No", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. I think the proposal of distribution-informed neural network that is aware of the input-output training dynamic is novel. Although not mentioned in the paper, the proposed approach is somewhat related to self-attention where the model encodes each individual input based on its relations with other inputs. The illustration in Fig. 1 on how \"local model evolution\" relates to domain adaptation is insightful.\n2. The proposed approach is theoretically grounded and the paper presents a unified framework for analyzing domain discrepancy that is connected to adaptive Gaussian process, sample re-weighting and domain-invariant representation learning.\n3. The model outperforms other baselines in the empirical studies.\n\n\nWeakness:\n1. It is unclear what the \"training dynamics\" is referring to in the proposal. In the introduction, the dynamics are hinting at how the model changes during training, but the main methodological contribution seems to be the \"distribution-informed neural network.\" If the domain divergence is defined on the training dynamics of a model, it would be susceptive to common problems when training a model, e.g., underfitting, overfitting, poor-calibration. \n2. Regarding the empirical results on MPI3D, it is counter-intuitive that the weakly supervised version \"DINO-INIT\" outperforms \"DINO-TRAIN\". The authors should clarify the difference between the two models and explain why additional training is detrimental.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-3dGOIC2DUb", "reviewer": "Reviewer_24HZ", "summary": "This paper proposes a domain adaptation method through a learning algorithm that knows the distribution of source and target well. The proposed method performs regression task when the distribution of source and target are different. Their learning algorithm consider maximizing RKHS mean discrepancy. Various experiments show that the proposed method improve the performance of the previous methods.", "questions": "Q1) A lot of domain adaptation algorithms and models have been studied in the classification task. Why are these algorithms and models not applicable to regression task?\nWhy is the proposed method particularly suitable for regression problems?\n\nQ2) In fact, domain adaptation for classification can be easily applied to regression problems. Can you do a performance comparison with them in regression task?\n\nQ3) It is recommended to draw a workflow figure that shows everything from input to output at once. Can you draw a figure to understand the concept of the proposed method?", "limitations": "The authors explain the societal impact and limitation of the paper well.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n+) Motivation and the description of the proposed algorithm is convincing.\n+) Various experiments are performed, and the performance of the proposed method is better than previous regression DA methods.\n\nWeaknesses\n-) The paper representation needs to be improved. There are too many formulas in the main paper. Except for essential formulas, authors need to pass them all to the appendix and explain the concept and analysis in the main paper.\n-) There are not enough SOTA methods in the experimental results. In particular, AT-GP was proposed in 2010 and the performance improvement of the proposed method is not remarkable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept