The Role of Embedding Complexity in Domain-invariant Representations
Authors: Ching-Yao Chuang
Antonio Torralba
Stefanie Jegelka
Venue:  (2020)
Keywords: domain adaptation
domain-invariant representations
model complexity
theory
deep learning

Abstract: Unsupervised domain adaptation aims to generalize the hypothesis trained in a source domain to an unlabeled target domain. One popular approach to this problem is to learn domain-invariant embeddings for both domains. In this work, we study, theoretically and empirically, the effect of the embedding complexity on generalization to the target domain. In particular, this complexity affects an upper bound on the target risk; this is reflected in experiments, too. Next, we specify our theoretical framework to multilayer neural networks. As a result, we develop a strategy that mitigates sensitivity to the embedding complexity, and empirically achieves performance on par with or better than the best layer-dependent complexity tradeoff.

Under review as a conference paper at ICLR 2020
THE ROLE OF EMBEDDING COMPLEXITY IN DOMAIN-INVARIANT REPRESENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Unsupervised domain adaptation aims to generalize the hypothesis trained in a source domain to an unlabeled target domain. One popular approach to this prob- lem is to learn comminn-nvvarinn study, theoretically and empirically, the effect of the embedding complexity on generalization to the target domain. In particular, this complexity affects an upper bound on the target risk; this reflected in experiments, too. Next, we specify our theoretical framework to multilayer neural networks. As result, we develop a strategy that mitigates sensitivity to the embedding complexity, and empirically achieves performance on par with better than the best layer-dependent com- plexity tradeoff.
embeddings for both domains. In this work, we
INTRODUCTION
Domain adaptation is critical in many applications where collecting large-scale supervised data is prohibitively expensive or intractable, or where conditions at prediction time can change. For in- stance, self-driving cars must be robust to different weather, change of landscape and traffic. In such cases, the model learned from limited source data should ideally generalize to different target domains. Specifically, unsupervised domain adaptation aims to transfer knowledge learned from a labeled source domain to similar but completely unlabeled target domains.
One popular approach unsupervised domain adaptation is learn
representa-
tions (Ben-David et al. 2007 Long et al. 2015 Ganin et al.) 2016), by minimizing divergence between the representations of source and target domains. The prediction function learned on
these "aligned" representations with the aim of making Noomain-indepente
A series of theoret-
ical works justifies this idea (Ben-David et al. 2007: Mansour et al.) 2009 Ben-David et al. 2010
Cortes & Mohri 2011).
Despite the empirical success of domain domain-invariane invariant representations, exactly matching the representa- tions of source and target distribution sometimes fail to achieve domain adaptation. For example, Wu et al.) (019)) show that exact matching may increase target error if label distributions are dif- ferent between source and target domain, and propose a divergence metric overcome this limitation. Zhao et al. 2019) establish lower and upper bounds on the risk when label distributions between source and target domains differ. Johansson et al. (2019) point the information lost in non-invertible embeddings, and propose different generalization bounds based on the overlap of the supports of source and target distribution.
In contrast to previous analyses that focus on changes in the label distributions or joint support, we study the effect of embedding complexity. In particular, we show a general bound on the target risk that reflects a tradeoff between embedding complexity and the divergence of source and target domains. A too powerful class of embeddings can result in overfitting the source data and the match- ing of source and target distributions, resulting in arbitrarily high target risk. Hence, restriction is needed. We observe that indeed, without appropriately constraining the embedding complexity, the performance of state-of-the-ee methods such conain-adversarial neural networks (Ganin et al. 2016) can deteriorate significantly.
Next, tailor the bound to multilayer neural networks. In realistic scenario. one may have total depth budget and divide the network into an encoder (embedding) and predictor by aligning the representations of source and target in chosen layer, which defines the division. In this case,

---

Under review as a conference paper at ICLR 2020
more complex encoder necessarily implies weaker predictor, and vice versa. This tradeoff is reflected the bound and, we see that, in practice, there is an "optimal" division.
To better optimize the tradeoff between encoder and predictor without having to tune the division, we propose to optimize the tradeoffs in all layers jointly via a simple yet effective objective that can easily be combined with most current approaches for learning domain-invarian representations. Implicitly, this objective restricts the more powerful deeper encoders by encouraging a simultaneous alignment across layers. In practice, the resulting algorithm achieves performance on par with or better than standard domain-invariant representations, without tuning of the division.
Empirically, we examine our theory and learning algorithms on sentiment analysis (Amazon review dataset), digit classification (MNIST, MNIST-M, SVHN) and general object classification (Office- 31). In short, this work makes the following contributions:
General upper bounds on target error that capture the effect of embedding complexity when learning dovain-invariant representations;
Fine-grained analysis for multilayer neural networks, and new objective with implicit regularization that stabilizes and improves performance;
Empirical validation of the analyzed tradeoffs and proposed algorithm on several datasets.
UNSUPERVISED DOMAIN ADAPTATION
For simplicity of exposition, consider binary classification with input space &' C R" and output space y = {0, 1}. Define H. to be the hypothesis class from X' to V. The learning algorithm obtains two datasets: labeled source data Xs from distribution ps, and unlabeled target data Xr from distribution pr. We will use ps and pr to denote the joint distribution on data and labels X, Y and the marginals, i.e., ps (X) and PY()). Unsupervised domain adaptation seeks hypothesis h E H that minimizes the risk in the target domain measured by a loss function & (here, zero-one loss):
RT Enyssp (((h(x), v)]-
We will not assume common support in source and target domain, in line with standard benchmarks for domain adaptation such adapting from MNIST to MNIST-M.
2.1 DOMAIN-INVARIANT REPRESENTATIONS
A common approach to domain adaptation is to learn a joint embedding of source and target data (Ganin al. 2016 zeng et al.) 2017). The idea that aligning source and target distributions in a latent space Z results in a domain invariant representations, and hence a subsequent classifier f from the embedding to y will generalize from source to target. Formally, this results the following objective function on the hypothesis h = fg == f g. where G is the class of embedding functions g to Z, and we minimize a divergence d between the distributions (Z) = ps(9(X)), = pr (x))) of source and target after mapping to Z:
min SEA,9E9
Rs(fg) ad(p' PZ))).
The divergence d could be, e.g., the Jensen-Shannon (Ganin et al. 2016) or Wasserstein distance (Shen et al. (017).
2.2
UPPER BOUNDS ON THE TARGET RISK
Ben-Davic al. (2007) introduced the diveeiiveeeee to bound the worst-case loss from extrap- olating between domains. Let Rn (n, h') Ez=nle(h(a) hx()) be the expected disagreement between two hypotheses. The H-dd-diveeeee measures whether there is any pair of hypotheses whose disagreement (risk) differs a lot between source and target distribution.
Definition 1. HHLAH-divergence Given two domain distributions ps and pr over X, and a hypoth- esis class H. the HAAH--ivergenn between ps and pr is
duAu(ps (PS.Pr) b.
sup
[Rs (h, h²) Rr h')l
2

---

Under review as a conference paper at ICLR 2020
The HLAH-divergence is determined by the discrepancy between source and target distribution and the complexity ofthe hypothesis class H. For a hypothesis class H. + {0, the disagreement between two hypotheses is equivalent to the exclusive function. Hence, one can interpret the HAA-ddvvergene as finding a classifier in function space HAH = H e H which attempts to max- imally separate one domain from the other (Ben-David al. 20TO). A restrictive hypothesis space may result in small H-A---iveeeence even if the source and larget domain do not share common support. This divergence allows bound the risk on the target domain:
Theorem 2. (Ben-David et al. 2010) For all hypotheses E H, the target risk is bounded as
(h) - Rh) + ddHHH(SSPP ) + Ax:
where dy is the best joint risk
Ny = inf (Rs(h) (h)) N'EH
Similar results exist for continuous labels (Cortes & Mohri 2011 Mansour et al. 2009)
Theorem 2 is influential theoretical result in unsupervised domain adaptation, and motivated work on domain invariant representations. For example, recent work (Ganin et al. (2016); Johansson et al. (2019) applied Theorem to the hypothesis space F that maps the representation space Z induced by an encoder g to the output space:
RT(fg) S Rs(f9) (FF + A=(g)
(3)
where (g) is the best hypothesis risk with fixed i.e., Az(g)= = inf gEE (s(()) + Rr( g)}-
The FAF divergence implicitly depends on the fixed g and can be small if g provides suitable representation. However, if induces a wrong aligmment, then the best hypothesis risk X> is large with any function class F. The following example will illustrate such a situation, motivating to explicitly take the class of embeddings into account when bounding the target risk.
3 INFLUENCE OF THE EMBEDDING COMPLEXITY
We begin with illustrative toy example. Fig- ure [ shows a binary classification problem in 2D with disjoint support and slight shift in the label distributions from source to target: ps(y = 1) = pr Pr = 2r. Assume the representation space 2 one dimensional, so the embedding g is a function from 2D ID. If we allow arbitrary, nonlinear em- beddings, then, for instance, the embedding in Fig- [b), together with an optimal predictor, achieves zero source loss and zero divergence which is op- timal according to the objective in equation (2). But the target risk of this combination of embedding and predictor is maximal: (f(g = 1.
0.5
Source
Target
y
5/
0.5 (n) Linear Embedding
0.5++
0.5 (b) Nonlinear Fimbedding
0.5
Figure 1: Illustrative example in 2D. The ID representation space is illustrated as a dotted line, and arrows indicate the embedding from 2D to ID. (a) Optimal embedding when g is the class of linear functions. (b) Optimal em- bedding with a complex nonlinear function class: zero source error and divergence loss, but the embedding destroys label consistency and leads to maximal target risk.
If we restrict the class G of embeddings to linear maps g(x) Wæ where W E IR. xx2 then the embeddings that are optimal with respect the ob- jective (2) are of the form W = Oj. Together with an optimal source classifier f. they achieve a non-zero value of 2e for objective (2) due to the shift in class distributions. However, these embeddings retain label correspondences and can thus minimize target risk.
This example illustrates that too rich class of embeddings "overfit" the alignment, and hence lead to arbitrarily bad solutions. Hence, the complexity of the encoder class plays an important role in learning domain invariant representations.
3

---

Under review as a conference paper at ICLR 2020
3.1 BOUNDS FOR DOMAIN-INVARIANT REPRESENTATIONS
Motivated by the above example, we next expose how the bound on the target risk depends on the complexity of the embedding class. To do so, we apply Theorem 2to the hypothesis / Ig:
(fg) - Rs(fg) + dFsaFF((s. (P) + XFg.
(4)
This bound differs in two ways from the previous bound (equation 33), which was based only on F: the best in-class joint risk now minimizes over both F and g, i.e.,
XFg
inf [Rs(f9) + RT (fg)).. SEF.gE9
(5)
which is smaller than (g) and reflects the fact that we are learning both f and In return, the divergence term dFGAFFG (Ps, PT) becomes larger than the one in equation (3). To better understand these tradeoffs, version of the -ivveriieee that explicitly measures variation of the embeddings in G:
will reformulate bound (4) to be more interpretable. To this end, we define
Definition 3. -ivereeivege For two domain distributions ps and pr over X, an encoder class G, and predictor class F, the (eagg-ivergeece between ps and Pr is
dreso (ps,PP))
sup SEF; 9:9 eg
RRs (Rs, R fg))|.
Importantly, the diverdiieeeeecc is smaller than the FGA.FG since the two hypothe- ses in the supremum, fg and I'g', share the same predictor f. Theorem 4. For all f E F and g E G. Rr(Jg) S Rs(Sg) + dFAr(1g(2), +
drese (ps, PT) +)rg(g)-
(6)
Latent Divergence
Embedding Complexity
where (g) is the best in- in-class joint risk defined as inf J'EF.''E
Xrg(g)=
We prove all theoretical results in the Appendix. This target generalization bound is small if (C1) the source risk small, (C2) the latent divergence is small (hecause the domains are well-aligned and/or F is restricted), (C3) the complexity of is restricted avoid overfitting of alignments, and (C4) good source and target risk is in general achievable with F. and G.
Comparison Previous Bounds. The last two terms in Theorem express a similar complexity tradeoff, but with respect to the overall hypothesis class H. which here combines encoder and pre- dictor. Directly applying Theorem to the composition H = FG (equation explicit as Theorem
treats both jointly 4 The recent bound (3) as-
and does not make the role of the embedding sumes a fixed embedding g and focuses on the predictor class F. As a result, it captures embedding complexity even less explicitly: the first two terms in bound (3) and Theorem 4 are the same. The last term in (3), Ag)) contains the target risk with the given g. Hence, bound (3) replaces (C3) and (C4) above by saying F and the specific g (which is much harder control since in practice is also optimized) can achieve good source and target risk. In contrast, Theorem 4 states an explicit complexity penalty on the variability of the embeddings, and uses the fixed g only in the source risk, which can be better estimated empirically.
If F is not too rich, the latent divergence can be empirically minimized by finding well-aligned embedding. Hence, we can minimize the upper bound in Theorem 4 minimizing the usual source loss and Nomain--nvarian loss (2) and by choosing and appropnately tradeoff the complexity penalty dreag, the latent divergence (which increases with complexity of F and decreases with complexity of 9), and the best in-class joint risk (which decreases with complexity of and G).
3.2 EMBEDDING COMPLEXITY TRADEOFFS EMPIRICALLY
To empirically verify the embedding complexity tradeoff, keep the predictor class F fixed, vary the embedding class G, and minimize the source loss and aligmment objective (2). Concretely, we train domain adversarial neural networks (DANNs) (Ganin et al.) 2016) the Amazon reviews
4

---

Under review as a conference paper at ICLR 2020
(8)
0.29
DANN
Figure 2:
Empirical ver-
6.25
Source
ification on Amazon reviews dataset. (a) Vary the number of layers in the encoder while fixing the predictor. (b) Fix the total number of layers and optimize the domain-invariant loss in different layers.
Target krme
Targo Error
0.35
0.25
6224
0.3
622
0.2
Number of Layers Enoodar
Numbar Layes Encoder
dataset (Book- + Kitchen). Our hypothesis class is a multi-layer ReLU network, and the divergence minimized against a discriminator. For more experimental details and results, please refer to section 6 We train different models by varying the number of layers in the encoder while fixing the predictor to 4 layers. Figure 3.2 a) shows that, when increasing the number of layers in the encoder, the target error decreases initially and then increases as more layers are added. This supports our theory: the smaller encoders are not rich enough to allow for good alignments and Arg(g), but overly expressive encoders may overfit.
Predictor Complexity. Theoretically, the complexity of the predictor class also affects the gener- alization bound in Theorem 4 Empirically, we found that the predictor complexity has much weaker influence on the target risk (see experiments in Appendix B). Indeed, theoretically, while the com- plexity of F affects the latent divergence, if the aligmment via is very good, this divergence can still be small. In addition, the FgAg- diverreeee is more sensitive to the embedding complexity than the predictor complexity. This offers a possible explanation for observations. In the remainder of this paper, we focus on the role of the embedding.
Discussion.
The results in this section indicate that, without constraining the embedding complexity, we may overfit the distribution alignment and thereby destroy label consistency as in Figure The bound suggests choose the minimal complexity encoder class G that is is still expressive enough to minimize the latent space divergence. Practically, this can be done by regularizing the encoder, e.g., restricting Lipschitz constants or norms of weight matrices. More explicitly, one may limit the number of layers of neural network, or apply inductive biases via network architectures. For instance, compared fully connected networks, convolutional neural networks (CNNs) restrict the output representations to be spatially consistent with respect to the input.
4 BOUNDS FOR MULTILAYER NEURAL NETWORKS
Due to their wide empirical success, multilayer neural networks have been adopted for learning innaiatinvariant representations. Next, we adapt the bound in Theorem D to multilayer networks. Specifically, we consider the number of layers as an explicit measurement of complexity. This will lead to simple yet effective algorithm to mitigate the negative effect of very rich encoders.
4.1 EFFECT OF LAYER DIVISIONS
Assume we have an N N-layer feedforward neural network h E H. The model can be decomposed as h =fig: E F&G. H for (1,2,...,NN 1} where the embedding gi is formed by the first layer to the i-th layer and the predictor fs is formed by the + 1-th layer to the last layer. We can then rewrite the bound in Theorem layer-specific form:
Ry (h) < (h) + dr.AF. (2), pZ) (Z))
dFig,ao, (Ps.Pr) +AF.g. (gi).
Latent Divergence in i-th layer
Embedding Complexity went
This yields N - layer-specific upper bounds. Importantly, minimizing the domain- -nvaaiiat loss in different layers leads to different tradeoffs between fit and complexity penalties. This reflected by the following inequalities that relate different layer divisions.
Proposition (Monotonicity) In an N-layer feedforward neural network h = fig. E F.G. for E 2,, .. 1}. the following inequalities hold for all i J: (embedding complexity)
H
drig,as, (ps)Pr) < d dr (ps. Pr)
(8)
dy,A7, (7), (2)) 2 dr, (1' (7), (7)1)) (7))
(latent divergence)
(9)
5

---

Under review as a conference paper at ICLR 2020
Proposition states that the latent divergence is monotonically decreasing and the complexity penalty is monotonically increasing with respect to the embedding's ' depth. This is a tradeoff within the fixed combined hypothesis class H. A deeper embedding allows for better alignments and si- multaneously reduces the depth (power) of F; both reduce the latent divergence. At the same time, it incurs larger
This suggests that there might be an optimal division that minimizes the bound on the target risk. In practice, this translates into the question: in which intermediate layer should we optimize the domain-invariant loss? Figure (.2(b) shows how the target error changes as function of the layer division, with total of target error, suggesting that for fixed H. i.e., total network depth, not all divisions are equal.
layers. Indeed, empirically there an optimal division with minimum
If the exact layer-specific bounds could be computed, one could simply select the layer division with the lowest bound. But, this is in general computationally nontrivial. Instead, we take different perspective. In fact, the layer-specific bounds 7 all hold simultaneously, independent of the layer selected for distribution alignment.
Corollary 6. Let h be an N-layer feedforward neural network h Seg. F,g 1), we have the layer-agnostic bound
H .or i
{1, N - 1}.
RT(h)= S (h) +
min (ISiEN)
(1Z,AF (2), (Z)) + dFis,so, (ps, + \F (n)))
where {Fo(9) is the best in-class joint risk defined in Theorem 4
The corollary implies that at least one of these bounds should be small. Recall that the bounds depend on how well we can minimize the source risk and align the distributions via sufficiently powerful embedding, while, at the same time, limiting the complexity F and G.
4.2 MULTILAYER DIVERGENCE MINIMIZATION AND REGULARIZATION
Corollary poritss to various algorithmic ideas: (1) Simultaneously optimizing several bounds may result approximately minimizing at least one of them, without having to select an optimal one. (2) We may attain small latent divergence with deeper encoder, if we achieve to restrict the complexity of G appropriately. It turns out that these two ideas are related.
Optimizing the domain invariant loss with alignment in specific layer may result in large bounds for the other layers, due to the monotonicity of the two divergences (Proposition [5) and potentially non-aligned embeddings in lower layers. Hence, optimization problem where we jointly align source and target distributions multiple layers. Let
propose to instead solve multi-objective
L C (1, 2, ................................ N 1} be a subset of layers. We minimize the weighted sum of divergences, and refer to this objective as Multilayer Divergence Minimization (MDM):
minneH Rs(h)= + Eisc ogdlp's (Z), 2% (.rf(()) (Z)).
(10)
This objective encourages alignment throughout the layer-wise embeddings in the network. First, a good alignment minimizes the latent divergence, if JF is not too rich. For the lower layers (shallow embeddings), this comes together with a very restricted class of embeddings, and hence limits both latent divergence and complexity penalty. Without the optimization across layers, the embeddings in lower layers are not driven towards alignment.
Second, enforcing alignment in lower layers implicitly restricts the deeper embeddings in higher layers, since the embeddings are such that alignment happens early on. This effect may be viewed as an implicit regularization. By this perspective, the bounds for higher layers profit from low latent divergences (deeper embeddings and shallow predictors) and restricted complexity of G.
In general, one can simply set 21,2...., N - 1}. To improve computational efficiency, we can sub-sample layers or exclude the first and the last few layers. MDM is simple and general, and can be combined with most algorithms for learning ionain-invariant representations. For DANN, for instance, we minimize the divergence in multiple layers by adding discriminators.
OTHER RELATED WORKS
Existing approaches for learning domain-invariant representations may be distinguised, e.g., by which divergence they measure between source and target domain. Examples include domain adver-

---

Under review as a conference paper at ICLR 2020
Thager Linor
Tagget Linor
Nimber Layers Unooder
Nimber Layers Uiscoder
Sumber of Layers
Number Linoddr
Lence
Torpet Lenor
Nimber of Layers in Enooder
Nimber of Layers Encoder
Suither of Layers Eincoder
Number of Lavens in Encoder
(a) B--K
(b) K-BB
(c) B-D
(d) D-B
Figure 3: Amazon reviews dataset. First row: Fixed predictor class, varying number of layers in the encoder. Second row: Fixed total number of layers and optimizing Nomain-invariin loss in a single intermediate layer or MDM.
sarial learning approaches (Ganin & Lempitsky 2014 I'enn et al. 2015 Ganin et al. 2016), max- imum mean discrepancy (MMD) (Lnng et al. 2014|[2015 2016) and Wasserstein distance (Courty
et al. 2016 2017 Shen et al. 2017 Lee & Raginsky 2018).
Other works improve performance by combining the domain-invariant loss with other objectives. Shu et al. (2188) penalize the violation of the cluster assumption. In addition to the shared feature encoder between source and target domain, |Bousmalis al. (2016) include private encoders for each domain to capture domain-specife information. [Long et al. (2018) propose domain discriminator that is conditioned on the cross-covariance of domain-specific embeddings and classifier predictions to leverage discriminative information. Besides the usual distribution alignment, |Hoffman et al.) (217)) further align the input space with a generative model that maps the target input distribution to the source distribution. These previous works can be interpreted as adding additional regularization via auxiliary objectives, and thereby potentially reducing the complexity penalty.
Some previous works also optimize the domain-invarte fuse the representations from a bottleneck layer and classifier layer by a tensor product and min- imize the domain divergence based the aggregated representations. Joint adaptation networks (JADs) {Long et al., (017) minimize the MMD in the last few layers to make the embeddings more transferable. MDM can be seen nearly every layer, driven by a strong theoretical motivation. Importantly, minimizing the divergence only in the last few layers could still be suboptimal, since the embeddings may not be sufficiently regularized.
loss in multiple layers. Long et al. (2016)
generalization of JADs that minimizes domain divergence in
EXPERIMENTS
We test our theory and algorithm on several standard benchmarks: sentiment analysis (Amazon reviews dataset), digit classification (MNIST, MNIST-M, SVHN) and general object classification train DANN (Ganin et al. 2016), which measures the latent
(Office-3l). In all experiments, divergence via domain discriminator (Jensen Shannon divergence). A validation set from the source domain is used as an early stopping criterion during learning. In all experiments, we use the Adam optimizer (Kingma & Bal (014) and progressive training strategy for the discriminator (Ganin et al. (016). We primarily consider three types of complexity: number of layers, number of hidden neurons, and inductive bias (CNNs). In all experiments, retrain each model for 5 times and plot the mean and standard deviation the target error.
For evaluating MDM, we consider three weighting schemes: uniform weights (a; aa), linearly decreasing (a; = ag cc decreasing weights encourage the network minimize the latent divergence in the first few layers, where the embedding complexity is low. This may also further restrict the deeper embeddings. More experimental details be found in Appendix
cxb i), and exponentially decreasing (a; = Og exp(-cxx i)) where c 2 0. The

---

Under review as a conference paper at ICLR 2020
MNSST + M-MNIST
311
030
DANN
Target Estor
Thandd
13rger
4:22
Taggst
6:00
600
3m
30
Number Layers Enceder
Encoder Hidden Wiah
Number Layers Encoder
SVIN + ANNIS
1275
115
10000
Number Layers Enceder
Encoder Tiiddem Width Rario
Number Layers Encoder
(a) Vury Layer Number
(b) Vary Hidden Width
(c) Fixed Hypothesis
Figure 4: Digit classification. (a) Fixed predictor class, varying number of layers in the encoder.
(b) Fixed predictor class, varying the hidden width of the encoder. (c) Fixed total number of layers and optimizing domain -amaii-nnvariant loss in single intermediate layer or MDM.
Sentiment Classification. We first examine complexity tradeoffs on the Amazon reviews data, which has four domains (books (B), DVD disks (D), electronics (E), and kitchen appliances (K) with binary labels (positive/ / negative review). Reviews are encoded into 5000 dimensional feature vectors of unigrams and bigrams. The hypothesis class multi-layer ReLU networks. We show the results on B-K, K- B-D, and D-B in Figure 3 To probe the effect of embedding complexity by itself, we fix the predictor class layers and vary the number of layers of the embedding. In agreement with the results in Section 3.2 the target eπror decreases initially, and then increases as more layers are added to the encoder.
Next, we probe the tradeoff when the total number of layers is fixed to 8. The bottom row of Figure 3 shows that there exists an optimal setting for all tasks. For MDM, we optimize alignment in all inter- mediate layers. The results suggest that MDM's performance is comparable to the hypothesis with the optimal division, without tuning the division. The three weighting schemes perform similarly, suggesting that MDM is robust weight selection.
Digit Classification. We next verify our findings on standard domain adaptation benchmarks: MNIST-MNIST-M (M-M-M) and SMHN (S+M)). We use standard CNNs as the hy-
pothesis class; architecture details are in Appendix C
To analyze the effect of the embedding complexity, we aug- ment the original two-layer CNN encoders with to 6 addi- tional CNN layers for M-M-M and to 24 for S-MM, leav- ing other settings unchanged. Figure a) shows the results. Again, the target error decreases initially and increase as the encoder becomes more complex. Notably, the target error creases by 19.8% in M-MMM and 8.8% in S-M compared to the optimal case, when more layers are added the encoder. We also consider the width of hidden layers as a complexity measure, while fixing the depth of both encoder and predictor. The results are shown in Figure b)). This time, the decrease in target error not significant compared to increasing encoder depth. This suggests that depth plays representations.
Target tiror
DANN
Source
Number of Layers in Encoder
Figure 5: DANN with FC layers.
more important role than width in learning
Next, we fix the total number of CNN layers the neural network to and 26 for M-+M-M and S-M, respectively, and optimize the domain-invariant loss in different intermediate layers. The results in Figure 4[c) again show a "U-curve", indicating the existence of an optimal division. Even with fixed total size of the network (7-). the performance gap between different divisions still reach 19.5% M M-M and 10.4% S M. For MDM, L contains all the augmented CNN layers for M M-M. For S S-M,, we sub-sample CNN layer every four layers form L. We also observe
8

---

Under review as a conference paper at ICLR 2020
Target
Trrstt
Number of Layers in Eacoder
Namber of Layen Finedee
Number of Layers lnnceder
Number of Layens Fineder
Number of Layers in Etnoddr
Number of Layens Encoder
Number Layers Unceder
Number of Layers Encoder
(a) A-W
(b) A-D
(c) W-+A
(d) D->A
Figure 6:
Office-31 Dataset. First row: Fixed predictor class, varying encoder depth. Second row:
Fixed total number of layers, optimizing domain- invariant loss in a single layer MDM.
that MDM with all weighting schemes consistently achieves comparable performance with the best division in S-M and even better performance in M-MMMM.
To investigate the importance of inductive bias in domain-invarian representations, replace the CNN encoder by an MLP encoder. The results for M-M-M are shown in Figure 5 Comparing to CNNs, which encode invariance via pooling and learned filters, MLPs do not have any inductive bias and lead to worse performance. In fact, the target error with MI.P-based domain adaptation is higher than merely training the source: without appropriate inductive bias, learning domain invariant representations can even worsen the performance.
Object Classification. Office-31 (Saenko et al. (010). one of the most widely used benchmarks in
domain adaptation, contains three domains: Amazon (A), Webcam (W), and DSLR (D) with 4,652
images and 31 categories. We show results for A A->,, A ADD, W WAA, and D->A in Figure 6 To overcome the lack of training data, similar (Li et al. 2018 Long al. 2018), we use ResNet- ImageNet (Deng al. 2009) for feature extraction. With the
50 (He et al., 2016) pretrained extracted features, we adopt multi-layer ReLU networks as hypothesis class. Again, we increase depth of the encoder while fixing the depth of the predictor to 2 and show the results Figure Even with a powerful feature extractor, the embedding complexity tradeoff still exists. Second, we fix the total network depth 14 and optimize MDM, with L containing all even layers in the network. MDM achieves comparable performance to the best division for most of the tasks, albeit slightly worse performance in D-
7 CONCLUSION
In this paper, theoretically and empirically analyze the effect of embedding complexity the target risk domain-invariant representations. We find a complexity tradeoff that has mostly been overlooked by previous work. In fact, without carefully selecting and restricting the encoder class, learning domain invariant representations might even harm the performance. We further develop simple yet effective algorithm approximately optimize the tradeoff, achieving performance across tasks that matches the best network division, i.c., complexity tradeoff. Interesting future directions of work include other strategies for model selection, and more refined analysis and exploitation of the effect of inductive bias.
REFERENCES
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In Advances in neural information processing systems, pp. 137-144, 2007.
Shai Ben-David. John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory learning from different domains. Machine learning. 79(1-2):151-175,
2010.

---

Under review as a conference paper at ICLR 2020
Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. In Advances in neural information processing systems, pp. 343-351, 2016.
Corinna Cortes and Mehryar Mohri. Domain adaptation in regression. In International Conference on Algorithmic Learning Theory, pp. 308-323. Springer, 2011.
Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamonjy. Optimal transport for do- main adaptation. IEEE transactions on pattern analysis and machine intelligence, 39(9):1853 1865, 2016.
Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamonjy. Joint distribution optimal transportation for domain adaptation. In Advances in Neural Information Processing Systems, pp. 3730-3739, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. arXiv preprint 1409 7495, 2014.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. The Journal of Machine Learning Research, ((1):2006-2033) 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pailern recognition, pp- 770-778, 2016.
Judy Hoffman, Eric Tzeng, Taesung Park, Yan Zhu, Phillip Isola, Kate Saenko, Alexei A Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. arXiv preprint arXiv:1711.03213 2017.
Fredrik D Johansson, Rajesh Ranganath, and David Sontag. Support and invertibility in domain- invariant representations. arXiv preprint
2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14/2.644/2.69880 2014.
Jaeho Lee and Maxim Raginsky. Minimax statistical learning with wasserstein distances. In Ad- vances Neural Information Processing Systems, pp. 2687-2696. 2018.
Shuang Li, Shiji Song, Gao Huang, Zhengming Ding, and Cheng Wu. Domain invariant and class discriminative feature learning for visual domain adaptation. IEEE Transactions on Image Pro- cexsing, 27(9):4260
2018
Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip Yu. Transfer joint matching for unsupervised domain adaptation. In Proceedings of the IEEE conference com- puter vision and pattern recognition, pp. 1410-1417, 2014.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. arXiv preprint rrXiv:1502.02791 2015.
Mingsheng Long. Han Zhu, Jianmin Wang, and Michael Jordan. Unsupervised domain adaptation with residual transfer networks. In Advances in Neural Information Processing Systems, pp. 136- 144, 2016.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2208-2217. JMLR. org, 2017.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, pp. 1640-1650, 2018.
10

---

Under review as a conference paper at ICLR 2020
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. arXiv preprint arXiv:0902.34300, 2009.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European conference on computer vision, pp. 213-226. Springer, 2010.
Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. arXiv preprini arXiv: 1707.01217 2017
Rui Shu, Hung H Bui, Hirokazu Narui, and Stefano Ermon. A dirt-t approach to unsupervised 2018.
domain adaptation. arXiv preprint arXiv:1802.0888
Eric Tzeng, Judy Hoffman, Trevor Darrell, and Kate Saenko. Simultancous deep transfer across domains and tasks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4068-4076, 2015.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7167-7176, 2017.
Yifan Wu, Ezra Winston, Divyansh Kaushik, and Zachary Lipton. Domain adaptation with asymmetrically-relxxe distribution alignment. arXiv preprint arXiv: 01669,,,,,,, 2019.
Han Zhao, Remi Tachet des Combes, Kun Zhang, and Geoffrey J Gordon. On learning invariant representation for domain adaptation. arXiv preprint MXXi::900.009550 2019.
A PROOFS
A.1 PROOF OF THEOREM 4
Theorem 4. For all f E F and g E G. RT(f9) S Rg(f9)
(Z), + droso (ps.PT) + )Fg(O).
where {g is the best in-class joint risk defined as 2Rs + R inf ge
}Fo(o)=
Proof. We first define the optimal composition hypothesis frg with respect to an encoder g to be the hypothesis which minimizes the following error
f"g" arg min (Rs( Rs( Rr(''
(11)
f'EF.g'EG
By the triangle inequality for classification error (Ben-Davide al. (007)),
(T( Rr([""")) + Rr(Jg,
(12)
< Rf g²) + (fg, f* + Rf
(13)
The second term in the R.H.S of Eq. 13 can be bounded as
fr(fg.f') < Rs(fg. f* + Rs(fg, Rr
(14)
s R + sup [Rs(fg. - Rr
(15)
f.rEF
= Rs(fg, d=)r+)
(16)
I Rs(fg) + Rs 9) + dy (2)
(17)
The third term in the R.H.S of Eq. 13 can be bounded as
(f < Rs(f"g,f'g") + Rs(l"g. [Rs(f g, f'g') Rr(f'g.
(f
(18)
s RR((f',,''')
sup
(19)
JEF-gg'EY
Rs(ff g²) +
(X))
(20)
s Rs(ffg) + Rs g²) droso (ps Px
(21)
11

---

Under review as a conference paper at ICLR 2020
Combine the above bounds, we have
Rf(fg) s Rs(fg) dF dress (X)( AFo(O)
(22)
where
+ Aro(g) = + Rs(l'g)) + Rr(
(23)
inf g'EFg'EG
2R
(24)
A.2 PROOF OF PROPOSITION 5
Proposition 5. In an N-layer feedforward neural network h fig. E F,9. H for
{1,2..... - 1}. the following inequalities hold for all i S
drig,as, (ps. Pr) - drso, 0g; (ps.Pr) Pr)
(r (Z), (2) dr, 22 (Z))
Proof. Given class of multilayer feedforward neural network, We define class of function Qiy to represent the function class formed by the intermediate hidden layer to layer j.
We now prove the first inequality. By the definition of Fgag-divergenee, for every S
diie, (ps.PT)
(25)
= sup [R((g. Rr(fg, sef;
(26)
9.9 E9:
sup g 9.9 Ec:
RRs(fgg fag') Rr fag'))
(27)
sup [Rs(fag, fd's' - Rr(fag, fd'gg)l
(28)
feF,
4.4 EQu
9.9 ES:
sup SEF, 9.9 EG;
Rs(fg, fg') Rf(f
(29)
=d do, (Ps.Pr)
(30)
We next prove the second inequality. By the definition of FAF-divergence, for every S
( (2),
(31)
sup
Rs(fg, (' - RT RT(fg,
(32)
f.''F
sup fS''
Rs(fqrgh ff
(33)
S sup geLy f.f'
Rs(fqge, f' Rr(fag
(34)
< sup q.eeQu
[Rs(fgg, f'd'g.) RT (Tqg. I'd ''99..) 9:)
(35)
f.f'F,
sup [Rs([ gis s' Rr(J9h. S.f'ET:
(36)
=dFAFF
(37)
12

---

Under review as a conference paper at ICLR 2020
B PREDICTOR COMPLEXITY
We investigate the effect predictor complexity on MNIST- Follow the procedure in section(6, we augment the original predictor with to 7 additional CNN layers and fix the number of layers in encoder to or vary the hidden width. The results are shown in Figure 7 The target error slightly decreases the number of layers in the predictor increases. Even we augment layers to the predictor, the target error only decrease 0.9% which is nearly ignorable. Therefore, we focus on the embedding complexity the main paper which is both theoretically and empirically interesting.
0.075
0.075
Targer Error
Target Error
0.070
0.065
0.065
0.055
0.055
0.050
1.00
2.00
3.00
4,00
5.00
Number of Layers in Predictor
Predictor Hidden Width Ratio
(a) Vary Layer Number
(b) Vary Hidden Size
Figure 7: Predictor complexity trade-off MNIST-MNIST-M Fix the encoder class and vary the number of layers in the predictor. (b) Fix the encoder class and vary the hidden width of the predictor.
C EXPERIMENT DETAILS AND NETWORK ARCHITECTURES
C.I AMAZON REVIEW DATASET
The learning rate of Adam optimizer We adopt the original progressive training strategy for discriminator (Ganin et al. (016)) where the weight for using the following schedule:
1 - and the model are trained for 50 epochs.
loss in equation is initiated O and is gradually changed
2
(38)
+ cxp(-100))))0p p)
where p is the training progress linearly changing from 1. The architecture of the hypothesis and discriminator are as follows:
Encoder
Predictor
nn. Len.Linnar(5000 128)
n1n Linear 128, 128)
nn Rn..el.U
nn. ReL.U
nn. Lineri 128, 128)
(depends the number of layers)
nn. ReLU
nn.Linear( 128,
xn (depends on the number of layers)
nn.tofxmax
Discriminator
nn. Linear 128, 256) nn.ReLU
nn. Lin. 256) Rn.ReLU
x5
nn. nn...nnar(256, 2) nn.Solx
13

---

Under review as a conference paper at ICLR 2020
C.2 DIGIT CLASSIFICATION
The learning rate of Adam optimizer is set to x e and the model are trained for 100 epochs. The weight a for domain-invariant loss in equation (2) is initiated and is gradually changed to 0.1
using the same schedule in section C.I The architecture of the hypothesis and discriminator are as
follows:
Encoder
nn. Cnn.Con2d(3, 64, kerne
nn. Batchhoorm2dd
nn.MaxPoo/2d(2)
nn. ReLU
(6,, 128,
nn. B..Bachhorr2dd
nn. Dropout2d (only added for MNIST -MNIST-M
Maxooo)))))
Rn.
nn.Conv2d( 128, 128, kernel _iiz==3, padding=1)
nn. nn.BatchNorm2d
RnLReLU
Xn (depends on the number of layers)
Predictor
nn. on.Conn2d(125 (128, 128, kernel size=3, padding=1)
an.BaachNNrm2d
RnLR
xn (depends on the number of layers)
flatten
nn. (2ne,, 256)
nn Nn.BaachNormid
Rn.ReLU
nn (25e 10)
nn.Softmax
Discriminator
nn. nn.Conv2d(128, 256, vernel_size=33) padding=1)
nn.ReLU
nn. on.Con22d(256, 256, kernel padding=1)
nn.ReLU
x4
Flatten
nn. Ln...nnerr (4096, 512)
nn.ReLU
Li.eane 12, 512)
nn.ReLU
nn.Linear(512, 2)
nn.Softmax
In the hidden width experiments, we treat the architectures above the pivot and multiply their hidden width with the ratios.
C.3
OFFICE-31
We exploit the feature after average pooling layer of the ResNet-50 (He et al,, 2016) pretrained on ImageNet (Deng et al., 2009) for feature extraction. The learning rate Adam optimizer set to 3 x e and the model are trained for 100 epochs. The weight for donain-invariant loss in
14

---

Under review as a conference paper at ICLR 2020
is initiated at 0 and is gradually changed to 1 using the same schedule in section C.1 The architecture of the hypothesis and discriminator are as follows:
equation
Encoder
Predictor
nn. Lin.Linnar(200) 256)
nnn Lin.Linca (256, 256)
nn ReIU
nn. ReLU
I10. Lin.Linarr(266 256)
(depends the number of layers)
nn. Rel.U
nn. nn. nn.L.iner((256
xn (depends the number of layers)
nn.Softmx
Discriminator
nn. Linearinnar55 256)
nn. ReI.U
x6
nn.L.nnarr256, 2)
nn.Softmax
C.4 MULTILAYER DIVERGENCE MINIMIZATION
In all the experiments, we minimize the divergence in multiple layers by augmenting additional discriminators for each layer-specific representations where the discriminators share the same archi- tecture the standard setting.
For uniform weighting scheme (a, = ag), a; is set to the normalized same value a in the stand setting. For linear decreasing scheme (a; ou 1), O4 decreases from ag a to 0 linearly. For exponentially decreasing scheme (a; = ag exp( -c x i)), ag set to a and increases from linearly.
15


Reviews:
[{"review_id": "SkxxRsGe9H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the problem of domain adaptation via learning invariant representations. The main argument here is that when the total depth of layers in a neural network is fixed, tradeoffs exist between feature alignment and prediction power. Furthermore, the authors argue that richer feature extractor can sometimes significantly overfit the source domain, leading to a large risk on the target domain. \n\nOverall the paper is well-written and easy to follow. My major concern is that the paper, including the motivation and illustrative example, are too similar to previous work [1-2]. More detailed discussions are needed to highlight the difference of this work compared with [1-2]. The main contribution lies in Theorem 4. However, the upper bound is both loose and misleading. Compared with the original generalization upper bound [3], the one proposed in this paper contains a constant $\\lambda$ that contains FOUR optimal error terms. Note that the original one in [3] only contains two such terms. In fact even a bound containing 2 such terms could potentially be very loose, since it's perfectly fine that a hypothesis can have large risk on the source domain while still attaining a small risk on the target domain. The bound is misleading in the sense that this $\\lambda$ term cannot be computed or approximated, hence only the first two terms in (6) could be minimized in practice. However, this again can potentially lead to large target risk when the label distributions of source and target domains differ. \n\nThe experiments on using different number of layers of the network as feature extractors are quite interesting. The main message here is that general tradeoff exists with richer encoding function class. However, similar phenomenons have already been observed [4, Section 6.4], and it's not clear to me what's new here. \n\n[1].    On Learning Invariant Representations for Domain Adaptation, ICML 2019.\n[2].    Domain Adaptation with Asymmetrically-Relaxed Distribution Alignment, ICML 2019.\n[3].    Analysis of representations for domain adaptation, NIPS 2007.\n[4].    A DIRT-T APPROACH TO UNSUPERVISED DOMAIN ADAPTATION, ICLR 2018.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Byg2pWhqqr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the impact of embedding complexity on domain-invariant representations. By incorporating embedding complexity into the previous upper bound explicitly, the authors demonstrate the limitations of previous theories and algorithms. Based on their theoretical findings, the authors propose to control the embedding complexity with implicit regularization. Specifically, aligning source and target feature distributions in multiple layers controls both embedding complexity and domain discrepancy. The proposed algorithm can achieve similar performance as DANN with manual selection of embedding depth.\n\nBy noting that the hypothesis space can be decomposed in to the feature extractor and the classifier, the authors propose to address the domain discrepancy separately. D_H\\DeltaH is termed latent divergence, which the algorithm attempts to minimize. D_G\\DeltaG is treated as embedding complexity, which is the intrinsic property of the feature extractor. Thus, domain-invariant representations should seek a proper tradeoff between those two terms. \n\nThe paper is well-written and the contributions are stated clearly. The exploration on the layer division is really insightful.\n\nHowever, I have several concerns: \n1.\tThe proposed upper bound is insightful, but it has several limitations. Compared to the version applied to the feature space in equation (3), the proposed upper bound is looser. The embedding complexity terms includes two encoders, which are deep neural networks in practice, thus it can be excessively large. As the authors point out, in equation (3), the embedding complexity is not addressed explicitly, but it is implicit in the adaptability \\lambda in a more reasonable way. Previous works [1], [2], [3] have already taken them into consideration. Proposition 5 is a direct application of proposition 1 in [1].\n2.\tOn the claim of implicit regularization. By applying domain adversarial training to multiple layers, the authors claim that the encoder in higher layers is implicitly restricted. However, they do not validate this regularization effect. Is the embedding complexity controlled? Theoretical analysis or experimental results would be helpful.\n3.\tThe proposed MDM method seems to be incremental. [4] has probed into the effect of multi-layer adaptation strategy. Besides, applying domain adversarial training to many layers leads to more computational cost and may slow down training significantly. \n\n\n[1]Fredrik D Johansson, Rajesh Ranganath, and David Sontag. Support and invertibility in domain- invariant representations. arXiv preprint arXiv:1903.03448, 2019.\n[2]Han Zhao, Remi Tachet des Combes, Kun Zhang, and Geoffrey J Gordon. On learning invariant representation for domain adaptation. arXiv preprint arXiv:1901.09453, 2019.\n[3] Hong Liu, Mingsheng Long, Jianmin Wang, and Michael Jordan. Transferable adversarial training: A general approach to adapting deep classifiers. In International Conference on Machine Learning, pp. 4013–4022, 2019.\n[4] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks. In Proceedings of the 32nd International Conference on International Conference on Machine Learning, volume 37, pp. 97–105, 2015.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BJlNYB0RFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new theory for domain adaptation considering the complexity of representation extractors. This paper gives a new bound for target error in domain adaptation, which contains the classic distribution distance related to the hypothesis space of high-level classifiers and a new distribution distance defined on the embedding space. This paper also proposes Multilayer Divergence Minimization algorithm based on the theory and evaluates it on real-world dataset. \nPositive points: \n(a) This paper proposes an interesting insight that the complexity of embeddings is also important in domain adaptation.\n(b) This paper defines a new distribution divergence and build an interesting theory based on it.\n(c) The proposed algorithm could automatically reach the best result of trying DANN on each layer.\nNegative points:\n(a) There is no proof that this new bound is better than classic domain adaptation theory (Ben-David et al., 2010). Although this bound involves new insight, the novelty is limited if it is looser than existing upper bound. Furthermore, there are no creative tools in the mathematical proof part, which is a direct extension of the classic theory. \n(b) There is no analysis about the generalization when estimating this upper bound from finite samples. It could be easily seen that the sample complexity of embedding complexity is at least of the same order than classic \\mathcal{H}\\Delta\\mathcal{H}-divergence (Ben-David et al., 2010).\n(c) The analysis on the monotonicity of the divergences across the layers is very limited. It will be better if there is a discussion about when the monotonicity is strict.\n(d) What is the role of embedding complexity in the algorithm? It seems that only high-level classifier divergence is minimized.\n(e) Why minimizing the sum of divergences computed on all layers can control the proposed upper bound? It seems that if the embedding complexity of each layer is a constant, minimize divergence of a single layer can further minimize the minimum. Furthermore, there are previous method that minimizes divergences on all layers [A]. Please give a discussion on this method.\n(f)The empirical evaluation is relatively weak. There is no experiment based on convolutional networks, which are widely used on the Digit and Office-31 datasets.\n\nAlthough the insight is interesting, the novelty of this paper is not enough for being accepted by ICLR. So I vote for rejecting this submission. \n\n[A] Zhang, Weichen, et al. \"Collaborative and adversarial network for unsupervised domain adaptation.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the impact of embedding complexity on domain-invariant representations by incorporating embedding complexity into the previous upper bound explicitly.

The idea of embedding complexity is interesting, the exploration has some useful insight, and the paper is well-written. However, Reviewers and AC generally agree that the current version can be significantly improved in several ways:
- The proposed upper bound has several limitations such as looser than existing ones.
- The embedding complexity is only addressed implicitly, which shares similar idea with previous works.
- The claim of implicit regularization has not been explored in-depth.
- The proposed MDM method seems to be incremental and related closely with the embedding complexity.
- There is no analysis about the generalization when estimating this upper bound from finite samples.

There are important details requiring further elaboration. So I recommend rejection.
Decision: Reject