Domain Adaptive Multibranch Networks
Authors: Róger Bermúdez-Chacón
Mathieu Salzmann
Pascal Fua
Venue:  (2020)
Keywords: Domain Adaptation
Computer Vision

Abstract: We tackle unsupervised domain adaptation by accounting for the fact that different domains may need to be processed differently to arrive to a common feature representation effective for recognition. To this end, we introduce a deep learning framework where each domain undergoes a different sequence of operations, allowing some, possibly more complex, domains to go through more computations than others.
This contrasts with  state-of-the-art domain adaptation techniques that force all domains to be processed with the same series of operations, even when using multi-stream architectures whose parameters are not shared.
As evidenced by our experiments, the greater flexibility of our method translates to higher accuracy. Furthermore, it allows us to handle any number of domains simultaneously.

Published as a conference paper at ICLR 2020
DOMAIN ADAPTIVE MULTIBRANCH NETWORKS
Róger Bermidez-Chacón, Mathieu Salzmann, & Pascal Fua Computer Vision Laboratory École Polytechnique Fédérale de Lausanne
Station 14, CH-1015 Lausanne, Switzerland
{roger. be rmedez,, z, mathieu. zmann, pascal fua (epf)@@efl.... ch
ABSTRACT
We tackle unsupervised domain adaptation by accounting for the fact that differ- ent domains may need to be processed differently to arrive to a common feature representation effective for recognition. To this end, we introduce a deep learning framework where each domain undergoes different sequence of operations, al- lowing some, possibly more complex, domains to go through more computations than others. This contrasts with state-of-the domain adaptation techniques that force all domains to be processed with the same series of operations, even when using multi- stream architectures whose parameters are not shared. As evidenced by our experiments, the greater flexibility of our method translates to higher accu- racy. Furthermore, allows us to handle any number of domains simultaneously.
INTRODUCTION
While deep learning has ushered in great advances in automated image understanding, still suffers from the same weaknesses as all other machine learning techniques: when trained with images obtained under specific conditions, deep networks typically perform poorly on images acquired under different ones. This is known as the domain shift problem: the changing conditions cause the statistical properties of the test, or target, data, to be different from those of the training, or source, data, and the network's performance degrades accordingly.
Domain adaptation aims to address this problem, especially when annotating images from the target domain difficult, expensive, or downright infeasible. The dominant trend is map images to features that are immune to the domain shift, so that the classifier works equally well on the source and target domains (Fernando et al., 2013; Ganin & Lempitsky. 2015; Sun Saenko, 2016). In the context of deep learning, the standard approach is to find those features using single architecture for both domains (Tzeng et al., 2014; Ganin & Lempitsky. 2015; Sun Saenko, 2016; Yan et al., 2017: Zhang et al., 2018). Intuitively, however, the domains have different properties, it is not easy to find one network that does this effectively for both. A better approach is to allow domains to undergo different transformations to arrive domain-invariant features. This has been the focus of recent work (Tzeng et al., 2017; Bermúdez-Chhar source and target data pass through two different networks with the same architecture but different weights, nonetheless related to each other.
et al.. 2018: Rozantsev et al., 2018: 2019), where
In this paper, we introduce novel, even more flexible paradigm for domain adaptation, that allows the different domains to undergo different computations, not only in terms of layer weights but also in terms of number of operations, while selectively sharing subsets of these computations. This enables the network to automatically adapt situations where, for example, one domain depicts simpler images, such as synthetic ones, which may not need as much processing power as those coming from more complex domains, such images taken in-the-wild. Our formulation reflects the intuition that source and target domain networks should be similar because they solve closely related problems, but should also perform domain-specific computations to offset the domain shift.
To turn this intuition into a working algorithm, we develop multibranch architecture that sends the data through multiple network branches in parallel. What gives it the necessary flexibility trainable gates that are tuned to modulate and combine the outputs of these branches, as shown in Fig. 1. Assigning to each domain its own set of gates allows the global network to learn what set of

---

Published as a conference paper at ICLR 2020
Bike!
(labeled)
Domain (omlabeled)
Bike!
Figure Domain Adaptive Multibranch Network sequence of computational units (() each which processes the data in parallel branches, whose outputs are then aggregated in a weighted manner by a gate to obtain a single response. To allow for domain-adaptivy computations, cach domain has its own set of gates, one for each computational unit, which combine the branches in different ways. As result, some computations shared across domains while others are -omain-specific.
computations should be carried out for each one. As additional benefit, in contrast to previous strategies for untying the source and target streams (Rozantsev et al., 2018; 2019), our formulation naturally extends more than two domains.
In other words, our contribution is a learning strategy that adaptively adjusts the specific compu- tation to be performed for each domain. To demonstrate that it constitutes effective approach to extracting domain-invariant features, we implement it in conjunction with the popular domain sessifiierbasee method of Ganin & Lempitsky (2015). Our experiments demonstrate that Do- main Adaptive Multibranch Networks, which we will refer to as DAMNets, not only outperform the original technique of Ganin & Lempitsky (2015), but also the state-of-the-ar strategy for untying the source and target weights of Rozantsev et al. (2019), which relies on the same domain classifier. We will make our code publicly available upon acceptance of the paper.
2 RELATED WORK
Domain Adaptation. Domain adaptation has achieved important milestones in recent years (Dai al., 2007; Gretton al., 2009; Pan et al., 2010; Fernando et al., 2013; Sun al., 2016; Shu al., 2018), with deep learning-based methods largely taking the lead in performance. The dominant approach to deep domain adaptation to learn a domain-invariant data representation. This is commonly achieved by finding a mapping to a feature space where the source and target features have the same distribution. the distribution similarity was measured in terms of Maximum Mean Discrepancy (Gretton et al., 2007), while other metrics based on second- and higher-order statistics were introduced in Sun & Saenko (2016); Koniusz et al. (2017); Sun et al. (2017). In Saito et al. (2018), the distribution alignment process was disambiguated by exploiting the class labels, and in Häusser et al. (2017); Shkodrani et al. (2018) by leveraging anchor points associating embeddings between the domains. Another popular approach to learning Nomain-invarian the domain from which sample was drawn, and use adversarial training arrive to features that the classifier can no longer discriminate (Tzeng et al., 2015; Ganin et al., 2016; 2017). This idea has spawned several recent adversarial domain adaptation classification (Hu al., 2018; Zhang al., 2018), semantic segmentation (Hoffman et al., 2018; Chen et al., 2018; Hong et al., 201 (018), and active learning (Su et al., 2019) techniques, and we will use such a classifier.
Tzeng et al. (2014); Long et al. (2015; 2017); Yan et al. (2017),
features is to train classifier to recognize
Closest in spirit to our approach are those that do not share the weights of the networks that process the source and target data (Tzeng al., 2017: Bermúdez--hhacol et al., 2018; Rozantsev et al., 2018: 2019). In Tzeng et al. (2017), the weights were simply allowed to vary freely. In Rozantsev et al. (2018); Bermúdez-Chacón et al. (2018), was shown that regularizing them to remain close to each other was beneficial. More recently, Rozantsev al. (2019) proposed train small networks to map the source weights to the target ones. While these methods indeed untie the source and target weights, the source and target data still undergo the same computations, i.e., number of operations.
In this paper, we argue that the amount of computation, that is, the network capacity, should adapt to each domain and reflect their respective complexities. We rely domain classifier as in Tzeng
2

---

Published as a conference paper at ICLR 2020
et al. (2015); Ganin et al. (2016; 2017). However, we do not force the source and target samples to go through the same transformations, which is counterintuitive since they display different appearance statistics. Instead, we start from the premise that they should undergo different computations and use domain-specific gates to turn this premise into our DAMNet architecture.
Dynamic Network Architectures. As the performance of neural network is tightly linked to its structure, there has been a recent push towards automatically determining the best architecture for the problem at hand. While neural architecture search techniques (Zoph & Le. 2017; Liu et al., 2018: 2019; Pham et al., 2018; Zoph et al., 2018; Real et al., 2019; Noy et al., 2019) aim to find one fixed architecture for given dataset, other works have focused dynamically adapting the network structure at inference time (Graves, 2016; Ahmed & Torresani, 2017: Shazeer et al., 2017; Veit & Belongie, 2018; Wu al., 2018). In particular, in Ahmed & Torresani (2017); Shazeer et al. (2017); Veit & Belongie (2018); 8); Bhatia et al. (2019), gates were introduced for this purpose. While our DAMNets also rely on gates, their role is very different: first, we work with data coming from different domains, whereas these gated methods, with the exception of Bhatia et al. (2019), were all designed work in the single-domain scenario. Second, and more importantly, these techniques aim to define different computational path for every test sample. By contrast, we seek to determine the right computation for each domain. Another consideration that we freeze our gates for inference while these methods must constantly update theirs. We believe this to be ill- suited to domain adaptation, particularly because learning to adapt the gates for the target domain, for which only unlabeled data is available, is severely under-constrained. This lack of supervision may be managcable when one seeks to define operations for whole domain, but not when these operations are sample-specific.
3 METHOD
We now describe our deep domain adaptation approach, which automatically adjusts the computa- tions that the different domains undergo. We first introduce the multibranch networks that form the backbone of our DAMNet architecture and then discuss training in the domain adaptation scenario.
3.1 MULTIBRANCH NETWORKS
Let first consider single domain. In this context, a traditional deep neural network can be thought of a sequence of N, operations (() (1ssiN,) each transforming the output of the previous one. Given an input image x, this can be expressed
f(0)
f(i)
1-1)
(i)
Σ
x (0) x
i
(1)
f())
K
x = (x(1-1))
Figure 2: A computational unit (0) is an aggregation
As general convention, each operation (+)(-) can represent either single layer multi- ple ones. Our formulation extends this defi- nition by replacing each f(i) by multiple par- allel computations, as shown in Fig. 2. More specifically, we replace each f() by computa- tional unit {f(() . و() consisting of K parallel branches. Note that this K can be different at each stage of the network and should therefore be denoted as K((), However, to simplify notation,
of the outputs of parallel computations, or branches, (t)
we drop this index below. Given this definition, we write the output of each computational unit
x ( = S (11"
(x()
(2)
where Σ(-) is an aggregation operator that could be defined in many ways. It could be simple sum- mation that gives all outputs equal importance, or, at the opposite end of the spectrum, a multiplexer that selects single branch and ignores the rest. To cover the range between these two alternatives,
3

---

Published as a conference paper at ICLR 2020
introduce learnable gates that enable the network to determine what relative importance the dif- ferent branches should be given. Our gates perform a weighted combination the branch outputs. Each gate is controlled by set of K activation weights {Ø) Visis' and unit returns
(x))
(3)
If FI of = 1, the gate performs a simple summation. If o = 1 for single j and 0 for the others, it behaves as a multiplexer. The activation weights of enable us to modulate the computational graph of network block f(i) To bound them and encourage the network to either select or discard each branch in computational unit, we write them in terms of sigmoid functions with adaptive steepness. That is,
of (1 (1+exp exp (-x(0) )²"))
(4)
where the 9j are learnable unbounded model parameters, and (+)) controls the plasticity of the (() varies between the extreme values O and for block i. During training, we initially set (4) to a small value, which enables the network to explore different gate configurations. We then apply cooling schedule on our activations, by progressively increasing (()) over time, so as to encourage the gates to reach firm decision. Note that our formulation does not require Σラ= 0/ 1, that is, we do not require the aggregated output x(() to be convex combination of the branch outputs f (x)-11)) This is deliberate because allowing the activation weights to be independent from one another provides additional flexibility for the network to learn general additive relationships.
activation- the rate at which o
Finally, a Multibranch Network is the concatenation of multiple computational units, as shown in
Fig. 1. For the aggregation within each unit f(i) be possible, the f3'ss outputs must be of matching shapes. Furthermore, as in standard networks, two computational units can be attached only if the output shape of the first one matches the input shape of the second. Although would be possible to define computational units at any point in the network architecture, in practice, we usually take them correspond groups of layers that are semantically related. For example, one would group a succession of convolutions, pooling and non-linear operations into the same computational unit.
3.2 DOMAIN ADAPTIVE MULTIBRANCH NETWORKS
3.2. TWO DOMAINS
Our goal is to perform domain adaptation, that is, leverage large amount of labeled images, X {xi,..., x } with corresponding annotations Y = "y",...,, y J. drawn from a source domain, to train a model for target domain, whose data distribution is different and for which we only have access to unlabeled images X = {xf..... x/r).
To this end, we extend the gated networks of Section 3.1 by defining two sets of gates, one for the source domain and one for the target one. Let {())))))
1-=1 and (0)))() ( f-=1 be the corresponding source and target activation weights for computational unit f(0), respectively. Given a sample x" coming from a domain d E {s, we take the corresponding output of the i-th computational unit to be
(x I004 1/00 (1x))6-1)).
(5)
Note that under this formulation, the domain identity of the sample is required in order to select the appropriate (0)
The concatenated computational units forming the DAMNet encode sample x from domain into feature vector f(x, d). Since the gates for different domains are set independently from
4

---

Published as a conference paper at ICLR 2020
another, the outputs of the branches for each computational unit combined in domain- specific manner, dictated by the activation weights (dd), Therefore, the samples are encoded to common space, but arrive to it through potentially different computations. Fig. 3 depicts this process. Ultimately, the network can learn to share weights for computational unit f()) by setting (6r) [ Vj. It can also learn fully untie the weights by having A n A 0, where A and AT denote the set of non-zero activations in the two domains. Finally, in contrast to Tzeng al. (2017); Bermidez-Chacóó
al. (2018); Rozantsev et al. (2018; 2019), can learn use more computation for one domain than for the other by setting (d"),) > for two different branches f/()
while having only a single non-zero (Ø*), for a particular computational unit f())
The above formulation treats all branches for each computational unit as potentially sharable between domains. However, it is sometimes de- sirable not to share at all. For example, batch- normalization layers that accumulate and update statistics of the data over time, even during the forward pass, are best exposed to single do- main to learn domain-specific statistics. We al- low for this by introducing computational units
Source domain
Target domain
whose gates are fixed, yet domain specific, and
that therefore act as multiplexers.
After the last computational unit, a small net- work Py operates directly on the encodings and returns the class assignment ý = Py (z), thus sub- jecting the encodings for all samples to the same set of operations.
Figure 3: Computational graphs for the source (top) and target (bottom) domains, for the same network. While both domains share the same computational units, their outputs are obtained by different aggre- gations of their inner operations, e.g., in the first unit, the source domain does not the middle two oper- ations, whereas the target domain does; by contrast, both exploit the fourth operation. In essence, this scheme adapts the amount of computation that each domain is subjected to.
3.2.2 MULTIPLE DOMAINS
The formulation outlined above extends naturally more than two domains, by assigning one of gates per domain. This enables us to exploit an- notated data from different source domains, and even to potentially handle multiple target domains simultaneously. In this generalized case, we in- troduce governing sets of gates with activations のه . ddo for D different domains. They act in the same way as in the two-domain case and the overall architecture remains similar.
3.2.3 TRAINING
When training our models, we jointly optimize the gate parameters (g*))² (?) from Eq. 4, along with the other network parameters using standard nack-propagation. To this end, we make use of a composite loss function, designed to encourage correct classification for labeled samples from the source domain(s) and align the distributions of all domains, using labeled and unlabeled samples. This loss can be expressed as
ReJal
LDAMNet
|e| L.
Σ
La(d,,, dn)
(6)
where l and u are the sets of labeled and unlabeled samples, respectively, and where we assumed, without loss of generality, that the samples are ordered.
The first term in this loss, Ly 5), the standard cross-entropy, which compares the ground- truth class probabilities with the predicted ones ý (z), where, as discussed in Section 3.2.1, z = =(x,d) d) the feature encoding sample x from domain For the second term, which encodes distribution alignment, we rely the domain confusion strategy of Ganin & Lempitsky (2015), which is commonly used in existing frameworks. Specifically, for D domains, we make use of an auxiliary domain classifier network pa that predicts D-dimensional vector of domain probabilities given the feature vector Following the gradient reversal technique of Ganin & Lempitsky
5

---

Published as a conference paper at ICLR 2020
(2015), express the second term in our loss La(d, D-dimensional binary vector encoding the ground-truth domain, d; indicates the i-th element of d, and d = Pa R(z)), with R the gradient reversal pseudofunction of Ganin & Lempitsky (2015) that enables incorporate adversarial training directly into back-proyy back-propagation. That is, with this loss, standard back- propagation trains jointly the domain classifier to discriminate the domains and the feature extractor f(-) to produce features that fool this classifier.
- - d, l,log(d,) where d is the
When training is complete and the gates have reached stable state, the branches whose activations are close to zero are deactivated. This prevents the network from performing computations that are irrelevant and allows us to obtain more compact network to process the target data.
EVALUATION
4.1 BASELINES
Since we rely on the domain confusion loss to train our model, we treat the Domain Domain-Adversarr Adversarial Neural Network (DANN) method of Ganin & Lempitsky (2015), as our first baseline.
To demonstrate the benefits of our approach over simply untying the source and target stream param- eters, we compare our approach against the Residual Parameter Transfer (RPT) method of Rozantsev et al. (2019), which constitutes the state of the art in doing so. Note that RPT also relies on the do- main confusion loss, which makes our comparison fair. In addition, we report the results of directly applying a model trained on the source domain to the target, without any domain adaptation, which refer to "No DA". We also provide the oracle accuracy of model trained on the fully-labeled target domain, referred to as "On TD".
4.2 IMPLEMENTATION DETAILS
We adapt different network architectures to the multibranch paradigm for different adaptation prob- lems. For all cases, we initialize our networks' parameters by training the original versions of those architectures the source domains, either from scratch, for simple architectures, or by fine-tuning weights learned on ImageNet, for very deep ones. We then set the parameters of all branches to the values from the corresponding layers. We perform this training the predefined training splits, when available, or on 75% of the images, otherwise. The initial values of the gate parameters are defined as set the activations R. for each of the K branches. This prevents our networks from initially favoring a particular branch for any domain.
To train networks, we use Stochastic Gradient Descent with momentum of 0.9 and variable learning rate defined by the annealing schedule of Ganin & Lempitsky (2015) as μp = [I+o-p)7 where p is the training progress, relative to the total number of training epochs, 8i0 is the initial learning rate, which we take to be 10 and a = 10 and 3 = 0.75 as in Ganin & Lempitsky (2015). We eliminate exploding gradients by f--norm clipping. Furthermore, of the activations at every gate (()) progresses. As data preprocessing, we apply mean subtraction, as in Ganin & Lempitsky (2015). We train for 200 epochs, during which the network is exposed to all the image data from the source and target domains, but only to the annotations from the source domain(s).
(40
modulate the plasticity p. that is, we make n decay linearly as training
Our "On TD" oracle is trained either the preset training splits, when available, or defined training data, and evaluated on the corresponding test data. For the comparison to this oracle to be meaningful, follow the same strategy for our DAMNets. That is, we the unlabeled target data from the training splits only and report results on the testing splits. This protocol differs from that of Rozantsev al. (2019), which relied on transductive evaluation, where all the target images, training and test ones, were seen by the networks during training.
4.3 IMAGE RECOGNITION
We evaluate method in the task of image recognition for which several domain adapta- tion benchmark problems: Digits, which comprises three domains: MNIST (LeCun et al., 1998), MNIST-M (Ganin& & Lempitsky, 2015), and SVHN (Netzer al., 2011); Office (Saenko et al.,

---

Published as a conference paper at ICLR 2020
Table 1: Domain Adaptation datasets and results. We compare the accuracy of DAMNet approach with that of DANN (Ganin & Lempitsky. 2015) and RPT (Rozantsev et al.. 2019), for image classification tasks commonly used to evaluate domain adaptation methods. Our DAMNets yield a significant accuracy boost in the presence of large domain shifts, particularly when using more than one souree domain. A more comprehensive
evaluation all datasets is provided in Appendix D.
Digits: MNIST (M), MNIST-M M (MM), SVIIN (S)
Office Home: Ari (A), Clipart (C), Product (P), Real (R)
Source(s)
M
5
M
MM
M.MM
M.MM
A
C
C
C.P
A.C.P
Target
MM
M
S
S
5
S+
No DA
52.25
54.90
25.57
27.49
33.52
22.88
37,03
36.67
29.65
50.91
53.12
43.03
46.42
59,39
58.72
DANN
76.66
73.90
31.69
37.43
44.16
49.02
58.50
70.50
47.93
57.68
$7.90
70.53
72.00
RPT
82.24
78.70
34.72
37.90
n/a
n/u
54.51
63.18
47.32
51,90
52.15
55.05
62.16
n/a
nda
Ours
88.80
81.30
37,95
39.41
51.83
79.45
59.30
77.50
51.24
60.74
59,90
62.70
65.00
72.25
77.65
On TD
96.21
99.26
89.23
89.27
89.23
96.07
87.66
87.66
64.42
64.42
77.80
77.80
77.80
77.80
77.80
2010), which contains three domains: Amazon, DSLR, and Webcam; Office-Home (Venkateswara et al., 2017), with domains Art, Clipart, Product, and Real; and VisDA17 (Peng et al., 2018), with Synthetic and Real images. As all these are well studied benchmark datasets, we provide full de- scriptions and image examples evidencing the different degrees of domain shift in Appendix B.
Sctup. As discussed in Section 3, our method is general and can work with any feed-forward net- work architecture. To showcase this, for the digit recognition datasets, we apply it to the LeNet and SVHNet architectures (Ganin & Lempitsky, 2015), which are very simple convolutional net- works, well suited for small images. Following Ganin & Lempitsky (2015), we employ LeNet when using the synthetic datasets MNIST and MNİST-M as source domains, and SVHNet when SVHN acts as source domain. We extend these architectures to multibranch ones by defining the compu- tational units as the groups of consecutive convolution, pooling and non-linear operations defined in the original model. For simplicity, we use many branches within each computational unit as we have domains, and all branches from a computational unit follow the same architecture, which provide in Appendix A, Figures and 2. As backbone network to process all the rest of the datasets, we use a ResNet-50 (He et al., 20,6), with the bottleneck layer modification of Rozantsev al. (2019). While many multibranch configurations can be designed for such a deep network, we choose to make our gated computational units coincide with the layer groupings defined in He et al. (2016), namely convl, conv2.x, conv3_x, conv4_x, and conv5 xx. The resulting multibranch network is depicted in Appendix A, Figure 4. We feed our DAMNets images resized to 224 x 224 pixels, as expected by ResNet-50.
Results. The results for the digit recognition and Office-Home datasets are provided in Table 1. Results for Office and VisDA17 datasets are presented in Appendix D. Our approach outperforms the baselines in all cases.
For the Digits datasets, in addition to the traditional two-domain setup, we also report results when using two source domains simultaneously. Note that the reference method RPT (Rozantsev et al., 2019) does not apply to this setting, since it was designed to transform a single set of source parame- ters to the target ones. Altogether, two columns correspond to the combinations reported in the literature. We believe, however, that the SVHN MNIST one is quite artificial, since, in practice, one would typically annotate simpler, synthetic images and aim to use real ones at test time. We therefore also report synthetic >> SVHN cases, which much more challenging. The multi-source version of our method achieves signif- icant boost over the basclines in this scenario. To further demonstrate the potential of our approach in this setting, we replaced its backbone with the much deeper ResNet-50 network and applied on upscaled versions of the images. As shown in the column indicated by a *, this allowed us to achieve an accuracy close to 80%, which is remarkable for such a difficult adaptation task.
method consistently outperforms the others. Note that the first
On Office-Home, the gap between DAMNet and the baselines is again consistent across the dif- ferent domain pairs. Note that, here, because of the relatively large number of classes, the overall performance is low for all methods. Importantly, our results show that we gain performance by training on more than one source domain, and by leveraging all synthetic domains to transfer to the real one, our approach reaches an accuracy virtually equal to that of using full supervision on the target domain. Despite our best efforts, we were unable to obtain convincing results for RPT using the authors' publicly available code, as results for this dataset were originally reported for RPT.
Gate dynamics. To understand the way our networks learn the Nomain--specife branch assign- ments, we track the state of the gates for all computational units over all training epochs. In Figure 4,
7

---

Published as a conference paper at ICLR 2020
plot the corresponding evolution of the gate activations for the DSLR+Webcam Amazon task on Office. Note that our DAMNet leverages different branches over time for each domain before reaching firm decision. Interestingly, we performs low-level computations, DSLR and Webcam share all branches. By contrast, Amazon, which has significantly different appearance, mostly uses its branches, except in compu- tational units. This evidences that our network successfully understands when domains are similar and can thus use similar computations.
that, with the exception of the first unit, which
4.4 OBJECT DETECTION
Average precision
We evaluate our method for the detection of drones from video frames, on the UAV-200 dataset (Rozantsev et al., 2018), which contains examples of drones both generated artificially and captured from real video footage. Full de- tails and example images are provided in Appendix B.3
Method
No adaptation
0.377
DANN (Ganin & Lempitsky, 2015)
0.715
ADDA (Tzeng ct.l,, 2017)
0.731
Two-stream (Rozantsev al., 2018)
0.732
Setup. Our domain adaptation leverages both the syn- thetic examples of drones, as source domain, and the limited amount of annotated real drones, as target domain, well predict the class
RPT (Rozantsev et al., 2019)
0.743
DAMNet
0.792
as the background negative examples, of patches from the validation set of real images. We fol- low closely the supervised setup and network architecture of Rozantsev et al. (2019), including the use of AdaDelta optimizer, cross-entropy as loss function, and average preci- sion as evaluation metric. Our multibranch computational units are defined groupings of succes- sive convolutions, nonlinearities, and pooling operations. The details of the architecture are provided in Appendix A, Figure
Table 2: Average precision of
DAMNet approach with several other ref- erence methods, for domain adaptation from synthetic to real images of drones.
Results. Our method considerably surpasses all the others in terms of average precision, as shown in Table 2, thus validating DAMNet as effective models for leveraging synthetic data for domain adaptation in real-world problems.
4.5 DAMNET AS A GENERAL FEATURE EXTRACTOR
We validate the effectiveness of our method as feature ex- tractor, by combining with the Maximum Classifier Dis- crepancy (MCD) method of Saito et al. (2018). As MCD op- crates on the extracted encodings, we replace the encoding strategy that MCD uses, which is the same as DANN, with our DAMNet. Or, in other words, we replace the domain classifier in our approach with the corresponding MCD term. Specifi- cally, we use a single computational unit with two branches, each of which replicates the architectures proposed in Saito et al. (2018).
Table 3: We boost the method of Saito al. (2018) by replacing their feature extraction with our DAMNets.
MCD accuracy
DAMNer
DAMNer
MNIST.M
SVHN
38.54
41.51
DSLR Amazon
67.24
67.81
Webcam
64.33
66.19
Clipart
Real
63.50
63.87
We present the results of combining MCD with DAMNet in Table 3. In all tested scenarios, obtain improvements over using MCD as originally proposed.
4.6 BRANCH ARCHITECTURES
To obtain more insights about specific branch decisions, we evaluate the effects of adding extra branches to the network, as well as using branches with different capacities.
4.6.1 BRANCHES WITH DIFFERENT CAPACITIES
When computational units are composed of branches of different capacities, DAMNets often assign branches with more capacity to more complex domains. To exemplify this, we trained a modified multibranch SVHNet for adaptation between MNIST and SVHN. Instead of the identical branches originally used, we replace the second branch in each computational unit with a similar branch where the convolution operation is performed by 1x1 rather than 5x5 kernels. These second branches, with
8

---

Published as a conference paper at ICLR 2020
conv2__
conve
Office WOX Office DIR
2
Aourch Coards
Clards
Crarcs
Pearch
0ffice
Earch
Darch
Figure 4: Evolution of the gates' activations for each of the computational units in a multibranch ResNet- 50 network, for the Office DSLR + Webcam & Amazon domain adaptation problem. In the top two rows, we show the gates for the source domains and in the bottom row for the target one. All branches are initialized parameters obtained from a single ResNet-50 trained on ImageNet. Note how for the first computational unit, convl, cach domain chooses to process the data with different branches. In the remaining units, the two source domains, which have similar appearance, share all the computations. By contrast, the target domain still uses its own branches in conv3_x. x. and conv4_x _x to account for its significantly different appearance. When arriving conv 5x, the data has been converted to domain-agnostie representation, and hence the same branch operate on all domains.
25 times fewer parameters each, are mostly used by the simpler domain in this case. We provide the gate evolution that reflects this in Appendix C, Figures 5 and 6.
4.6.2 ADDITIONAL BRANCHES
We explore the effects of using more branches than domains, so as provide the networks with alternative branches from where to choose. In particular, we explore the case where K = D + 1. We evaluate multibranch LeNet and ResNet architectures under this setting. We show the gate activation evolution in Appendix C, Figures and 8. During the training process, we have observed that the networks quickly choose to ignore extra branches when K D. This suggests that they did not contribute to the learning of our feature extraction. We did not find experimental evidence to support that K > is beneficial.
CONCLUSION
We have introduced a domain adaptation approach that allows for adaptive, separate computations for different domains. Our framework relies on computational units that aggregate the outputs of multiple parallel operations, and on a set of trainable domain- specific gates that adapt the aggrega- tion process to each domain. Our experiments have demonstrated the benefits of this approach over the state-of-the-art weight untying strategy; the greater flexibility of our method translates into a consistently better accuracy.
Although we only experimented with using the same branch architectures within a computational unit. our framework generalizes to arbitrary branch architectures, the only constraint being that their outputs are of commensurate shapes. An interesting avenue for future research would therefore be to automatically determine the best operation to perform for each domain, for example by combining our approach with neural architecture search strategies.
9

---

Published as a conference paper at ICLR 2020
REFERENCES
K. Ahmed and L. Torresani. Connectivity Learning in Multi-Branch Networks, In NIPS Meta- learning Workshop, 2017.
R. Bermúdez-Chhaón, P. Márquez-Neila, M. Salzmann, and P. Fua. A Domain-Adaptive Two- Stream U-Net for Electron Microscopy Image Segmentation. In International Symposium Biomedical Imaging, pp. 400-404, April 2018.
P. Bhatia, K. Arumae, and E. B. Celikkaya. Dynamic transfer learning for named entity recognition. In International Workshop on Health Intelligence, pp. 69-81. Springer, 2019.
Y. Chen, W. Li, and L. Van Gool. ROAD: Reality Oriented Adaptation for Semantic Segmentation of Urban Scenes. In Conference on Computer Vision and Patlern Recognition, pp. 7892-7901, 2018.
W. Dai, Q. Yang, G.R. Xue, and Y. Yu. Boosting for Transfer Learning. In Machine Learning, pp. 193-200, 2007.
B. Fernando, A. Habrard, M. Sebban, and T. Tuytelaars. Unsupervised Visual Domain Adaptation Using Subspace Alignment. In International Conference on Computer Vision, pp. 2960-2961, 2013.
Y. Ganin and V. Lempitsky. Unsupervised Domain Adaptation by Backpropagation. In International Conference Machine Learning, pp- 1180-1189, 2015.
Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and S. Lempitsky. Comain-Adversarile Training of Neural Networks. Journal of Machine Learning Research, 17:591-5999 -595, 2016.
Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. S. Lempitsky. Domain-Adversarial Training of Neural Networks. In Domain Adaptation in Computer Vision Applications., pp. 189-209. 2017.
A. Graves. Adaptive Computation Time for Recurrent Neural Networks. In arXiv Preprint, 2016.
A. Gretton, K.M. Borgwardt, M.J. Rasch, B. Schölkopf, and A.J. Smola. A Kernel Method for the Two-lample Problem. In Advances in Neural Information Processing Systems, pp. 513-520, 2007.
A. Gretton, A. Smola, J. Huang, M. Schmittfull, K. Borgwardt, and B. Schölkopf. Covariate Shift by Kernel Mean Matching. Journal of the Royal Statistical Society, 3(4):5-13, 2009.
P. Häusser, T. Frerix, A. Mordvintsev, and D. Cremers. Associative Domain Adaptation. In Inter- national Conference on Computer Vision, pp. 2784-2792, 2017.
K. He, X. Zhang, S. Ren, and Sun. Deep Residual Learning for Image Recognition. In Conference Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Hoffman, Tzeng, Park, J. Zhu, Isola, Saenko, A. Efros, and T. Darrell. CyCADA: Cycle Consistent Adversarial Domain Adaptation. In International Conference on Machine Learning, pp. 1989-1998, 2018.
W. Hong, Z. Wang. M. Yang, and J. Yuan. Conditional Generative Adversarial Network for Struc- tured Domain Adaptation. In Conference Computer Vision and Pattern Recognition, pp. 1335- 1344, 2018
L. Hu, M. Kan, S. Shan, and X. Chen. Duplex Generative Adversarial Network for Unsupervised Domain Adaptation. In Conference on Computer Vision and Pattern Recognition, pp. 1498-1507, 2018.
P. Koniusz, Y. Tas, and F. Porikli. Domain Adaptation by Mixture of Alignments of Second- or Higher- Higher-Order Scatter Tensors. In Conference on Computer Vision and Pattern Recognition, pp- 4478 4487, 2017.
10

---

Published as a conference paper at ICLR 2020
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient- Beaeed Learning Applied to Document Recognition. Proceedings of the IEEE, pp. 2278 22224, 1998.
C. Liu, B. Zoph, J. Shlens, W. Hua, L. Li, L. Fei-Fei, A. Yuille, J. Huang, and K. Murphy, Progres- sive Neural Architecture Search. In European Conference on Computer Vision, 2018.
H. Liu, K. Simonyan, and Y. Yang. DARTS Differentiable Architecture Search. In International Conference on Learning Representations, 2019.
M. Long, Y. Cao, J. Wang, and M. I. Jordan. Learning Transferable Features with Deep Adaptation Networks. International Conference on Machine Learning, pp- 97-105, 2015.
M. Long, J. Wang, and M.I. Jordan. Deep Transfer Learning with Joint Adaptation Networks. In International Conference Machine Learning, pp. 2208-2217, 2017.
Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Ng. Reading Digits in Natural Images with Unsupervised Feature Learning. In Advances in Neural Information Processing Systems, 2011.
A. Noy, N. Nayman, T. Ridnik, N. Zamir, S. Doveh, T. Friedman, R. Giryes, and L. Zelnik-Manon ASAP: Architecture search, anneal and prune. arXiv preprint arXiv: 190.:444444123, 2019.
S.J. Pan, I. Tsang, J. Kwok, and Q Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, 22(2):199-210 2010.
X. Peng, B. Usman, N. Kaushik, D. Wang, J. Hoffman, and K. Saenko. VisDA: A synthetic-to-ren benchmark for visual domain adaptation. Conference on Computer Vision and Pattern Recogni- fion Workshops, 2018.
H. Pham, M. Guan, Zoph, Le, and J. Dean. Efficient Neural Architecture Search via Parameter Sharing. In International Conference on Machine Learning, 2018.
E. Real, A. Aggarwal, Y. Huang, and Quoc V Le. Regularized evolution for image classifier chitecture search. In American Association for Artificial Intelligence Conference, volume 33, pp- 4780-4789, 2019.
A. Rozantsev, M. Salzmann, and P. Fua. Residual Parameter Transfer for Deep Domain Adaptation. In Conference on Computer Vision and Pattern Recognition, pp. 4339-4348, 2018
A. Rozantsev, M. Salzmann, and P. Fua. Beyond Sharing Weights for Deep Domain Adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(4))801-814 2019.
K. Saenko, B. Kulis, M. Fritz, and T. Darrell. Adapting Visual Category Models to New Domains.
In European Conference Computer Vision, pp. 213-226, 2010.
K. Saito, K. Watanabe, Y. Ushiku, and T. Harada. Maximum Classifier Discrepancy for Unsuper- vised Domain Adaptation. In Conference Computer Vision and Pattern Recognition, 2018.
N. Shazeer, A. Mirhoseini, K. Maziarz, Davis, Q. Le, G. Hinton, and Dean. Outrageously Large Neural Networks: The Sparsely- Gated Mixture-Of-Expert Layer. In International Conference on Learning Representations, 2017.
S. Shkodrani, M. Hofmann, and F. Gavves. Dynamic adaptation on non-stationary visual domains.
In European Conference on Computer Vision, 2018.
R. Shu, H. Bui, H. Narui, and S. Ermon. A DIRT-T approach to unsupervised domain adaptation. In International Conference on Learning Representations, 2018.
J. Su, Y. Tsai, K. Sohn, B. Liu, S. Maji, and M. Chandraker. Active adversarial domain adaptation. Conference on Computer Vision and Pattern Recognition Workshops, 2019.
B. Sun and K. Saenko. Deep CORAL: Correlation Alignment for Deep Domain Adaptation. In European Conference Computer Vision, pp. 443-450, 2016.

---

Published as a conference paper at ICLR 2020
B. Sun, J. Feng, and K. Saenko. Correlation Alignment for Unsupervised Domain Adaptation. arXiv Preprint, 2016.
B. Sun, J. Feng, and K. Saenko. Correlation Alignment for Unsupervised Domain Adaptation. In Domain Adaptation in Computer Vision Applications, pp. 153-171. 2017.
E. Tzeng, Hoffman, N. Zhang, K. Saenko, and T. Darrell. Deep Domain Confusion: Maximizing for Domain Invariance. In arXiv Preprint, 2014.
E. Tzeng, J. Hoffman, T. Darrell, and K. Saenko. Simultaneous Deep Transfer Across Domains and Tasks. In International Conference on Computer Vision, pp. 4068-4076, 2015.
E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell. Adversarial Discriminative Domain Adaptation. In Conference Computer Vision and Puttern Recognition, pp. 7167-7176, 2017.
A. Veit and S. Belongie. Convolutional Networks with Adaptive Inference Graphs. European Conference on Computer Vision, pp. 3-18, 2018.
H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan. Deep Hashing Network for Unsupervised Domain Adaptation. Conference on Computer Vision and Pattern Recognition, pp. 2017.
Z. Wu, T. Nagarajan, A. Kumar, S. Rennie, L. Davis, K. Grauman, and R. Feris. Blockdrop: Dy- namic Inference Paths in Residual Networks. In Conference on Computer Vision and Pattern Recognition, pp. 8817-8826, 2018.
H. Yan, Ding, Li, Q. Wang, Y. Xu, and W. Zuo. Mind the Class Weight Bias: Weighted Maximum Mean Discrepancy for Unsupervised Domain Adaptation. In Conference on Computer Vision and Pattern Recognition, pp. 2272-2281, 2017.
W. Zhang, W. Ouyang, W. Li, and Xu. Collaborative and Adversarial Network for Unsupervised Domain Adaptation. In Conference Computer Vision and Pattern Recognition, pp. 3801-3809, 2018.
B. Zoph and Q. Le. Neural Architecture Search with Reinforcement Learning. In International Conference on Learning Representations, 2017.
B. Zoph, V. Vasudevan, Shlens, and Q Le. Learning transferable architectures for scalable image recognition. In Conference on Computer Vision and Pattern Recognition, pp. 8697-8710, 2018.
12

---

Published as a conference paper at ICLR 2020
Appendices
A MULTIBRANCH ARCHITECTURES
Below, we provide the network architectures and detailed building blocks of our Domain Adaptive Multibranch Networks, for the single source domain case (D 2). Fach computational unit enclosed by dotted lines. The input and output shapes for all layer groupings provided.
Source
conv1
conv2
Feature classtfter
Domain
Tanget Domain
classifier
conv1
conv2
Feature classifier
Domain classifier
Figure I: Multibranch LeNet. This architecture is multibranch extension the LoNet used by DANN (Ganin Lempitsky. 2015).
Sourcce
convi
ConV2
conv3
Feature classifter
Target
conv1
conv2
conv3
Donata classifter
CORV2
COnV2
coav2
Feature classtfter
Donata classifler
Figure 2: Multibranch SVHNet.
This architecture multibranch extension to the SVHNet used by
DANN (Ganin & Lempitsky. 2015).
Source Domain
convi
conv2
conv3
Feature classifier
Target Domain
convi
conv2
conv3
Domain classifier
cOnv1
CORV2
conv3
Feature classtfter
Domatu classtfier
Figure 3: Multibranch architecture for drone detection. architecture multibranch extension to one used by Rozantsev et al. (2019).
13

---

Published as a conference paper at ICLR 2020
Target Domain
Source Domain
76,,
ZAnOz
Botchssorn
batcittorm
Batck
FeLU
ReUU
Paxhvol
conv4 a
conv2.4n a
Seationdd
Sancwiorn
Batchllore
conv1
conv1
Belli
Relli
Corv Ixl, 1004
356
conv2.
BatoMMarm
BaLLiAore
Helli
BELLI
56c501356
36x24x2854
36v58x238
5oc594356
conv2_X
corV2
conv3 X
255
conv4 bb, c,d,e,f)
conv2 c)
Con 1e3, 255
Cony Iul, 66
20c2he512
katclssre
Corv Icl, 1006 dn
Cote Ixd, 356
Eatchiion
Batcchlore
conv4
364D4r2024
56x50e256
14:2402024
lal,
conv4
Fannissonn
Rarzhlore
14x2600804
conv5 X
8 SAUO
253, 512
263, 128
ButcWiorm
Batchloree
Beill
Butchiorm
Batchioor
bott Leneck
7x7x2800
Iverage
Bell
fally-ase.
255
Feature classifier
Pally com.
Sancviorn
Batchlore
bottleneck
convs (b, c)
TelUi
conv3 (b. c.
Relli
512
138
NalcMarm
NUU
INUUi
colky-comm
Core Ixl, 3046
512
Donatr Ln class Lfsstffer
ReLlil
Pully-coen:
Donain
Feature classifier
ReLU
Fully-carn..
7RNORAS12
Figure 4: Multibranch ResNet-50. This architecture is adapted from the original ResNet-50 (Hl et al., 2016). We preserve the groupings described in the original paper (He et al., 2016). N denotes the number of classes the dataset.
14

---

Published as a conference paper at ICLR 2020
B BENCHMARK DATASET DESCRIPTIONS
B. DIGIT RECOGNITION
MNIST (LeCun al., 1998) consists of black and white images of handwritten digits from
images are size 28 pixels. The standard training lesting splits contain 60,000 and 10,000 examples, respectively.
MNIST-M (Ganin & Lempitsky, 2015) is synthetically generated by randomly replacing the foreground and background pixels of random MNIST samples with natural images. Its image size is 32 x 32, and the standard training and testing splits contain 59,001 and 9,001 images, respectively.
SVHN (Netzer et al., 2011), the Street View House Numbers dataset, consists of natural scene images of numbers acquired from Google Street View. Its images are also of size 32 x 32 pixels, and its preset training and testing splits are of 73,257 and 26,032 images, respectively. The SVHN images are centered at the desired digit, but contain clutter, visual artifacts, and distractors from its surroundings.
/
2
3
S
6
7
8
9
MUST
/
3
4
5
6
7
10
2
3
6
28
19
SVHN
B.2 OBJECT RECOGNITION
Office (Saenko et al., 2010) is a multiclass object recognition benchmark dataset, containing images of 31 categories of objects commonly found in office environments. contains color images from three different domains: 2,817 images of products scraped from Amazon, 498 images acquired using DSLR digital camera, and 795 images captured with webcam. The images are of arbitrary sizes and aspect ratios.
Bike
Desk
Fil cabkiel
Backpack
Bicycle
helmnt
Bookicsse
Botte
Calculattor
Chail
lemp
Computer
Headphones
Amazon
DSLR
Webcom
Office- Home (Venkateswara et al., 2017) comprises larger corpus of color, urbitrarily-sizer images from 65 different classes of objects commonly found in office and home environments, coming from four different domains. contains 2.427 images extracted from paintings (Art), 4,365 clipart images (Clipart), 4,439 pho- tographs of products (Product), and 4,357 pictures captured with a regular consumer camera (Real world).
clock
Backpack
Bettery
Bed
Bicycle
Bollle
Bucket
Caloulator
Coler dar
Cuncle
Chair
Gisboard
Computer
Hib
Clean
1208
Product
Heal world
15

---

Published as a conference paper at ICLR 2020
VisDA 2017 (eng al., 2018) includes images of diverse sizes from 12 different categories, coming from two different domains: $5,368 synthetic renders models, 152.397 photographs real-world objects. It is larger than the other two datasets, and exhibits more significant domain shift.
Aeroptare
Bayye
Motorcydle
Plant
Skateboard
Train
Truck
Real
Synthdo
B.3 OBJECT DETECTION
UAV-200 aggregates 200 images of real drones and around 33,000 synthetic ones, well as around 190,000 patches obtained from the background of the video, which do not contain drones, used negative examples. examples are of size pixels. We evaluate performance on validation set comprising 3,000 positive and 135,000 negative patches.
Real
Synthelic
Backoround
16

---

Published as a conference paper at ICLR 2020
C ADDITIONAL EXPERIMENTS
2.8
3.3
2.5
0.
0.
0.1
30
15
Fgnch
Epech
Fpech
Figure 5: Gate evolution for multibranch SVHN network with branches of different capacities, Branch I is the original branch that applies 5x5 convolutions to the image, whereas branch 2 is similar architecture but with Ixl convolutions instead. The network quickly recognizes SVHN requires more complex processing and hence assigns the respective branch it for computational units 1 and
Dranch
2.6
2.4
0.2
Epach
Figure 6: Gate evolution for multibranch LeNet network with branches of different capacities. We have simplified the architecture to encapsulate the feature extraction into a single computational unit in this case. Similarly the above, modify the second branch for simpler computation. The original branches apply convolution operations extract 32 channels with 5x5 kernel, and then extract 48 channels from those with a 5x5 kernel. We replace them in the second branch with 24 channels 3x3 kernel and channels 1xl kernel convolutions, respectively, which yields commensurate shapes with the original branch, but with more than 20 times fewer parameters. Unlike in the above experiment, we do not force the gates to open close. The network still assigns combinations of branches that reflect the difference in visual complexity of the domains.
17

---

Published as a conference paper at ICLR 2020
unitl
unit2
MMWY
38
16
24
0
16
24
32
40
56
1.0
1.0
Brance
Brarth12
Branch
Browc
Bearch
Branch
3.0
0.0
Eooch
Epoch
Figure 7: Effect of adding extra branches to LeNet multibranch network. We augment the original multibranch LeNet with third hranch under the branch architecture the original one. The network rapidly decides to ignore this overparametrization. The additional branch does not have an effect in the final activation of gates, nor does it help during training.
Figure 8: Augmenting a multibranch ResNet-50 a similar effect as the above. One of branches discarded early on by each computational unit.
18

---

Published as a conference paper at ICLR 2020
D FULL RESULTS
Table 4: Domain Adaptation results. compare accuracy of our DAMNet approach with that DANN (Ganin & Lempitsky, 2015) and of RPT (Rozantsev et al.. 2019), for image classification tasks com- monly used to evaluate domain adaptation methods. As illustrated in Appendix B, different source and target domain combinations present various degrees of domain shift, and some combinations elearly more chal- lenging than others. Our DAMNets yield significant accuracy boost the presence of large domain shifts, particularly when using more than one source domain.
Datasets
Source(s) Target
No DA
DANN
RPT
DAMNet On TD
MNIST MNIST-M
52.25
76.66
82.24
88.80
96.21
SVHN MNIST
54.90
73.90t
78.70
81.30
99.26
MNIST SVHN
25.57
31.69
34.72
37.95
89.23
Digits
MNIST-M SVHN
27.49
37.43
37.90
39.41
89.23
MNIST MNNST-M SVHN
33.52
44.16
n/a
51.83
89.23
MNIST MNISTTM & SVHN
22.88
49.02
n/a
79.45
96.07
Webcam DSLR
93.60
99.20
99.401
99.62
95.20
Amazon DSLR
32.80
79.10t
82.701
84.14
95.20
DSLR Webcam
90.45
97.70t
98.00t
98.11
98.49
Office
Amazon
Webcam
34.67
78.90t
81.50f
85.28
98.49
Webcam
Amazon
41.42
62.80
63.601
65.67
85.11
DSLR
Amazon
34.47
63.60
64.701
64.82
85.11
DSLR Webcam Amazon
45.82
64.86
n/a
68.87
85.11
Art Product
37.03
58.50
54.51
59.30
87.66
Clipart Product
36.67
70.50
63.18
77.50
87.66
Clipart & Art
29.65
47.93
47.32
51.24
64.42
Real world Art
50.91
57.68
51.90
60.74
64.42
Office-Home
Art D Real world
53.12
56.40
52.15
59.90
77.80
Clipart Real world
43.03
57.90
55.05
62.70
77.80
Product Real world
46.42
62.30
62.16
65.00
77.80
Clipart+ Product Real world
53.39
70.53
n/a
72.25
77.80
Art + Clipart Product & Real world
58.72
72.00
n/a
77.65
77.80
Synthetic Real
35,46
59.90
61.10
61.40
84.72
VisDA 2017
Real & Synthetic
51.12
83.10
82.15
85.20
99.34
UAV-200
Synthetic & Real*
0.377
0.715
0.743
0.792
0.858
Acccrrcyy reported Ganin Lempitsky (2015) and Rozantsev et al. (2019) 'Evaluated with ResNet-50
*Results reported as Average Precision
19


Reviews:
[{"review_id": "HygvQy4otr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "After Discussion Period:\n\nI stick to my original score. My issues are largely resolved.\n\n----\nThe submission is using adaptive computation graphs for domain adaptation. Multi-flow network is the main architectural element proposed in the submission. And, it is composed of parallel blocks of computations which aggregated using weighted summation with learnable weights. The domain adaptation is performed by setting different weights for source and target dataset. The adaptive weights and network parameters are all learned jointly by minimizing the combination of classification loss and domain difference loss.\n\nAlthough the idea of adaptive computation is not novel and has been explored, their application to the domain adaptation problem is novel to the best of my knowledge. Moreover, the proposed method is sensible and technically sound.\n\nThe submission talks about different amount of computation needed per domain as an intuition behind the method. This is sensible and intuitive; however, it has not been experimented. The paper uses the same amount of layers for all domains making the amount of computation exactly same. It would be interesting to see the performance when different paths actually lead different computations. For example, parallel blocks can have different number of layers etc.\n \nThe submission only provides result for RPT and DANN. These are clearly not state-of-the-art domain adaptation methods. Proposed method does not necessarily need to have state-of-the-art adaptation results to be accepted, but not reporting what state-of-the-art performance is makes the experimental results incomplete.\n\nFigure 4 suggests that there is no real parameter sharing at the end of the training. And, all domains have different computations. Authors should try to explain this behaviour since it is quite counter-intuitive. \n\nIn summary, proposed method is somewhat novel, interesting and seems to be working well. Improved discussion on the experimental study is definitely needed.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1lBZPAc_B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed to address the information asymmetry between domains in unsupervised domain adaptation. Innovatively, they resort to a multiflow network where each domain adaptatively selects its own pipeline. I quite appreciate the idea itself, while there are many essential issues to be addressed first. \n\nPros:\n-\tThe way tackling the information asymmetry, or untie weights, between domains is novel and interesting. \n-\tThe proposed network/framework can be easily extended to the multi-task setting, or multi-source/multi-target domain adaptation.\n-\tThe paper is well-written and easy to follow. \n\nCons:\n-\tThe most critical downside of this paper is its insufficient experiments to support the whole idea, where we will detail in the next.\n\nExperimental issues:\n-\tComparison with other state-of-the-art UDA methods (e.g., CDAN) is a must. This paper improves UDA in terms of adaptive parameters sharing, which is completely independent from most of the UDA contributions (including the DANN you compared) which improve the distribution alignment between feature representations. Therefore, it is imperative to compare that line of SOTA methods, otherwise why should we consider adaptative parameters sharing instead of distribution alignment? At best, the proposed multiflow network combined with the SOTA feature alignment method (e.g., CDAN other than DANN) should be considered and expected to beat CDAN itself. \n-\tMany ablation studies or hyperparameter sensitivity analyses are missing. \no\tHow do you determine the number of parallel flows, i.e., K? Is it possible that 3 or 4, more than 2 flows, are better even in the UDA between two domains? \no\tDo you try any other possibilities of grouping a computational unit, and how will different configurations influence the performance? \no\tIs there a possibility that none of the gates in the final layer is activated? Do you need some constraints?\n-\tSince the authors mentioned the potential of the multi-flow network in adaptation between multiple domains, it is necessary to investigate multi-source or multi-target domain adaptation. Only in this case may the significance of different K values be demonstrated. \n-\tThe baseline results in Table 1 are not comparable to some reported papers, and even lower than those reported in other UDA papers. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "B1x4SgJRFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper provides an unsupervised domain adaptation approach\nin the context of deep learning. The motivation is clear, related work\nsufficient and experimental settings and results convincing. \nI have only very minor comments:\n- I would prefer to get the paper additionally linked to a few more\n  transfer learning techniques out of the deep learning domain\n  which is important as well\n- do you really need to call it (multi) flow network .... - a flow network\n  is a well established concept in algorithmics and refers to a graph problem\n  ... to avoid name clashes ...\n- in the references you have provided back links to the pages where the references\n  are used - this is handy but also confusing and a bit unusual - I think it was not part \n  of the standard template\n- please avoid using arxiv references but replace them by reviewed material. In parts\n  I am willing to accept such kind of gray literature provided by well known authors but\n  this should not become a standard habit\n- I am happy to see that the code will be published - I hope this is really done, because\n  from the material it maybe hard to reconstruct the method", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Although some criticism remains for experiments, I suggest to accept this paper.
Decision: Accept (Poster)