Disentangled cyclic reconstruction for domain adaptation
Authors: David Bertoin
Emmanuel Rachelson
Venue:  (2021)
Keywords: Domain adaptation
Disentanglement

Abstract: The domain adaptation problem involves learning a unique classification or regression model capable of performing on both a source and a target domain. Although the labels for the source data are available during training, the labels in the target domain are unknown. An effective way to tackle this problem lies in extracting insightful features invariant to the source and target domains. In this work, we propose splitting the information for each domain into a task-related representation and its complimentary context representation. We propose an original method to disentangle these two representations in the single-domain supervised case. We then adapt this method to the unsupervised domain adaptation problem. In particular, our method allows disentanglement in the target domain, despite the absence of training labels. This enables the isolation of task-specific information from both domains and a projection into a common representation. The task-specific representation allows efficient transfer of knowledge acquired from the source domain to the target domain. We validate the proposed method on several classical domain adaptation benchmarks and illustrate the benefits of disentanglement for domain adaptation.

Under review as a conference paper at ICLR 2021
DISENTANGLED CYCLIC RECONSTRUCTION FOR DO- MAIN ADAPTATION
Anonymous authors Paper under double-blind review
ABSTRACT
The domain adaptation problem involves learning unique classification or regres- sion model capable of performing on both a source and a target domain. Although the labels for the source data are available during training, the labels in the target domain are unknown. An effective way to tackle this problem lies in extracting insightful features invariant the source and target domains. In this work, we propose splitting the information for each domain into a task-related representa- tion and its complimentary context representation. We propose an original method to disentangle these two representations in the single-domain supervised case. We then adapt this method to the unsupervised domain adaptation problem. In partic- ular, method allows disentanglement in the target domain, despite the absence training labels. This enables the isolation of task-specific information from both domains and projection into common representation. The task-specific representation allows efficient transfer of knowledge acquired from the source do- main the target domain. We validate the proposed method several classical domain adaptation benchmarks and illustrate the benefits of disentanglement for domain adaptation.
INTRODUCTION
The wide adoption of Deep Neural Networks in practical supervised learning applications is hin- dered by their sensitivity to the training data distribution. This problem, known as domain shift, can drastically weaken, in real-life operating conditions, the performance of model that seemed perfectly efficient in simulation. Learning a model with the goal of making it robust to a specific domain shift is called domain adaptation (DA). Often, the data available to achieve DA consist of labeled training set from a source domain and an unlabeled sample set from a target domain. This yields the problem of unsupervised domain adaptation (UDA).
In this work, we take an information disentanglement perspective on UDA. We argue that a key to efficient UDA lies in separating the necessary information complete the network's 's task (classifica- tion or regression), from a task-orthogonal information which we call context or style. Disentangle- ment in the target domain seems however a difficult endeavor since the available data is unlabeled. Our contribution is two-fold. We propose formal definition of the disentanglement problem for UDA which, to the best of our knowledge, is new. Then we design a new learning method, called DiCyR (Disentangled Cyclic Reconstruction), which relies on cyclic reconstruction of inputs in or- der to achieve efficient disentanglement, including in the target domain. We derive DiCyR both in the supervised learning and in the UDA cases.
This paper is organized as follows. Section 2 presents the required background on supervised learn- ing and UDA, and proposes definition of disentanglement for UDA. Section reviews recent work in the literature that allow for a critical look at our contribution and put in perspective. Section introduces DiCyR, first for the dingle-domain supervised learning case and then for the UDA prob- lem. Finally, Section empirically evaluates DiCyR against state-of-te tafe-of-theaatt art methods and discusses its strengths, weaknesses and variants. Section summarizes and concludes this paper.

---

Under review as a conference paper at ICLR 2021
2 PROBLEM DEFINITION
In this section, we introduce the notations and background upon which we build the contributions of Section 4 Let A' be imput space of descriptors and J an output space of labels. A supervised learning problem is defined by a distribution (a, y) over elements X x y. In what follows, Ps will be called the source distribution. One wishes to estimate a mapping f that minimizes a loss function of the form B(2,))>p. [((f(r),)) The optimal estimator is denoted ʃ and one often writes the distribution F(y(z) as y f(x) + 7,, where v captures the deviations between y and fz)). Hence, one tries to learn f. In practice, the loss can only be approximated using finite set of samples {(x,, )>] all independently drawn from ps and is parametrie function (such deep neural network) of the form y ())
Domain adaptation (DA) consists in considering target distribution Pt over X' x ) that differs from Pa. and the transfer of knowledge from learning in the source domain (p,) the target domain (pt). Specifically, unsupervised DA exploits the knowledge of labelled training set {( "²) sam- pled according to Ps. and an unlabelled data set ) sampled according to pt. For instance, the source domain data could be set of labelled photographs of faces, and the target domain data, set of unlabelled face photographs, taken with different camera under different exposure conditions.
The problem consists in minimizing the target loss (x.,))>p. ((f(z))))
We suppose that a necessary condition to benefit from the knowledge available in the source domain and transfer the target domain is the existence of common information manifold between do- mains, where an input's projection is sufficient to predict the labels. We call this useful information mask-specific or task-related. The complimentary information should be called task-orthogonal: is composed of information that is present in the input but is not relevant to the task at hand. For the sake of naming simplicity, we will call this information style. However we insist that this should not be confused with the classical notion of style.
Let II, : X + and II, : + denote two projection operators, where T and denote respectively the latent task-related information space and the latent style-related information space. Let II be the joint projection II(x)) = (II,(z) (z), II (x)).. Conversely, we shall note : T x S + X reconstruction operator. And finally,
T y' will denote the labeling operator which only uses information from T. We consider that the information of the elements of X' is correctly disentangled by πI = (I,, IT) if one find IT and such that:
C1: c a II, minimizes the loss (and thus fits on the appropriate domain),
C2: IT o II fits the identity operator idx,
C3: With X, T, the random variables in X. S, the mutual information T(T,S)X)
C4: There is no function g
X such that II, idx.
Condition Cl imposes that the projection into T retains enough information correctly label sam- ples. Condition C2 imposes that all the information necessary for the reconstruction is preserved by the separation performed by II. Condition C3 states that no information present in both T and S. Condition C4 impose that the information contained in T alone is insufficient to reconstruct an input, and thus the information of sary, since the combination of C1 and C3 already guarantees that S cannot contain the task-related information. Overall, solving this disentanglement problem for DA implics finding a quadruplet (II,, MI, 11, c) that meets the conditions above. In particular, note that conditions C3 and C4 open perspective to a formulation of disentanglement in the general case.
necessary. Note that the symmetrical condition is unneces-
RELATED WORK
task-related information and the comain--pecific. oath-orth,gonal, style information is a desirable property to have for DA. In the next paragraphs, we cover important work in representation disentanglement, domain adaptation and their interplay.
Disentanglement between the
Before deep learning became popular, Tenenbaum & Freeman (2000) presented a method using bi-linear models able separate style from content. More recently, methods based generative
2

---

Under review as a conference paper at ICLR 2021
models have demonstrated the ability to disentangle factors of variations from elements of a single domain (Rifai et al. 2012 Mathieu et al., 2006 Chen et al. 2016 Higgins et al. 2017 Sanchez al. (0118) use pairs of images with the same labels from different domains to separate representations into a shared information common to information. We note that these approaches do not explicitly
et al. 2019). In cross- doms-n setting, Gonzalez
both domains and a domain-exclusinn aim at respecting all conditions listed in Section _ Additionally, most require labeled datasets (and in some cases even paired datasets) and thus do not address the wnsupervised DA problem.
One approach to UDA consists in aligning the source and target distributions statistics, a topic closely related to batch normalization (loffe Szegedy 2015). Sunn al.] (2017) minimize the dis- tance between the covariance matrices of the features extracted from the source and target domains. Assuming the domain-specifi information is contained inside the batch normalization layers, [Li et al. (2017) align the batch statistics by adopting a specific normalization for each domain. Cari- ucci et al. (2017) aim to align source and target feature distributions to reference one and introduce domain alignment layers to automatically learn the degree of feature alignment needed at different levels of the network. Similarly, Roy et al. (2019) replace batch normalization layers with domain alignment layers implementing a so-called feature whitening. A major asset of these methods is the possibility to be used jointly with other DA methods (including the one we propose in Section 4.. These methods jointly learn a common representation for elements from both domains. Con- versely, veingly.Liang et al. (2020) freeze the representations learned in the source domain before training a target-specific encoder align the representations of the target elements by maximizing the mutual information between intermediate feature representations and outputs of the classifier.
Ensemble methods have also been applied UDA (Laine & Aila 2017: Tarvainen & Valpola 2017). French et al. (2018) combine stochastic data augmentation with self-ensembling to minimize the prediction differences between a student and a teacher network in the target domain.
Another approach involves learning domain- invariant features, that do not allow to discriminate whether sample belongs to the source or target domain, while still permitting accurate labeling in the source domain. This approach relies on the assumption that such features allow efficient labeling in the target domain. Ghifary et al.] (2016) build two-headed network sharing common layers; one head performs classification in the source domain, while the second is decoder that performs reconstruction for target domain elements. Ganiin et al. (2016) propose the DANN method and introduce Gradient Reversal Layers to connect a domain discriminator and feature extractor. These layers invert the gradient sign during back proodk-pppgtttonn so that the feature extractor is trained to fool the domain discriminator. [Shen et al. (2018) modify DANN and replace the domain discriminator by a network that approximates the Wasserstein distance between domains. Tzeng et al. (2017) optimize, in an adversarial setting, a generator and a discriminator with inverted label loss.
Other methods focus on explicitly disentangling an information shared between domains (analogous to the Nomain-invarian features above) from domain-specifie information. Inspired by Chen al. 2016). Liu et al.] (2018b) isolate a latent factor, representing the domain information, from the rest of an encoding, by maximizing the mutual information between generated images and this latent factor. Some domain information may still be present in the remaining part of the encoding and thus may not comply with conditions C3 and C4. Liu et al. (2018a) combine an encoder, an image generator, domain discriminator, and fake images discriminator to produce cross-domain images. The encoder is trained jointly with the domain discriminator to produce
representations. Li et al. (020) disentangle a latent representation into a global code and local code. The global code captures category information via an encoder with a prior, and the local code is transferable across domains, which captures the style-related information via implicit decoder. [Bousmalis et al. (2016) also produce domain-invariant features by training a shared encoder to fool domain discriminator. They train two domain- private encoders with a difference loss that encourages orthogonality between the shared and the private representations (similarly to condition C3). Cao et al. (2018); :: Cai et al.) (2019); Peng et al. (2019) combine a domain discriminator with an adversarial classifier to separate the information shared between domains from the domain-specific information. All these methods build shared representation that prevents discriminating between source and target domains, while retaining enough information to correctly label samples from the source domain. However, because they rely on an adversarial classifier that requires labeled data, they do not guarantee that the complimentary, domain-specifie information for samples in the target domain does not contain information that overlaps with the shared representation. In other words,
3

---

Under review as a conference paper at ICLR 2021
they only enforce C3 in the source domain. They rely the assumption that the disentanglement will still hold when applied on target domain elements, which might not be true.
Another identified weakness in methods that achieve a domain-invarian feature space is that their representations might not allow for accurate labeling in the target domain. Indeed, feature alignment does not necessarily imply correct mapping between domains. To illustrate this point, consider binary classification problem (classes Cj and (2) and two domains (d, and d2). Let (c), denote samples of class ej in dj. It is possible to construct an encoding that projects (ci,di) and (c2, d2) to the same feature values. The same holds for (c1,d2) and (c2, di) for different feature values. This encoding allows discriminating between classes d1. It also fools a domain discriminator since it does not allow predicting the original domain of a projected element. However, applying the classification function learned on d1 to the projected d2 elements leads to catastrophic predictions. Transforming a sample from one domain to the other, while retaining its label information can be accomplished by image-to-image translation methods. |Hoffman et al. (2018) extend CycleGAN's cycle consistency (Zhu al. 2017) with semantic consistency to translate from source to target domains. The translated images from the source domain to the target domain are then used to train classifier on the target domain using the source labels. Similarly, |usso al. (2018) train two conditional GANs (Mirza & Osindero 2014) to learn bi-directional image mappings constrained by class consistency loss and transformed target samples. By relaxing CycleGAN's cycle consistency constraint and integrating the discriminator in the training phase, (oosseii--Asi et al. 2019 address the DA problem in the spe- cific setting where the number of target samples is limited. Takahashi et al. (2020) use CycleGAN to generate cross-domain pseudo-pain pairs and train two domann-pecific encoders to align features ex- tracted from each pseudo-pair in the feature space. A major asset of the method is to address the bass-unbalanced UDA problem by oversampling with the learned data augmentation. (Yang et al.] 2019) use separate encoders to produce domain-invarian and domain-specifff features in both do- mains. They jointly train these encoders with two generators to produce cross-domain elements able to fool domain-specinf
source domain classifier to produce pseudo-labels
source-like
discriminators. Using cyclic loss features, they force the information
contained in the representation to be preserved during the generation of cross-domain elements. However, the eyclic loss features does not prevent the information sharing between features pressed in C3. More importantly it does not prevent the domain-specific features to be constant. A major drawback of these methods lies in the instability during training that might be caused by min-max optimization problem induced by the adversarial training of generators and discriminators.
In the next section, we introduce a method that does not rely on a domain discriminator and an adversarial label predictor. but directly minimizes the information sharing between representations. This allows to guarantee that there is no information redundancy between the task-related and the tath-orth style information in both the source and the target domains. Along the way, it pro- vides an efficient mechanism to disentangle the task-related information from the style information in the single domain case. Our method combines information disentanglement, intra-domain and cross-domain cyclic consistency to enforce more principled mapping between each domain.
DISENTANGLEMENT WITH GRADIENT REVERSAL LAYERS AND CYCLIC RECONSTRUCTION
First, we propose original method to disentangle the task-related information from the style in- formation for a single domain in a supervised learning setting. In a second step, we propose an adaptation of this method to learn these disentangled representations in both domains for UDA. This disentanglement allows, in turn, to efficiently predict labels in the target domain.
4.1 TASK-STYLE DISENTANGLEMENT IN THE SUPERVISED CASE
Our approach consists in estimating jointly II. and c as a deep feed-forward neural network. We shall note 9n, 8n. and 0g the parameters of the respective sub-parts of the network. II o II takes the form of an auto-encodes while o c is a task-related (classification or regression) network. Figure Ta summarizss the global architecture which we detail in the following paragraphs.
4

---

Under review as a conference paper at ICLR 2021
GRL
GRLL
11
II
T
GRI
SFat
(a) Supervised learning
(b) Unsupervised domain adaptation
Figure 1: Network architectures
Conditions Cl and C2 are expressed through the definition of task-specific loss Ltask (e.g. cross- entropy for classification, L2 loss for regression) and reconstruction loss Creco- Therefore, the update of On should follow on (Ltask + Lreco), the update of On relies on Von and that of 0g uses Ctask
In order to achieve condition C3, we exploit Gradient Reversal Layers (Ganin et al. 2016 GRL). We train two side networks r,
and T, T S whose purpose to attempt to predict T given S, and S given T respectively. For given let write (T. = (r) We train r- and Ta to minimize the losses fr, = ||| - Fll and L.,, ||o - क 12 Let Linso Cr- L,, denote the combination of these losses. We connect these two sub-networks to the whole architecture using GRLs. GRLs behave as the identity function during the forward pass and invert the gradient sign during the backward pass, hence pushing the parameters maximize the output loss. During training, this architecture constrains to produce features T and with the least information shared between them. Consequently, the update of On follows +Ven Linfo-
[2x)), 7 = (a)) and
This constraint efficiently avoids information redundancy between T and S. However, does not avoid all the information being pushed into T. Preventing this undesirable behavior is the purpose of condition C4. To that end, we use a cyclic reconstruction scheme. Consider two elements x and x' from X. and their associated (r. reconstruction of 7 that uses the style o' of x', A correct allotment of the information between T and S requires that the task and style information be preserved in (7,Ø)
11(x) and (7',0') 11(x'). Let ā II(T, o') be the
[()).. So, we wish to have f as close as possible to or, alternatively, to have c(f) as close as possible to c(T). Similarly, we wish to have as close as possible to o'. To enforce C4 and avoid the degenerate case where encoder predicts constant style for all samples, we force ā to lie sufficiently far from to avoid style confusion. We achieve this with a triplet loss (Schroff et al. 2015) using as the anchor, and as, respectively, the positive and negative inputs, and margin m. Thus C4 results in minimizing the cyclic reconstruction loss Legetsc = || Tll max o' lā o 0|l2 + m, 0}.
Overall, the gradient-based update procedure of the network parameters boils down to:
Oπ + 0T - a Von (Lrask + Creco Linfo + Leyclie),
0n < pn Ven (Creco + Leyelte),
0. - 0. Vo, Leask: Or, - Or. - a aV 0., Lre
0,, + 0,, - a Vo., fry,
We call this method DiCyR for Disentangled Cyclic Reconstruction.
4.2 TASK-STYLE DISENTANGLEMENT IN THE UNSUPERVISED DOMAIN ADAPTATION CASE
We propose variation of DiCyR for UDA, where replace the decoder 1l by two domain-specine decoders, 11, and 11.- We shall compensate for the lack of labeled data the target domain by computing dooss-domain cyclic reconstructions.
Let (xg, y.) be a sample from the source domain and 2, be a sample from the target domain. Let us denote (T,, σ.) II(,,) and (7r, 0,) II(x,), the corresponding projections in the latent task and style-related information spaces. Then one can define, as in the previous section, Lrask as the task- specific loss the source domain, and Lyeco, and freco, as the reconstruction losses in the source
5

---

Under review as a conference paper at ICLR 2021
Style Tuk
Style
Tack
5
5
8
8
18
18
8
2
Figure 2: Swapping styles on SVHN and 3D Shapes
and target domains respectively. As previously, we constrain the task-related representation and the style representation not to share information using two networks and Ta. connected to the main architecture by GRL layers (Figure lb) allowing the definition of the L,,, C,, and Lin fo losses. Lastly, we exploit cyclic reconstructions in both domains to correctly disentangle the information and hence define the same Leyelie loss as above.
This disentanglement in the target domain separates the global information in two but does not guarantee that what is being pushed into t is really the task-related information. This can only be enforced by cross-domain knowledge (since no correct labels are available in the target domain). Thus, finally, we would like to allow projections from one domain into the other while retaining the task-related information, hence allowing domain adaption. Using the notations above, we construct Its = I1, a.), the reconstruction of x; 's task-related information, in the style of zς- This creates an artificial sample in the target domain, whose label is ye- Then, with (Tts: dts) [I(,)s), one wishes to have Tzs match closely т, (or, alternatively, c(Tis) match closely y.) in order to prevent the loss of task information during the cross-domain projection and thus constrain the task representa- tions to be domain-invariant. Symmetrically, one can construct the artificial sample ss = [,, o.) and enforce that Fur closely matches тІ. Note that the label of at unknown and yet is still pos- sible to enforce the disentanglement by cyclic reconstruction.
Overall, these terms boil down to cross-domain cyclic reconstruction loss for UDA Ldomain-cycli = |T -cll22 + Th Teell2-
Finally, the network parameters are updated according to:
In 8T Von Ven (Frask + Lreco, + freco
Linfo + -eeyotio + Ldomain cyelio) pn. t- OT. a aVon, (Ereco, + Leeatice)
0 n1 -- 1T. a aVon, (Ereco, + Layetic);
0. 0. Vo, Lrusk:
0r, + Or, Vo., frr,
Pr. t Gr. - -a aooo Fr..
EXPERIMENTAL RESULTS AND DISCUSSION
We first evaluate DiCyR's ability disentangle the task-related information from the style informa- tion in the supervised context. Then we demonstrate DiCyR 's efficiency on UDA
5.1 SUPERVISED DISENTANGLEMENT
We evaluate the disentanglement performance of DiCyR by following the protocol introduced by Mathieu et al. (2016). Since we do not use generative models, we only focus on their two first items: swapping and retrieval. We evaluate DiCyR on the SVHN (Netzer al. 2011), and 3D Shapes (Burgess & Kim 2018) disentanglement benchmarks. The task is predicting the central digit in the image for the SVHN dataset, and the shape of the object in the scene for the 3D Shapes dataset.
Swapping involves swapping styles between samples and visually assessing the realism of the gen- erated image. It combines the task-related information 7; of sample x;; with the style oj of another sample zy. We use the decoder to produce an output æ.y. Figure shows randomly generated outputs on the two datasets. DiCyR produces visually realistic artificial images with the desired styles.
"Code and pre-trained networks available

---

Under review as a conference paper at ICLR 2021
Quayy
Neighforr
Query
Neighouy
Query
Neidhboss
Quary
Nughlors
54
5
0
6
3
7 8
3
33
131
6
62
16
60
4
4
35
24
22
35
20
(a) Query on the taskerelated representation
(b) Query on the style representation
Figure 3: Nearest neighbors according to each representation
floor hue
wall hue
object hue
Method
SVHN
3D Shape
scale
orientation
Full features
0,98
1
0,94
0.94
0.89
0.6
0.5
Task related features
0.98
0.11
0.12
0.13
0.15
0.10
Style features only
0.17
0.26
0.89
0.95
0.88
0.59
0.42
Random guess
0.10
0.25
0.10
0.10
0.10
0.125
0.067
Table 1: Accuracies of a classifier trained predict factors of style variation on 3D shapes
Retrieval concerns finding, in the dataset, the nearest neighbors in the embedding space for an image query. We carry out this search for nearest neighbors using the Euclidean distance on both the task-related and the style representations. A good indicator of the effectiveness of the information disentanglement would be to observe neighbors with the same labels as the query when computing distances on the task-related information space, and neighbors with similar style when using the style information. Figure 3 demonstrate that the neighbors found when using the task-related information are samples with the same label as the query's 's label and that the neighbors found using the style representation share many characteristics with the query but not necessarily the same labels.
We ran a quantitative evaluation of disentanglement by training neural network classifier with a single hidden layer of 32 units to predict labels, using either the task-related information alone, or the style information alone. If the information is correctly separated, we expect the classifier trained with task-related information only to get similar performance to a classifier trained with full information. Conversely, the classifier trained with the style information only should reach similar performance to random guess (10% accuracy SVHN 25% on 3D Shapes). Table [ the obtained testing accuracies. It appears that the task-related representation contains enough informa- tion correctly predict labels. We also observe that full disentanglement is closely but not perfectly achieved as the classifier trained only with style information behaves slightly better that random choice. To quantify how much style information is being unduly encoded in the task-related repre- sentation, we ran a similar experiment to predict the five other style variation factors in 3D Shapes (floor hue, wall hue, object hue, scale and orientation). The trained classifier reaches accuracies that are very close to a random guess, thus validating the disentanglement quality.
5.2 UNSUPERVISED DOMAIN ADAPTATION PROBLEM
We evaluate DiCyR by performing domain adaptation between the MNIST LeCun et al. 1998 SVHN, and USPS (Hull, 1994) datasets, and between the Syn-Signs Ganin & Lempitsky 2015) and the GTSRB (Stallkamp al. 2011) datasets. Following common practice in the literature, trained our network on five different settings: MNIST-USPP USPS MNIST, We measure the classification performance in the target
BEENNMNIITT
MNIST-SVHN, and Syn-Signs--GTSAE domain and compare it with state-of-the-art methods (Table 2}. We also compare with a base- line classifier that is only trained on the source domain data. Values reported in Table 2 are quoted from their original p Our method, without extensive hyperparameter tuning, appears be on par with the best state-of-the-ar methods. DiCyR also outperforms others disentangle-
CComparioons might be inexact due to reproducibility concerns (Pineau et al. 2020) and these figures mostly indicate which are the top competing methods.
7

---

Under review as a conference paper at ICLR 2021
ment and image-to-imen methods. Specifically, DiCyR is only slightly outmatched by DWT and SEDA on the MNIST+USSPS and by SEDA and SHOT in the SVHN-MNISS -MNIST benchmarks. The MNIST~SSVHN case is a particularly challenging benchmark since MNIST images greyscale and the adaptation to SVHN requires adapting to color images. SEDA makes extensive use of data augmentation to tackle this challenge and is thus the only method displaying convincing results. This hints to a possible enhancement of DiCyR in order to improve its performance. Finally, by intro- ducing a new variation on batch normalization, DWT's contribution orthogonal ours and both could be combined. We emphasize that beyond these enhancements, major advantage of DiCyR lies in the ability to disentangle the information in the target domain without direct supervision.
DiCyR uses GRLs to ensure that no information is shared between T and S. Similarly to most methods in Table 2 GRLs induce an adversarial optimization problem which is known to yield instability and variance in the resolution performance. In our case, this induces several distinct modes in the distribution of accuracies. It interesting to note that the majority mode (that of the median) SVHN-MNIST matches the best known performance. For this reason, we report both the mean and the median on this specific experiment. One might object that condition C3 was expressed in terms of mutual information. Thus, DiCyR only indirectly implements this condition using GRLs. An alternative could be to use an estimator of the mutual information, such as proposed by Belghazi et al. 2018), directly minimize it (and thus avoid the adversarial setting altogether). Such an approach was explored in the work of Sanchez et al. (2019) to disentangle representations between pairs of images, and would constitute promising perspective of research for DA.
Source Target
MNIST
USPS
SVHN MNIST
MNIST
Syn-Signs
Method
USPS
MNIST
SVHN
GTSRB
Baseline
78.1
58.0
60.2
20.0
79.0
DANN (Ganin et al. 2016)
85.1
73.0
73.9
35.7
88.6
ADDA (Tzeng et al. 2017)
89.4
90.1
76.0
DSN (Bousmalis et al.
2016)
91.3
82.7
93.1
DRCN (Ghitary et al.
2016)
91.8
73.7
82.0
40.1
DiDA (Cao et al. 2018)
92.5
83.6
SBADA-GAN (Russo et al.
2018)
97.6
95.0
76.1
61.1
96.7
CyCADA (Hoffman et al. 2018)
95.6
96.5
90.4
DWT (Roy et [ll.]2001)
99.1
98.8
97.7
28.9
SEDA (Tench et al. (018)
98.2
99.5
99.3
97.0
99.3
ACAL Hosseini-Aso et al.
2019)
98.3
97.2
96.5
60.8
SHOT (Liang et al.) 2020)
98.4
98.0
98.9
DiCyR (ours)
98.4
98.3
98.5
23.8
97.4
median accuracy reported (average accuracy: 95.7 full results distrihution are reported in Appendix
Table 2: Target domain accuracy, reported as percentages
As in Section 5.1 we evaluate qualitatively the effectiveness of disentanglement, especially in the target domain, and produce visualizations of cross-domain style and task swapping. Here, we com- bine one domain's 's task information with the other domain's styles to reconstruct the images of Figures 4a
4b 4d and 4e The most important finding is that the style information was correctly disentangied from the task-related information in the target domain without the use of any label. Specifically, the rows in these figures show that the class information is preserved when new style is applicd, while the columns illustrate the efficient style transfer allowed by disentanglement.
A desirable property of the task-related encoding is its domain invariance. To evaluate this aspect, we built a tSNE representation (Hinton & Roweis 2003) of the task-related features, in order to verify their alignment between domains. Figures 4c and demonstrate this property.
The previous experiments illustrated the use of DiCyR in the context of image classification. The method is, however, quite generic and can be applied in many more contexts. Figure 5 the improvement due to applying DiCyR for domain adaptation between the GTAS (Richter et al. 2016) and the Cityscapes (Cordts et al. 2016) segmentation problems (detailed results in Appendix
8

---

Under review as a conference paper at ICLR 2021
2
7
6
8
8
16
8
0
8
8
8
8
7
5
5
3
3
3
3
O
4
4
4
4
9 8
6
0
6
6
6
(a) SVHN source class with MNIST target style
(b) MNIST target class with I-SNE task-related features, SVHN source style
SVHN (blue) - MNIST (red)
Styk Toce
70
(d) Syn-Signs source class with GTSRB target style
(e) GTSRB target class (1) I-SNE task-related features,
with Syn-Signs source style Syn- (blue) GTSRB (red)
Figure 4: Cross-domain swapping and feature alignment visualization
Test image
(b) Source prediction
DiCyR
(d) Ground truth
Figure 5: GTA5 to Cityscapes segmentation
Finally, directly computing the distances the task-related features in Ldomain eyyolic often leads to unstable results. As hinted in Section 4 using instead a task oriented loss Ctomain_eyclle [[c(T.) +lcc2 c(Tar)]l2 stabilizes training and improves the target domain accuracy. Train- ing with cross- domos- projections from the source domain and the corresponding labels improves its generalization the target domain and forces the encoder produce task-related features com- mon to both domains. To illustrate this property, consider the following example. In one domain, the digit "7" is written with middle bar, while in the other it has none. This domain-specific middle bar feature should not be expressed in T; it should be considered as trsk-orthogonal style feature. Thus using c's predictions within the domain cyclic loss, instead of distances T, prevents the encoder from representing the domain-specific features in T and encourages their embedding in S.
CONCLUSION
In this work, we introduced a new disentanglement method, called DiCyR, to separate task-related and task-orthogonal style information into different representations in the context of unsupervised domain adaptation. This method also provides a simple and efficient way to obtain disentangled representations for supervised learning problems. main features are its overall simplicity, the use of intra-domain and cross-domain cyclic reconstruction, and information separation through Gradi- ent Reversal Layers. The design of this method stems from a formal definition of disentanglement for domain adaptation which, the best of our knowledge, is new. The empirical evaluation shows that DiCyR performs as well as state-of-the-aa methods, while offering the additional benefit of disentanglement, including in the target domains where label information is available.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In 35th International Con- ference on Machine Learning, pp. 531-540, 2018.
Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan.
Domain separation networks. In Advances in neural information processing systems, pp. 343-351, 2016.
Chris Burgess and Hyunjik Kim. 3d shapes dataset. http://giiintccommmeeepmi///asss 2018.
Ruichu Cai, Zijian Li, Pengfei Wei, Jie Qiao, Kun Zhang, and Zhifeng Hao. Learning disentangled semantic representation for domain adaptation. In International Joint Conference on Artificial Intelligence, pp. 2060, 2019.
Jinming Cao, Oren Katzir, Peng Jiang, Dani Lischinski, Danny Cohen-Or, Changhe Tu, and Yangyan Li. Dida: Disentangled synthesis for domain adaptation. arXiv, pp. arXiv-1805, 2018.
Fabio Maria Cariucci, Lorenzo Porzi, Barbara Caputo, Elisa Ricci, and Samuel Rota Bulo. Autodial: Automatic domain alignment layers. In 2017 IEEE International Conference Computer Vision (ICCV), pp. 5077-5085. IEEE, 2017.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp. 2172-2180, 2016.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Geoff French, Michal Mackiewicz, and Mark Fisher. Self-ensembling for visual domain adaptation. In International Conference on Learning Representations, 2018.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain adversaial training of neural net- works. The Journal of Machine Learning Research, 17(1):20962000 2030, 2016.
Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, David Balduzzi, and Wen Li. Deep ctwpaarucctiioccassfcaaatttt networks for unsupervised domain adaptation. In European Con- ference on Computer Vision, pp. 597-613, 2016.
Abel Gonzalez-Garcia, Joost Van De Weijer, and Yoshua Bengio. Image-to-image translation for cross-domain disentanglement. In Advances neural information processing systems. pp. 1287- 1298, 2018
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. Beta-VAE: Learning basic visual concepts with constrained variational framework. In 5th International Conference on Learning Representationss 2017.
Geoffrey E Hinton and Sam T Roweis. Stochastic neighbor embedding. In Advances in neural information processing systems, pp. 857-864, 2003.
Judy Hoffman, Dequan Wang, Fisher Yu, and Trevor Darrell. Fens in the wild: Pixel-level adversar- ial and constraint- adaptation. arXiv preprint arXiv: wel..066222444 2016.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International Conference Machine Learning, pp. 1989-1998, 2018.
10

---

Under review as a conference paper at ICLR 2021
Ehsan Hosseini-Asi, Yingbo Zhou, Caiming Xiong, and Richard Socher. Augmented cyclic ad- versarial learning for low resource domain adaptation. In International Conference on Learning Representations, 2019.
Jonathan J. Hull. A database for handwritten text recognition research. IEEE Transactions
Pattern Analysis and Machine Intelligence, 16(5):550-554, 1994.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 448-456, 2015.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In Sth Interna- lional Conference on Learning Representations, 2017.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2244 1998.
Haoliang Li, Renjic Wan, Shiqi Wang, and Alex Kot. Unsupervised domain adaptation in the wild via disentangling representation learning. International Journal of Computer Vision, pp. 1-17, 2020.
Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normaliza- tion for practical domain adaptation. In 5th International Conference Learning Representa- fions, 2017.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. arXiv preprint arXiv:2002.08546, 2020.
Alexander H Liu, Yen-Cheng Liu, Yu-Yin Yu-Ying Yeh, and Yu-Chiang Frank Wang. A unified feature disentangler for multi-doman image translation and manipulation. In Advances in neural infor- mation processing systems, pp. 2590-2599, 2018a.
Yen- Ven-Cheng Liu, Yu- Ying Yeh, Tzu-Chien Fu, Sheng- De Wang. Wei-Chen Chiu, and Yu-Chiang Frank Wang. Detach and adapt: Learning cross-domain disentangled deep representation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8867- 8876, 2018b.
Michael F Mathieu, Junbo Jake Zhao, Junbo Zhao, Aditya Ramesh, Pablo Sprechmann, and Yann LeCun. Disentangling factors of variation in deep representation using adversarial training. In Advances in neural information processing systems, pp. 5040-5048, 2016.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint /4 2014.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
Xingchao Peng, Zijun Huang, Ximeng Sun, and Kate Saenko. Domain agnostic learning with dis- entangled representations. In International Conference on Machine Learning, pp. 5102-5112, 2019.
Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence Alché Buc, Emily Fox, and Hugo Larochelle. Improving reproducibility in ma- chine learning research (a report from the neurips 2019 reproducibility program). arXiv preprint arXiv:2003.12206. 2020.
Stephan R. Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), European Conference on Computer Vision (ECCV), volume 9906 of LNCS, pp. 102-118. Springer International Publishing, 2016.
11

---

Under review as a conference paper at ICLR 2021
Salah Rifai, Yoshua Bengio, Aaron Courville, Pascal Vincent, and Mehdi Mirza. Disentangling fac- tors of variation for facial expression recognition. European Conference on Computer Vision, pp. 808- 822, 2012.
Eduardo Romera, José M Alvarez, Luis M Bergasa, and Roberto Arroyo. Erfnet: Efficient resid- ual factorized convnet for real-time semantic segmentation. IEEE Transactions on Intelligent Transportation Systems, 19(1):263-27,, 2017.
Subhankar Roy, Aliaksandr Siarohin, Enver Sangineto, Samuel Rota Bulo, Nicu Sebe, and Elisa Ricci. Unsupervised domain adaptation using feature-whitening and consensus loss. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 9471-9480, 2019.
Paolo Russo, Fabio M Carlucei, Tatiana Tommasi. and Barbara Caputo. From source target and back: symmetric bi-directional adaptive GAN, In IEEE Conference on Computer Vision and Puttern Recognition, pp. 8099-8108 8108, 2018.
Eduardo Hugo Sanchez, Mathieu Serrurier, and Mathias Ortner. Learning disentangled representa- tions via mutual information estimation. arXiv preprint arXiv: 2019.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In IEEE conference on computer vision and pattern recognition, pp- 815-823, 2015.
Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. In AAAI, 2018.
Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: multi-class classification competition. In The 2011 international joint conference on neural networks, pp. 1453-1460. IEEE, 2011.
Baochen Sun, Jiashi Feng, and Kate Saenko. Correlation alignment for unsupervised domain adap- tation. In Domain Adaptation in Computer Vision Applications, pp. 153-171. Springer, 2017.
Ryuhei Takahashi, Atsushi Hashimoto, Motoharu Sonogashira, and Masaaki liyama. Partially- shared variational auto-encoders for unsupervised domain adaptation with target shift. arXiv preprint arXiv:2001.0788 2020.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consis- tency targets improve semi semi-supervised deep learning results. In Advances in neural information processing systems, pp. 1195- 1204, 2017.
Joshua B Tenenbaum and William T Freeman. Separating style and content with bilinear models. Neural computation, 12(6): 1247-11242228 2000.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In IEEE conference on computer vision and pattern recognition, pp. 7167-7176, 2017.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint arXiv:1607.08022, 2016.
Junlin Yang. Nicha C Dvorec, Fan Zhang, Julius Chapiro, MingDe Lin, and James Duncan. Unsupervised domain adaptation via disentangled representations: Application to eross-modality liver segmentation. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pp. 255-263. Springer, 2019.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired taage-to-image transla- tion using cycle-consistent adversarial networks. In IEEE international conference on computer vision, pp. 2223-2232, 2017.
12

---

Under review as a conference paper at ICLR 2021
A
CROSS-DOMAIN DISENTANGLEMENT VISUALIZATIONS
We report extra cross-domain visualizations similar to those of Section 5.2 Figures 6 and
Style
4
6
2
6
4
Style
9
8
Thak
8
6
8
8
8
8
9
9
99999
9
9
3
3
3
3
3
S
SSSSS
9
9
9
9
9
9
3
3
3
3
3
3
S
6
5
S
5
5
(a) Combination of source class with target style
(b) Combination of target class with source style
Figure 6: Cross-domain swapping, MNIST
Style
3
3
4
4
7
Tusk
Task
2
2
2
8
8
8
8
8
8
3
3
3
3
3
9
9
9
9
9
9
0
0
0
0
2
2
2
2
2
S
5
/
/
(a) Combination of source class
(b) Combination of target class with source style
with target style
Figure 7: Cross-domain swapping, MNIST-USPS
B NETWORK ARCHITECTURE AND TRAINING HYPER-PARAMETER
The paragraphs below detail the network architctures used in the experiments of Section 5 It should be noted that neither these architectures, nor the associated hyper-parameters have been extensively and finely tuned their respective tasks, as the goal of this contribution was to provide generic, robust method. Thus it is likely that performance gains can still be obtained on this front.
B. SINGLE DOMAIN SUPERVISED DISENTANGLEMENT EXPERIMENTS
This section describes the network architecture and the hyper-parameters used in the experiments of Section 5.1 The encoder II composed of shared layers (layers 1 to in the table below), followed by the specific taskerelated and style encodings. Those final layers are denoted II, and II, the tables below. For the sake of implementation simplicity, we chose to project samples from the source domain and samples from the target domain into two separate style embeddings (one for each domain). Thus II, is actually duplicated two heads II,,s and II, with the same structure and output space. We used the exact same network architectures for both the 3D shapes and SVHN datasets, the only difference being the dimension of the embeddings T and S. In all experiments, applied coefficient Brewo 5 to Lrecu and Beyelic 0.1 to Leyelic in the global loss. We also use Binoo on Linfoi this coefficient increases linearly from 10 to 10 over the first 10 epochs and remains at 10 afterwards (see Section C for a discussion on this coefficient). Convergence was reached within 50 epochs. We used Adam (Kingma & Ba 2015) as optimizer with a learning rate Ir = 5 10 4 for the first 30 epochs and Ir 3. for the last 20 epochs. The following tables summarize the architectures of all sub-networks.
13

---

Under review as a conference paper at ICLR 2021
Architecture of II, single domain case
Layer
Type
Parameters
Conv2D
filters=32, kemne=55 x 5, stride=1, padding-,, activation-ReLU
Max Pooling
filters=2 2, stride=2
3
Conv2D
filters=64 for 3DShapes or 32 for SVHN
kemnel55 5, stride=1, padding=2, activation=ReLU
4
Max Pooling
filters22 2, stride=2
5
Conv2D
filters=128 for 3DShapes or 64 for SVHN
kerne=33 stride=1. L activation-ReLLU
Dense
nb nb.neeroon==1024, activation=ReLU
II,
Dense
nb neuronss==20 for 3D shapes or 150 for SVHN, activation-ReLU
II.
Dense
nb_neurons=20 for 3D shapes or 150 for SVHN, activation-ReLU
Architecture of II, single domain case
Layer
Type
Parameters
Dense
nb 1024, activation-ReLU
2
Dense
nb ab_neurons=64, activation=ReLUU
3
Conv2D
filters=64, kernel33 x strid==1, padding=1, Retivatio==RelL
4
Upsample
scale_factor=2
5
Conv2D
filters322, kernel=5 x 5, stride=1, padding=2, activation=Rel.U
6
Upsample
scale factor=2
7
Conv2D
filter=3,, kernl=55 5, stride=1, padding=2, activation=SSiimmod
Architecture of c, single domain case
Layer
Type
Parameters
Dropout
p=0.2 for 3DShapes and 0.55 for SVHN
Dense
ab.neurons=nb_label. activation=Soltmax
Architecture of r, and Ya. 3DShapes single domain case
Layer
Type
Parameters
Gradient Reversal Layer
2
Dense
nb jbb_neuron=1100 100, activation=Rel.U nb nb_neurons=20, activation=L.nnar
3
Dense
Architecture of r, and ro. SVHN single domain case
Layer
Type
Parameters
1
Gradient Reversal Layer
2
Dense
nb 1000 activation-ReL.U
3
Dense
nb ob_neurros=100 100, activation=ReLU
4
Dense
nb seurons=150. activation=Liinar
B.2 UNSUPERVISED DOMAIN ADAPTATION EXPERIMENTS
This section describes the network architecture and the hyper-parameters used in the experiments of Section 5.2 The encoder is composed of shared layers (layers 1 to in the tables below), followed by the specific task-related and style encodings. Those final layers are denoted II, and II in the tables below. For the sake of implementation simplicity. we chose project samples from the source domain and samples from the target domain into two separate style embeddings (one for each domain). Thus II, actually duplicated two heads TT.,, and TT,, with the same structure and output space. In all experiments, in the global loss, we applied a coefficient Seyetic = 0.1 to Loyetic and Baomeen cyylic to Ldtomain.ccch cyclies with 3ttomain cyclic increasing linearly from to 10 during the 10 first epochs and remaining 10 afterwards (see Section C for discussion on this coefficient). Convergence was reached within 50 epochs (generally within 30 epochs). We used Adam (Kingma & Ba 2015) as optimizer with learning rate Ir = 5- 10 for the first 30 epochs and Ir 3- 10 5 for the last 20 epochs. The following tables summarize the architectures of all sub-networks.
14

---

Under review as a conference paper at ICLR 2021
Architecture of II, SVHN case
Layer
Type
Parameters
Conv2D
filters=32, kernel=5 5, stride=1, padding=2, activation-Linear
Instance Normalization
Max Pooling
filters=2 x 2, stride=2
Conv2D
filters332 for SVHN-MNIST 64 MNIST-SVHN
kernel=5 5, stride=1, padding=2, activation-L.nnar
Instance Normalization
4
Instance Normalization
Max Pooling
filters=2 x 2, stride=2
Conv2D
filters-32 for SVHN- 128 for MNIST-SVHN
kernl=33 stride=1, padding=l, activation=Linear
Instance Normalization
Dense
nb-neurons=1024, activation=ReLU
Dense
nb mb_neurons=75 for SVHN-MNIS' activation=Rel.U
nb ab_neurons=200 for MNIST SSHN, activation=ReLU
IT,
Dense
nb _neurons=775 for SVHN-MNIST activation=ReLU
nb neurons=200 for MNIST-SVVHN, activation=ReLU
Architecture of II, and II, SVHN++NNIS case
Layer
Type
Parameters
Dense
nb neurons=1024 activation=ReLU
2
Dense
nb_ neunonsss=6, activation-ReLU
3
Conv2D
filters332 for SVHN for MNIST-SVHN
kernel=3 x 3, stride=1, padding=1, padding,, activation-ReLU
4
Upsample
scale factor=2
5
Conv2D
filter=32,, kernel=5 5, stride=1, padding=2, activation-ReLU
6
Upsample
fcate_factor=2
7
Conv2D
filters33, kernel=5 5, stride=1, padding-2, activation=Sigmoid
Architecture of c, SVHN+MNIST case
Layer
Type
Parameters
Dropout
p=0.55
2
Dense
nb menerons=100, activation=Softmax
Architecture of r, and ra. SVHN+-MNIST case
Layer
Type
Parameters
Gradient Reversal Layer
2
Dense
nb D0.neuron==100 100, activation=ReLU
3
Dense
nb _ebunroos=75 for SVHN-MNIS activation=Linear nb ubbnnuuros=200 for MNIST -SVHN activation=Linear
Architecture of II. -NIST case
Layer
Type
Parameters
Conv2D
filters=50, kernel55 x 5, stride=1, padding=2, aclivatiiRll Batch Norm.
2
Max Pooling
filters=2 x 2, stride=2
3
Conv2D
filters=75, kernel=5 5, stride=], padding=2, activation=ReLL, Batch Norm.
4
Max Pooling
filters=2 x 2, stride=2
5
Conv2D
filters=100, kernel=3 3, stride=1, paddig= activvatoooi))) Batch Norm.
6
Dense
nb Ab_neurons=1024, activation=Rel.U
11,
Dense
nb neurons=150, activation=ReluU
11,
Dense
nb ncurons=150, activation=ReLU
15

---

Under review as a conference paper at ICLR 2021
Architecture of II, and II, MNNSST+UUSSS case
Layer
Type
Parameters
Dense
nb_ 1bonneurons=1024, activation=Rel.U
2
Dense
nb. Jb.nneuros=66,, Retivation=ReLL
3
Conv2D
filters=100, kemnel=3 x 3, stride=l, padding=l, activtioRRU
4
Upsample
scale factor=2
5
Conv2D
filters=50, kemel=5 x 5, stride=1, padding=2, activation-ReLU
6
Upsample
scale_factor=2
7
Conv2D
filters=1, 1lters==1, kernelã x 5, stride=1, padding-2, p...ttieel..dddingg22iiiiiiioiid activttiiiiiiiiiiiiiiiiniii
Architecture of MNIST++USPS case
Layer
Type
Parameters
Layer
Type
Parameters
Dropout
p=0.55
2
Dense
nb neurons= 10, activation=Softmax
Architecture of r, and ra, MNIST++SSPS case
Layer
Type
Parameters
1
Gradient Reversal Layer
2
Dense
nb n0uuron===10, activation-ReLU
3
Dense
nb 15_neuroos=150, activation=Linear
Architecture of II, GySSS case
Layer
Type
Parameters
Conv2D
filters332, kernel=5 x 5, stride=l, peci Rel.V Batch Norm.
2
Max Pooling
filters=2 2, stride=2
3
Conv2D
filters=64, kernel=5 x 5, stride=], padding=2 activaiiiiiti Batch Norm.
4
Max Pooling
filters=2 x 2, strid==2
5
Conv2D
filters=128, kernel=3 3, stride= padding-,, acctvvtiiRL Batch Norm.
6
Max Pooling
filters=2 x 2, stride=2
7
Conv2D
filters=128, kernel=3 3, stride=1, padding=I, Rctiiiin-LL, Batch Norm.
8
Dense
nb neurons=1024, activation-ReLU
11,
Dense
nb ub__eurons=75, activation=RelLU
11
Dense
nb neurons=75, activation=Rel.u
Architecture of II, and 11, Sy--SignnnnS5 case
Layer
Type
Parameters
Conv2D
filters664, kernel=4 4, padding=l,
hatp::/4eeeaaaaaa....aea//.////a activatoRR..
2
Upsample
scale
3
Conv2D
filters=32, kernel=3 x
stride= padding=2, 1addde==,,, activainnRRL
4
Upsample
scale factor=2
5
Conv2D
filters332, kernel33 x 3, stride=1, 2
puiidell..uunggggeeaaiiiiiiailia activttioo
6
Upsample
scale_factor=2
7
Conv2D
filters=I, kernel33 3, stride=1, paddin acciriigggm
Architecture of Syn-Signx-GGTSRB
Layer
Type
Parameters
Dropout
p=0.55
2
Dense
nb ab_neurons=10, activation=Softmax
Architecture of r, and r.. Syn-Signs++GTSRB case
Layer
Тype
Parameters
Gradient Reversal Layer
2
Dense
nb neurons= 100, activation=Rel.u
3
Dense
nb neurons=150, activation=Lincar
16

---

Under review as a conference paper at ICLR 2021
For the GTA5 - Cityscapes experiment, based network's architecture the proposed by (Romera et al. 2017).
Architecture of 11 GTA5 - Cityscapes case
Layer
Type
Downsampler block, filters=16
2
Downsampler block, filters=64
3-7
Non-bt-ID.
8
Downsampler block, filters1128
9
Non-bt-ID (dilated 2)
10
Non-bt-1D (dilated 2)
11
Non-bt- ID (dilated 2)
12
Non-bt-ID (dilated 2)
13
Non-bt- (dilated 2)
14
Non-bt- ID (dilated 2)
15
Non-bt-ID (dilated 2)
16
Non-bt- ID (dilated 2)
11,
Conv2d filter= 14, kemel=1,, activation=ReLU
11,
Conv2d filter= k activation=ReLU
Architecture of c, II, and II, GTA5 Cityscapes case
Layer
Type
Conv2D filters=64 kernel=3 x 3, stride=2, activtion,
2
Upsample scale_factor-2
3-4
non-bottleneck-ID
5
Conv2D filters=16 kernel=3 x 3, stride=2, padding=1, activation-ReLU
6
Upsample ffato_fccto-22
7-8
non-bottleneck-11
9
Upsample scale factor=2
10
Conv2D filters=20 for and filters=3 for II, and IT, kernel=3 3, stride=2, padding=l, aciivinggl,
C DISCUSSION ON THE Binso AND Baoomin cyelic COEFFICIENTS SCHEDULE
Although the schedule on iinfo (single domain case) and Bdomain.cyclic (domain adaptation case) not absolutely necessary, we found out it helped the overall convergence. These coefficients gradually increase the weight the information disentanglement objective and the cross-domain reconstruction objective. This assigns early stages. From this perspective, gradually increasing 3info can be seen as gradually removing task-useless information from T and transferring S. Similarly, increasing 3domain eyelic responds to letting the network discover disentangled representations before aligning them across domains.
importance learning a good predictor II during
As previously mentioned, our goal in this study was to provide a robust disentanglement method that permits domain adaptation. Therefore, performed and these findings are thus reported as such and might be incomplete. Refining the understanding of the influence of the different 8 coefficients is closer the problem of meta-learning and is beyond the scope of this paper.
complete param-errr study and tuning was
D INFLUENCE OF BATCH NORMALIZATION AND DROPOUT ON DICYR
Batch normalization (Toffe & Szegedy 2015) is an efficient way to reduce the discrepancy between the source and target distributions statistics. We noticed that, for the specific SVHN + MNIST setting, using instance normalization (lyanov et al.) 2016) slightly improves the target domain accuracy. Normalizing across chanels, the instance normalization layers helps the networks to be agnostic to the image contrast which is particularly strong MNIST. We also noticed that using large dropout in the sub-network and small embedding dimensions for IT's ouputs improves
17

---

Under review as a conference paper at ICLR 2021
both the disentanglement quality and the target domain accuracy. We conjecture that the informa- tion bottleneck induced forces the task-related representation to be as concise as possible and thus encourages disentanglement.
E COMPLIMENTARY INFORMATION RELATED TO THE MACHINE LEARNING REPRODUCIBILITY CHECKLIST
All the experiments from section were run on a Google Cloud Platform nl-standard-8 virtual machine (8 virtual cores, 30Go RAM, Nvidia P100 GPU) except the experiment on GTAS - Cityscapes for which two Nvidia P100 GPU were used. The code corresponding to the experi- ments, list of dependencies, and pre-trained models are available htttps: / //gittubl com/ AnonymoouDD.Cy#//Dicyy Details about each experiments are reported on Table
Number of experiment repetitions
Batch size
Number of epochs
Time epoch
by
Experiment
Image size
5.1 SVHN
32x32
64
50
35 s
50
5.1 3D Shapes
32x32
64
50
15 s
5
5.2 MNIST
32x32
128
150
11 S
20
5.2 USPS SOPSS-MMNT MNIST
32x32
128
150
11 s
20
5.2 MINISTSSVHN
32x32
128
50
40 s
50
5.2 SVHN-MNIST
32x32
64
50
40 s
50
5.2 Pnn-Signs--GTSEE
64x64
64
150
65 s
10
5.2 GTA5 -ityyspisss
256 x 256
100'
200
128 s
3
To account for GPU memory limits, the gradients were accumulated over batches of size 20.
Table 3: Experimental setup
F DISTRIBUTION OF RESULTS
In Figure we report distribution testing accuracies for SVHN-MNIST benchmark of Section 5.2 Most of the accuracies are distributed within three modes with low variance, centered respectively around 89%, 94% and 98%, the latter being also the median and the majority mode. This sheds more detailed light the results reported in Section 5.2 As discussed therein, jecture these modes stem from the local equilibria the adversarial optimization problem induced by the GRLs in DiCyR. Thus expect that avoiding this problem altogether in the formulation of DiCyR could improve these resulting distributions.
14
CUURT
8-
6-
2-
85.0
92.5
97.5
accuracy
Figure 8: Distribution of testing accuracies across experiments for
18

---

Under review as a conference paper at ICLR 2021
G DOMAIN ADAPTATION FOR THE GTAS TTASS*CITYSCAAR +CITYSCAPES SEGMENTATION TASK
Here we report on the application of DiCyR for the segmentation task in GTA5 (Richter et al. 2016 and Cityscapes (Cordts al. 2016 GTA5 is the source domain, where the ground truth of image segments is provided, and the goal is to reach efficient segmentation on the Cityscapes dataset.
Figure provided first visual illustration of the benefits provided by DiCyR. Figure 9 provides yet other such examples. Column [a presents the testing image from the Cityscapes dataset, col- umn 9b shows the application the testing image of classifier trained only the GTA5 images, column 9c shows the segmentation obtained by DiCyR, which can be compared to the ground truth
(column
(a) Test image
(b) Source only
(c) DiCyR
(d) Ground truth
Figure 9: GTAS to Cityscapes segmentation
Table reports the Intersection over Union criterion (the Jaccard index) for each object class in the
Cityscapes images. We compare the seminal approach of Hotfman et al. (2016) coined "FFNNs in the wild", which served as a baseline for CyCADA (Hoffman al. 2018 and the application of DiCyR. These results are preliminary! and did not benefit from any hyperparameter or architecture tuning. The purpose of this table report the out-of-the- performance of DiCyR and validate the rationale behind performing disentanglement, on challenging, large scale problem.
'They were obtained following very valuable suggestion by a reviewer.
19

---

Under review as a conference paper at ICLR 2021
Tight
sign
vegetation
motoncycle
sidewalk
building
person
bieycle miou
wall
Fence
4ky
riler
CAP
trick
bos
train
Source only 68.6 11.8
wild 57.4 5.4
2.5
11.3
6.6
0.9
65.5
12.9
42
13
1.7
41.9
4.7
3.8
2.8
1.8
0.0
18.7
10,9
27.1
CyCADA 85.2 37.2
76.5
21.
15.0
23.8
22.9
21.5
80.5
31.3
60.7
50.5
9.0
76.9
17.1
28.2
4.5
9.8
0.0
35,4
DICYR 69.2 15.5 68.2
15.4
92
12
9.6
1.2
70.1
32
60.9
28.6
59.6
8.9
8.7
2.0
4.4
0.0
25.0
Table 4: Intersection over Union (IoU) criterion for each object class
20


Reviews:
[{"review_id": "tM0tNkNWY2", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the domain adaptation problem by addressing the challenge of splitting task-specific and task-orthogonal information in the target domain using the proposed disentangled cyclic reconstruction method. The authors further develop a variant for the unsupervised domain adaption (UDA) task. The authors argue that the existing adversarial classifier based UDA solutions do not guarantee that the domain specific information does not contain any information that overlaps with the shared information in the target domain. Another shortcoming of these baselines is the learned representation in their domain-invariant feature space might not allow for accurate labeling in the target domain. To solves these limitations, the authors directly minimize the information sharing between representation, instead of using domain adversarial classifier and adversarial label predictor.\n\nThis paper is well structured and easy to follow. The authors make several interesting and insightful statements. While the justifications of these main statements are not well presented theoretically/experimentally. \n\nIn page 3, the authors argue that the existing solutions achieve a domain-invariant feature space in which the representation might not allow for accurate labeling in the target domain. The following explanation reads vague to me. A qualitive example or an analytical experiment is encouraged to have to justify the mentioned “some elements” will hinder the performance.\n\nMy major concern is the evaluation results compared with the baselines is not competitive. As shown in Table 2, in the benchmark MNIST/USPS/SVHN datasets, the proposed method is inferior to the baselines like SEDA, DWT, SBADA-GAN. Since these datasets are relatively simple, I would encourage the authors to conduct experiments on other more challenging UDA benchmarks, like DomainNet [ref-1] and VisDA2017 [ref-2]\n[ref-1] X. Peng, et al., Moment matching for multi-source domain adaptation. In ICCV, 2019.\n[ref-2] X . Peng, et al. Visda:  The visual domain adaptation challenge.arXiv preprint arXiv:1710.06924, 2017.\n\nMissing some recent related works, including but not limted to\n[ref-3] Ryuhei Takahashi, et al: Partially-Shared Variational Auto-encoders for Unsupervised Domain Adaptation with Target Shift. ECCV 2020\n[ref-4] Jian Liang, et al: Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation, ICML 2020\n\nOther minor point. In page 3, last sentence in paragraph 4, “ a discriminator with a an inverted label loss”. Remove ‘a’.\n\nUpdates: Thanks for the authors' response. Some of my queries were clarified. However, unfortunately, I still think more needs to be done to show the superiority of the results. I retain my original decision.", "extra_scores": {}, "extra_text": {}}, {"review_id": "iRDdZMQRk0t", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#####################\n\nSummary:\n\nThe paper proposes a disentanglement method, named DiCyR in the problem of unsupervised domain adaptation.\n\n#####################\n\nReason for score:\n\nOverall, the paper is above the borderline. I like the idea of utilizing disentanglement learning with a cycle constraint into the unsupervised domain adaptation issue. My major concern is about some unclear parts described in the paper and insufficient experimental comparison (see cons below). Hopefully, it would be grateful that the authors could address my concerns during the rebuttal period.\n\n#####################\n\nPros:\n\n(1) The idea in the paper to introduce disentanglement learning based on cyclic reconstruction to unsupervised domain adaptation is very novel and interesting, which can split the captured information from data into task-related representation and context-based features to help improve the domain generality of the model.\n\n(2) The disentanglement learning to split the information into a task-related representation and a context representation is reasonable and promising, which is beneficial to the task model adapted in different domains.\n\n(3) Extensive experimental results on several widely-adopted benchmarks show the effectiveness of the proposed model comparing with other state-of-the-art methods.\n\n#####################\n\nCons:\n\n(1) In the paper, the introduced disentanglement learning method can split information captured from images into a task-specific feature and another task-orthogonal representation. However, the proposed method cannot consider the domain-specific and domain-orthogonal information, which may degrade the performance in experiments and deserves to be discussed further. So how to learn the domain-invariant representation in the model? Please explain more details.\n\n(2) In the part of the experiment in the paper, the authors conduct the experiments on the task of image style transfer. However, the task of semantic segmentation on Cityscapes is always widely adopted in related UDA papers, which is not included in the paper. Please consider adding such experimental results and analysis to demonstrate the effectiveness of the proposed method in the paper if feasible.\n\n(3) In the paper, the author claims the proposed model can achieve high computational efficiency, but there are no figures or tables to demonstrate it. Please consider listing the comparison results of training time and test time in the experimental part.\n\n#####################\n\nQuestions during the rebuttal period: \n\nPlease address and clarify the cons above. Thank you!\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5Z38T3YAu6B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new framework for unsupervised domain adaptation by applying the disentangled representations learning (DiCyR). The core idea of DiCyR is to split the raw feature into the task-related one and its complimentary context where the task-related representations are projected into a shared space for alignment. From my point of view, disentangled representations learning is the most interesting part of this work.\n\n\n-Pros:\n1. This paper is well organized with clear logic to follow. The authors have introduced the problem statement and related works very well.\n2. The motivation of disentangled representations learning for domain adaptation is very straightforward and clear. \n3. The authors provide the demo code of this work to prove its reproductivity.\n4. There are some visual results that could attract readers a lot.\n\n-Cons:\n1. Novelty: \nTaking the cross-domain image translation is a smart way of data augmentation, which has been well studied. Since 2017, CyCADA has introduced the CycleGAN framework for domain adaptation and has received a significant impact since then. This work extends the image translation idea by applying the disentangled image generation based on cyclic consistency. Moreover, there are a lot of previous papers working on similar topics [1, 2, 3, 4]. In this way, I think the novelty of this paper is minor.\n\n2. Experiment:\nIn the domain adaptation task, the authors only evaluate its model on the toy datasets (Digital Datasets), which cannot fully prove this model's effectiveness. I am curious why not using the latest office-home, domainnet, or the Cityscapes-to-GTA5 as the main experiment. On MNIST to SVHN, the DiCyR performs rather weakly. Moreover, there are more recent works, such as [5] should be compared.\n\n3. Time and Computational Cost\nCycleGAN-based image translation usually takes a lot of training time. The author mainly used the small size images (32 \\times 32) for evaluation, and it requires a P100 GPU for training. What the time and computational cost will be if using large images such as 224 \\times 224? \n\nIf the authors could address all the concerns above, I will consider upgrading the score.\n\n\n[1] Cai R, Li Z, Wei P, et al. Learning disentangled semantic representation for domain adaptation. IJCAI, 2019.\n\n[2] Cao J, Katzir O, Jiang P, et al. Dida: Disentangled synthesis for domain adaptation[J]. arXiv preprint arXiv:1805.08019, 2018\n\n[3] Li H, Wan R, Wang S, et al. Unsupervised Domain Adaptation in the Wild via Disentangling Representation Learning. IJCV, 2020\n\n[4] Yang, Junlin, et al. \"Unsupervised domain adaptation via disentangled representations: Application to cross-modality liver segmentation.\" MICCAI, 2019.\n\n[5] Hosseini-Asl, Ehsan, et al. \"Augmented cyclic adversarial learning for low resource domain adaptation.\" ICLR, 2019.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the problem of unsupervised domain adaptation and proposes a framework based on a specific type of disentangled representations learning. The paper is well written and the proposed method seems plausible. However, according to Reviewers #3 and #4, the proposed framework does not seems to be sufficiently different from existing ones, and the empirical results do not seem convincing enough. 

Please also double check in C3, whether T and S should be marginally independent or conditionally independent conditioning on X.
Decision: Reject