Title: DOMAIN ADAPTATION FOR STRUCTURED OUTPUT VIA DISENTANGLED PATCH REPRESENTATIONS

Abstract: Predicting structured outputs such as semantic segmentation relies on expensive per-pixel annotations to learn strong supervised models like convolutional neural networks. However, these models trained on one data domain may not generalize well to other domains unequipped with annotations for model finetuning. To avoid the labor-intensive process of annotation, we develop a domain adaptation method to adapt the source data to the unlabeled target domain. To this end, we propose to learn discriminative feature representations of patches based on label histograms in the source domain, through the construction of a disentangled space. With such representations as guidance, we then use an adversarial learning scheme to push the feature representations in target patches to the closer distributions in source ones. In addition, we show that our framework can integrate a global alignment process with the proposed patch-level alignment and achieve state-of-the-art performance on semantic segmentation. Extensive ablation studies and experiments are conducted on numerous benchmark datasets with various settings, such as synthetic-to-real and cross-city scenarios.

1. INTRODUCTION:

Recent deep learning-based methods have made significant progress on vision tasks, such as object recognition (Krizhevsky et al., 2012) and semantic segmentation (Long et al., 2015a) , relying on large-scale annotations to supervise the learning process. However, for a test domain different from the annotated training data, learned models usually do not generalize well. In such cases, domain adaptation methods have been developed to close the gap between a source domain with annotations and a target domain without labels. Along this line of research, numerous methods have been developed for image classification (Saenko et al., 2010; Ganin & Lempitsky, 2015) , but despite recent works on domain adaptation for pixel-level prediction tasks such as semantic segmentation (Hoffman et al., 2016) , there still remains significant room for improvement. Yet domain adaptation is a crucial need for pixel-level predictions, as the cost to annotate ground truth is prohibitively expensive. For instance, road-scene images in different cities may have various appearance distributions, while conditions even within the same city may vary significantly over time or weather.

Existing state-of-the-art methods use feature-level (Hoffman et al., 2016) or output space adaptation (Tsai et al., 2018) to align the distributions between the source and target domains using adversarial learning (Goodfellow et al., 2014; Zhu et al., 2017) . These approaches usually exploit the global distribution alignment, such as spatial layout, but such global statistics may already differ significantly between two domains due to differences in camera pose or field of view. Figure 1 illustrates one example, where two images share a similar layout, but the corresponding grids do not match well. Such misalignment may introduce an incorrect bias during adaptation. Instead, we consider to match patches that are more likely to be shared across domains regardless of where they are located.

One way to utilize patch-level information is to align their distributions through adversarial learning. However, this is not straightforward since patches may have high variation among each other and there is no guidance for the model to know which patch distributions are close. Motivated by recent advances in learning disentangled representations (Kulkarni et al., 2015; Odena et al., 2017) , we adopt a similar approach by considering label histograms of patches as a factor and learn discriminative Figure 1 : Illustration of the proposed patch-level alignment against the global alignment that considers the spatial relationship between grids. We first learn discriminative representations for source patches (solid symbols) and push a target representation (unfilled symbol) close to the distribution of source ones, regardless of where these patches are located in the image.

representations for patches to relax the high-variation problem among them. Then, we use the learned representations as a bridge to better align patches between source and target domains. Specifically, we utilize two adversarial modules to align both the global and patch-level distributions between two domains, where the global one is based on the output space adaptation (Tsai et al., 2018) , and the patch-based one is achieved through the proposed alignment by learning discriminative representations. To guide the learning process, we first use the pixel-level annotations provided in the source domain and extract the label histogram as a patch-level representation. We then apply K-means clustering to group extracted patch representations into K clusters, whose cluster assignments are then used as the ground truth to train a classifier shared across two domains for transferring a learned discriminative representation of patches from the source to the target domain. Ideally, given the patches in the target domain, they would be classified into one of K categories. However, since there is a domain gap, we further use an adversarial loss to push the feature representations of target patches close to the distribution of the source patches in this clustered space (see Figure 1 ). Note that our representation learning can be viewed as a kind of disentanglement guided by the label histogram, but is different from existing methods that use pre-defined factors such as object pose (Kulkarni et al., 2015) .

In experiments, we follow the domain adaptation setting in (Hoffman et al., 2016) and perform pixellevel road-scene image segmentation. We conduct experiments under various settings, including the synthetic-to-real, i.e., GTA5 (Richter et al., 2016) /SYNTHIA (Ros et al., 2016) to Cityscapes (Cordts et al., 2016 ) and cross-city, i.e., Cityscapes to Oxford RobotCar (Maddern et al., 2017) scenarios. In addition, we provide extensive ablation studies to validate each component in the proposed framework. By combining global and patch-level alignments, we show that our approach performs favorably against state-of-the-art methods in terms of accuracy and visual quality. We note that the proposed framework is general and could be applicable to other forms of structured outputs such as depth, which will be studied in our future work.

The contributions of this work are as follows. First, we propose a domain adaptation framework for structured output prediction by utilizing global and patch-level adversarial learning modules. Second, we develop a method to learn discriminative representations guided by the label histogram of patches via clustering and show that these representations help the patch-level alignment. Third, we demonstrate that the proposed adaptation method performs favorably against various baselines and state-of-the-art methods on semantic segmentation.

2. RELATED WORK:

Within the context of this work, we discuss the domain adaptation methods, including image classification and pixel-level prediction tasks. In addition, algorithms that are relevant to learning disentangled representations are discussed in this section. Domain Adaptation. Domain adaptation approaches have been developed for the image classification task via aligning the feature distributions between the source and target domains. Conventional methods use hand-crafted features (Gong et al., 2012; Fernando et al., 2013) to minimize the discrep-ancy across domains, while recent algorithms utilize deep architectures (Ganin & Lempitsky, 2015; Tzeng et al., 2015) to learn domain-invariant features. One common practice is to adopt the adversarial learning scheme (Ganin et al., 2016) and minimize the Maximum Mean Discrepancy (Long et al., 2015b) . A number of variants have been developed via designing different classifiers (Long et al., 2016) and loss functions (Tzeng et al., 2017; 2015) . In addition, other recent work aims to enhance feature representations by pixel-level transfer (Bousmalis et al., 2017) and domain separation (Bousmalis et al., 2016) .

Compared to the image classification task, domain adaptation for structured pixel-level predictions has not been widely studied. Hoffman et al. (2016) first introduce to tackle the domain adaptation problem on semantic segmentation for road-scene images, e.g., synthetic-to-real images. Similar to the image classification case, they propose to use adversarial networks and align global feature representations across two domains. In addition, a category-specific prior is extracted from the source domain and is transferred to the target distribution as a constraint. However, these priors, e.g., object size and class distribution, may be already inconsistent between two domains. Instead of designing such constraints, the CDA method (Zhang et al., 2017) applies the SVM classifier to capture label distributions on superpixels as the property to train the adapted model on the target domain. Similarly, as proposed in (Chen et al., 2017) , a class-wise domain adversarial alignment is performed by assigning pseudo labels to the target data. Moreover, an object prior is extracted from Google Street View to help alignment for static objects.

The above-mentioned domain adaptation methods on structured output all use a global distribution alignment and some class-specific priors to match statistics between two domains. However, such class-level alignment does not preserve the structured information like the patches. In contrast, we propose to learn discriminative representations for patches and use these learned representations to help patch-level alignment. Moreover, our framework does not require additional priors/annotations and the entire network can be trained in an end-to-end fashion. Compared to the recently proposed output space adaptation method (Tsai et al., 2018) that also enables end-to-end training, our algorithm focuses on learning patch-level representations that aid the alignment process.

Learning Disentangled Representation. Learning a latent disentangled space has led to a better understanding for numerous tasks such as facial recognition (Reed et al., 2014) , image generation (Chen et al., 2016b; Odena et al., 2017) , and view synthesis (Kulkarni et al., 2015; Yang et al., 2015) . These approaches use pre-defined factors to learn interpretable representations of the image. Kulkarni et al. (2015) propose to learn graphic codes that are disentangled with respect to various image transformations, e.g., pose and lighting, for rendering 3D images. Similarly, Yang et al. (2015) synthesize 3D objects from a single image via an encoder-decoder architecture that learns latent representations based on the rotation factor. Recently, AC-GAN (Odena et al., 2017) develops a generative adversarial network (GAN) with an auxiliary classifier conditioned on the given factors such as image labels and attributes.

Although these methods present promising results on using the specified factors and learning a disentangled space to help the target task, they focus on handling the data in a single domain. Motivated by this line of research, we propose to learn discriminative representations for patches to help the domain adaptation task. To this end, we take advantages of the available label distributions and naturally utilize them as a disentangled factor, in which our framework does not require to pre-define any factors like conventional methods.

3. DOMAIN ADAPTATION FOR STRUCTURED OUTPUT:

In this section, we describe our proposed domain adaptation framework for predicting structured outputs, our adversarial learning scheme to align distributions across domains, and the use of discriminative representations for patches to help the alignment.

3.1. ALGORITHMIC OVERVIEW:

Given the source and target images I s , I t ∈ R H×W ×3 and the source labels Y s , our goal is to align the predicted output distribution O t of the target data with the source distribution O s . As shown in Figure 2 (a), we use a loss function for supervised learning on the source data to predict the structured output, and an adversarial loss is adopted to align the global distribution. Based on this For target data, we employ another adversarial loss to align the patch-level distributions between F s and F t , where the goal is to push F t to be closer to the distribution of F s .

Objective Function. As described in Figure 2 (b), we formulate the adaptation task as composed of the following loss functions:

EQUATION

where L s and L d are supervised loss functions for learning the structured prediction and the discriminative representation on source data, respectively, while Γ denotes the clustering process on the ground truth label distribution. To align the target distribution, we utilize global and patch-level adversarial loss functions, which are denoted as L g adv and L l adv , respectively. Here, λ's are the weights for different loss functions. The following sections describe details of the baseline model and the proposed framework. Figure 3 shows the main components and loss functions of our method.

3.2. GLOBAL ALIGNMENT WITH ADVERSARIAL LEARNING:

We first adopt a baseline model that consists of a supervised cross-entropy loss L s and an output space adaptation module using L g adv for global alignment as shown in Figure 2 (a). The loss L s can be optimized by a fully-convolutional network G that predicts the structured output with the loss summed over the spatial map indexed with h, w and the number of categories C:

EQUATION

where O s = G(I s ) ∈ (0, 1) is the predicted output distribution through the softmax function and is up-sampled to the size of the input image. Here, we will use the same h and w as the index for all the formulations. For the adversarial loss L g adv , we follow the practice of GAN training by optimizing G and a discriminator D g that performs the binary classification to distinguish whether the output prediction is from the source image or the target one.

EQUATION

Then we optimize the following min-max problem for G and D g , with inputs to the functions dropped for simplicity: min

G max Dg L s (G) + λ g adv L g adv (G, D g ).

(4)

3.3. PATCH-LEVEL ALIGNMENT WITH DISCRIMINATIVE REPRESENTATIONS:

Figure 1 shows that we may find transferable structured output representations shared across source and target images from smaller patches rather than from the entire image or larger grids. Based on this observation, we propose to perform a patch-level domain alignment. Specifically, rather than naively aligning the distributions of all patches between two domains, we first perform clustering on patches from the source-domain examples using ground truth segmentation labels to construct a set of prototypical patch patterns. Then, we let patches from the target domain adapt to this disentangled (clustered) space of source patch representations by guiding them to select the closest cluster regardless of the spatial location via adversarial objective. In the following, we describe details of the proposed patch-level alignment.

Learning Discriminative Representations. In order to learn a disentangled space, class labels (Salimans et al., 2016) or pre-defined factors (Odena et al., 2017) are usually provided as supervisory signals. However, it is non-trivial to assign some sort of class membership to individual patches of an image. One may apply unsupervised clustering of image patches using pixel representations, but it is unclear whether the constructed clustering would separate patches in a semantically meaningful way. In this work, we take advantage of already available per-pixel annotations in the source domain to construct semantically disentangled space of patch representations.

To achieve this, we use label histograms for patches as the disentangled factor. We first randomly sample patches from source images, use a 2-by-2 grid on patches to extract spatial label histograms, and concatenate them into a vector, where each histogram is a 2 • 2 • C dimensional vector. Second, we apply K-means clustering on these histograms, whereby the label for any patch can be assigned as the cluster center with the closest distance on the histogram.

To incorporate this clustered space during training the network G on source data, we add a classification module H after the predicted output O s , to simulate the procedure of constructing the label histogram and learn a discriminative representation. We denote the learned representation as F s = H(G(I s )) ∈ (0, 1) U ×V ×K through the softmax function, where K is the number of clusters.

Here, each data point on the spatial map F s corresponds to a patch of the input image, and we obtain the group label Γ(Y s ) for each patch accordingly. Then the learning process to construct the clustered space can be formulated as a cross-entropy loss:

EQUATION

Patch-level Adversarial Alignment. The ensuing task is to align the representations of target patches to the clustered space constructed in the source domain. To this end, we utilize another adversarial loss between F s and F t , where F t is generated in the same way as described above. Our goal is to align patches regardless of where they are located in the image, that is, without the spatial and neighborhood supports. Thus, we reshape F by concatenating the K-dimensional vectors along the spatial map, which results in U • V independent data points. We note that a similar effect can be achieved by using a convolution layer with a proper stride and kernel size. We denote this reshaped data as F and formulate the adversarial objective:

EQUATION

where D l is the discriminator to classify whether the feature representation F is from the source or the target domain. Finally, we integrate ( 5) and ( 6) into the min-max problem in (4):

EQUATION

3.4 NETWORK OPTIMIZATION Following the standard procedure for training a GAN (Goodfellow et al., 2014) , we alternate the optimization between three steps: 1) update the discriminator D g , 2) update the discriminator D l , and 3) update the network G and H while fixing the discriminators.

Update the Discriminator D g . We train the discriminator D g to distinguish between the source output distribution (labeled as 1) and the target distribution (labeled as 0). The maximization problem on D g in ( 7) is equivalent to minimizing the binary cross-entropy loss:

EQUATION

)

Update the Discriminator D l . Similarly, we train the discriminator D l to classify whether the feature representation F is from the source or the target domain:

EQUATION

Update the Network G and H. The goal of this step is to push the target distribution closer to the source distribution using the optimized D g and D l , while maintaining good performance on the main tasks using G and H. As a result, the minimization problem in ( 7) is the combination of two supervised loss functions, namely, (2) and ( 5), with two adversarial loss functions, where the adversarial ones can be expressed as binary cross-entropy loss functions that assign the source label to the target distribution:

EQUATION

We note that updating H would also update G through back-propagation, and thus the feature representations are enhanced in G. In addition, we only require G during the testing phase, so that runtime is unaffected compared to the baseline approach.

3.5. NETWORK ARCHITECTURE AND IMPLEMENTATION DETAILS:

Discriminator. For the discriminator D g using a spatial map O as the input, we adopt an architecture similar to (Radford et al., 2016) For the discriminator D l , input data is a K-dimensional vector and we utilize 3 fully-connected layers similar to (Tzeng et al., 2017) , with leaky ReLU activation and channel numbers {256, 512, 1}.

Generator. The generator consists of the network G with a categorization module H. For a fair comparison, we follow the framework used in (Tsai et al., 2018) that adopts DeepLab-v2 (Chen et al., 2016a) with the ResNet-101 architecture (He et al., 2016) pre-trained on ImageNet (Deng et al., 2009) as our baseline network G. To add the module H on the output prediction O, we first use an adaptive average pooling layer to generate a spatial map, where each data point on the map has a desired receptive field corresponding to the size of extracted patches. Then this pooled map is fed into two convolution layers and a feature map F is produced with the channel number K. Figure 3 illustrates the main components of the proposed architecture. Implementation Details. We implement the proposed framework using the PyTorch toolbox on a single Titan X GPU with 12 GB memory. To train the discriminators, we use the Adam optimizer (Kingma & Ba, 2015) with initial learning rate of 10 -4 and momentums set as 0.9 and 0.99. For learning the generator, we use the Stochastic Gradient Descent (SGD) solver where the momentum is 0.9, the weight decay is 5 × 10 -4 and the initial learning rate is 2.5 × 10 -4 . For all the networks, we decrease the learning rates using the polynomial decay with a power of 0.9, as described in (Chen et al., 2016a) . During training, we use λ d = 0.01, λ g adv = λ l adv = 0.0005 and K = 50 for all the experiments. Note that we first train the model only using the loss L s for 10K iterations to avoid initially noisy predictions and then train the network using all the loss functions for 100K iterations. More details of the hyper-parameters such as image and patch sizes are provided in the appendix.

4. EXPERIMENTAL RESULTS:

We evaluate the proposed framework for domain adaptation on semantic segmentation. We first conduct an extensive ablation study to validate each component in the algorithm on the GTA5-to-Cityscapes (synthetic-to-real) scenario. Second, we show that our method performs favorably against state-of-the-art approaches on numerous benchmark datasets and settings.

4.1. EVALUATED DATASETS AND METRIC:

We evaluate our domain adaptation method on semantic segmentation under various settings, including synthetic-to-real and cross-city scenarios. First, we adapt the synthetic GTA5 (Richter et al., 2016) dataset to the Cityscapes (Cordts et al., 2016) dataset that contains real road-scene images. Similarly, we use the SYNTHIA (Ros et al., 2016) dataset with a larger domain gap compared to Cityscapes images. For the above experiments, we follow (Hoffman et al., 2016) to split the training and test sets. To overcome the realistic case when two domains are in different cities under various weather conditions, we adapt Cityscapes with sunny images to the Oxford RobotCar (Maddern et al., 2017) dataset that contains rainy scenes. We manually select 10 sequences in the Oxford RobotCar dataset annotated with the rainy tag, in which we randomly split them into 7 sequences for training and 3 for testing. We sequentially sample 895 images as training images and annotate 271 images with per-pixel semantic segmentation ground truth as the test set for evaluation. The annotated ground truth will be made publicly available. For all the experiments, intersection-over-union (IoU) ratio is used as the metric to evaluate different methods.

4.2. ABLATION STUDY:

In Table 1 , we conduct an ablation study on the GTA5-to-Cityscapes scenario to understand the impact of different loss functions and design choices in the proposed framework.

Loss Functions. In the first row of Table 1 , we show different steps of the proposed method, including disentanglement, global alignment, and patch-level alignment. Interestingly, we find that adding disentanglement without any alignments (L s + L d ) also improves the performance (from 36.6% to 38.8%), which demonstrates that the learned feature representation enhances the discrimination and generalization ability. Finally, as shown in the last result of the second row, our method that combines both the global and patch-level alignments achieve the highest IoU as 43.2%.

Impact on L d and L l adv . In the first two results of the second row, we conduct experiments to validate the effectiveness of our patch-level alignment. We show that both losses, L d and L l adv , are necessary to assist this alignment process. Removing either of them will result in performance loss, i.e., 1.9% and 1.5% lower than our final result. The reason behind this is that, L d is to construct a clustered space so that L l adv can then effectively perform patch-level alignment in this space.

Our Method Without Patch Adaptation Without Reshaped F . In the module H that transforms the output distribution to the clustered space, the features are reshaped as independent data points F to remove the spatial relationship and are then used as the input to the discriminator D l . To validate the usefulness, we show that without the reshaping process, the performance drops 2.4% in IoU. This result matches our assumption that patches with similar representations should be aligned regardless of their locations.

Visualization of Feature Representations. In Figure 4 , we show the t-SNE visualization (van der Maaten & Hinton, 2008) of the patch-level features in the clustered space of our method and compare with the one without patch-level adaptation. The result shows that with adaptation in the clustered space, the features are embedded into groups and the source/target representations overlap to each other well. Example patch visualizations are provided in the appendix.

4.3. COMPARISONS WITH STATE-OF-THE-ART METHODS:

In this section, we compare the proposed method with state-of-the-art algorithms under various scenarios, including synthetic-to-real and cross-city cases.

Synthetic-to-real Case. We first present experimental results for adapting GTA5 to Cityscapes in Table 2 . The methods in the upper group adopt the VGG-16 architecture as the base network and we show that our approach performs favorably against state-of-the-art adaptations via feature (Hoffman et al., 2016; Zhang et al., 2017) , pixel-level (Hoffman et al., 2018) , and output space (Tsai et al., 2018) alignments. In the bottom group, we further utilize the stronger ResNet-101 base network and compare our result with (Tsai et al., 2018) under two settings, i.e., feature and output space adaptations. We show that the proposed method improves the IoU with a gain of 1.8% and achieves the best IoU on 14 out of the 19 categories. In Table 3 , we show results for adapting SYNTHIA to Cityscapes and similar improvements are observed comparing with state-of-the-art methods. In addition, we shows visual comparisons in Figure 5 and more results are presented in the appendix.

Cross-city Case. Adapting between real images across different cities and conditions is an important scenario for practical applications. We choose a challenge case where the weather condition is different (i.e., sunny v.s rainy) in two cities by adapting Cityscapes to Oxford RobotCar. The proposed Target Image Ground Truth Before Adaptation Global Alignment Ours

Figure 5 : Example results for GTA5-to-Cityscapes. Our method often generates the segmentation with more details (e.g., sidewalk and pole) while producing less noisy regions. framework achieves a mean IoU of 65.0% averaged on 10 categories, significantly improving the model without adaptation by 8.8%. To compare with the state-of-the-art method (Tsai et al., 2018) , we run the authors' released code and obtain a mean IoU of 63.6%, which is 1.4% lower than the proposed method. Further results and comparisons are provided in the appendix.

5. CONCLUSIONS:

In this paper, we present a domain adaptation method for structured output via a general framework that combines global and patch-level alignments. The global alignment is achieved by the output space adaptation, while the patch-level one is performed via learning discriminative representations of patches across domains. To learn such patch-level representations, we propose to construct a clustered space of the source patches and adopt an adversarial learning scheme to push the target patch distributions closer to the source ones. We conduct extensive ablation study and experiments to validate the effectiveness of the proposed method under numerous challenges on semantic segmentation, including synthetic-to-real and cross-city scenarios, and show that our approach performs favorably against existing algorithms.

D RESULT OF ADAPTING CITYSCAPES TO OXFORD ROBOTCAR:

In Table 5 , we present the complete result for adapting Cityscapes (sunny condition) to Oxford RobotCar (rainy scene). We compare the proposed method with the model without adaptation and the output space adaptation approach (Tsai et al., 2018) . More qualitative results are provided in Figure 7 and 8. We provide more visual comparisons for GTA5-to-Cityscapes and SYNTHIA-to-Cityscapes scenarios from Figure 9 to Figure 11 . In each row, we present the results of the model without adaptation, output space adaptation (Tsai et al., 2018) , and the proposed method. We show that our approach often yields better segmentation outputs with more details and produces less noisy regions. We sequentially show images in a video and their adapted segmentations generated by our method.

Target Image Ground Truth Before Adaptation Global Alignment Ours 

:



Figures and Tables:

Figure 2: Overview of the proposed framework. (a) A baseline model that utilizes global alignment in (Tsai et al., 2018). (b) Our algorithm combines global and patch-level alignments in a general framework that better preserves local structure. Note that O s , O t ∈ (0, 1) H×W ×C are distributions and the figures used here are only for illustration purposes. baseline model, we further incorporate a classification loss in a clustered space to learn patch-level discriminative representations F s from the source output distribution O s , shown in Figure 2(b).For target data, we employ another adversarial loss to align the patch-level distributions between F s and F t , where the goal is to push F t to be closer to the distribution of F s .

Figure 3: The proposed network architecture that consists of a generator G and a categorization module H for learning discriminative patch representations. In the clustered space, solid symbols denote source representations and unfilled ones are target representations pulled to the source distribution.

Figure 4: Visualization of patch-level feature representations via t-SNE. In each figure, two thousand patches are sampled from each source and target domain.

Figure 7: Example results of adapted segmentation for the Cityscapes-to-OxfordRobotCar setting.We sequentially show images in a video and their adapted segmentations generated by our method.

Figure 8: Example results of adapted segmentation for the Cityscapes-to-OxfordRobotCar setting.We sequentially show images in a video and their adapted segmentations generated by our method.

Figure 9: Example results of adapted segmentation for the GTA5-to-Cityscapes setting. For each target image, we show results before adaptation, output space adaptation Tsai et al. (2018), and the proposed method.

but use fully-convolutional layers. It contains 5 convolution layers with kernel size 4 × 4, stride 2 and channel numbers {64, 128, 256, 512, 1}. In addition, a leaky ReLU activation (Maas et al., 2013) is added after each convolution layer, except the last layer.

Table content: <table/>

Ablation study on GTA5-to-Cityscapes using the ResNet-101 network. We also show the corresponding loss functions used in each setting.

Table content: <table><tr><td/><td/><td/><td>GTA5 → Cityscapes</td><td/></tr><tr><td/><td>Without Adaptation</td><td>Disentanglement</td><td>Global Alignment</td><td>Patch-level Alignment</td></tr><tr><td/><td>Ls</td><td>Ls + L d</td><td>Ls + L g adv</td><td>Ls + L d + L l adv</td></tr><tr><td>mIoU</td><td>36.6</td><td>38.8</td><td>41.4</td><td>41.3</td></tr><tr><td/><td>Without L d Ls + L g adv + L l adv</td><td>Without L l adv Ls + L d + L g adv</td><td>Without Reshaped F Ls + L d + L g adv + L l adv</td><td>Ours (final) Ls + L d + L g adv + L l adv</td></tr><tr><td>mIoU</td><td>41.3</td><td>41.7</td><td>40.8</td><td>43.2</td></tr></table>

Results of adapting GTA5 to Cityscapes. In the first and second groups, VGG-16 and ResNet-101 base networks are adopted, respectively.

Table content: <table><tr><td>GTA5 → Cityscapes</td></tr></table>

Results of adapting SYNTHIA to Cityscapes. In the first and second groups, VGG-16 and ResNet-101 base networks are adopted, respectively.

Table content: <table><tr><td>SYNTHIA → Cityscapes</td></tr></table>

Results of adapting Cityscapes to Oxford RobotCar.

Table content: <table><tr><td/><td/><td/><td/><td colspan="5">Cityscapes → Oxford RobotCar</td><td/><td/><td/></tr><tr><td>Method</td><td>road</td><td>sidewalk</td><td>building</td><td>light</td><td>sign</td><td>veg</td><td>sky</td><td>person</td><td>automobile</td><td>two-wheel</td><td>mIoU</td></tr><tr><td colspan="6">Without Adaptation 79.2 49.3 73.1 55.6 37.3</td><td>4.5</td><td colspan="4">36.1 54.0 81.3 49.7</td><td>56.2</td></tr><tr><td>Tsai et al. (2018)</td><td colspan="10">95.1 64.0 75.7 61.3 35.5 10.5 63.9 58.1 84.6 57.0</td><td>63.6</td></tr><tr><td>Ours</td><td colspan="5">94.9 64.4 82.8 62.3 35.2</td><td>8.7</td><td colspan="4">76.4 57.4 85.0 57.5</td><td>65.0</td></tr><tr><td colspan="4">E QUALITATIVE COMPARISONS</td><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

Peer review (iclr2019):
 This paper introduces a domain adaptation approach for structured output data, with a focus here on semantic segmentation. The idea is to model the structure by exploiting image patches, but account for the fact that these patches may be misaligned, and thus not in exact correspondence. This is achieved by defining new patch classes via clustering the source patches according to the semantic information, and making use of an adversarial classifier on the predicted patch-class distributions. Strengths: - Modeling the structure via patches is an interesting idea. - The proposed method achieves good results on standard benchmarks. Weaknesses: Method: - The idea of relying on patches to model the structure is not new. This was achieved by Chen et al., CVPR 2018, "ROAD: Reality Oriented Adaptation...". In this work, however, the patches were assumed to be in correspondence, which leaves some novelty to this submission, although reduced. - In essence, the patch-based adversarial alignment remains global; this can be thought of as working at a lower resolution and on a different set of classes, defined by the clusters, than the global alignment. The can be observed by comparing Eq. 3 and Eq. 6, which have essentially the same form. This is fine, but was not clear to me until I reached Eq. 6. In fact, what I understood from the beginning of the paper was an alternative formulation, where one would essentially assign each patch to a cluster and aim to align the distributions of the output (original classes) within each cluster. I suggest the authors to clarify this, and possibly discuss the relation with this alternative approach. - I am not convinced by the claimed relationship to methods that learn disentangled representations. Here, in essence, the authors just perform clustering of the semantic information. This is fine, but I find the connection a bit far-fetched and would suggest dropping it. Experiments: - The comparison to the state of the art is fine, but I suggest adding the results of Chen et al, CVPR 2018, which achieves quite close accuracies, but still a bit lower. The work of Saito et al., CVPR 2018, "Maximum Classifier Discrepancy..." also reports results on semantic segmentation and should be mentioned here. I acknowledge however that their results are not as good as the one reported here. - While I appreciate the ablation study of Section 4.2, it only provides a partial picture. It would be interesting to study the influence of the exact values of the hyper-parameters on the results. These hyper-parameters are not only the weights .lambda_d, .lambda^g_{adv} and .lambda^l_{adv}, but also the number of clusters and the size of the patches used. Summary: I would rate this paper as borderline. There is some novelty in the proposed approach, but it is mitigated by the relation to the work of Chen et al., CVPR 2018. The experiments show good results, but a more thorough evaluation of the influence of the hyper-parameters would be useful.