Domain Adaptive Semantic Segmentation andImage Classification

Guoliang Kang
Postdoctoral Research Associate
Carnegie Mellon University
kgl.prml@gmail.com

• Introduction• Contrastive Adaptation Network• Pixel-Level Cycle Association• SummaryDeep learning for Computer Vision Tasks• Image Classification• Semantic Segmentation• Object Detection• Tracking• ……

ImageNet

Cross-Domain Prediction• The distribution of test data is diﬀerent that of training data
Style, layout, shape, context, illumination, etc.

Training data

Test data

Cross-Domain Prediction• Performance degenerates due to the domain shiftBicycle

Motocycle

Deep ModelDomain Adaptation
The setting of domain adaptation• Target distribution is diﬀerent from the source one• Same task (shared label sets)• Large amounts of labeled source data and unlabeled target data
Why do we need domain adaptation?

• Costly to label large amounts of in-domain data
• Unrealistic to collect and annotate a dataset covering all the domain variations• The knowledge of the task can be potentially reuse/shared across domains

Discriminative Domain-Invariant Feature Learning
Through domain adaptation, we expect the learned features satisfy:• Domain-Invariant: indistinguishable from features• Discriminative: good inter-class separability and high intra-class compactness

Discriminative Domain-Invariant Feature Learning
Through domain adaptation, we expect the learned features satisfy:• Domain-Invariant: indistinguishable from features• Discriminative: good inter-class separability and high intra-class compactness
Conventional way to learn domain-invariant features• Ground-truth supervision from source data
• Sharing network parameters
Domain Discrepancy Minimization
image style transfer; adversarial loss; Maximum Mean Discrepancy (MMD); etc.
Consistency Regularization
self-ensemble method; attention alignment; etc.
Self-training based methods

Domain Discrepancy Minimization
Style Transfer

Domain Discrepancy Minimization
Adversarial Loss / Reverse Gradient[1] Ganin, Yaroslav, and Victor Lempitsky. "Unsupervised domain adaptation by backpropagation.”ICML, 2015.

Domain Discrepancy Minimization
Maximum Mean Discrepancy (MMD) Based∑∑∑DANJAN[1] Long, Mingsheng, et al. "Learning transferable features with deep adaptation networks.” ICML, 2015.
[2] Long, Mingsheng, et al. "Deep transfer learning with joint adaptation networks.” ICML, 2017.

Consistency Regularization
Self-ensembling[1] French, Geoﬀrey, Michal Mackiewicz, and Mark Fisher. "Self-ensembling for visual domain adaptation." ICLR, 2017.

Consistency Regularization
Style transfer + adversarial training + semantic consistency[1] Hoﬀman, Judy, et al. "Cycada: Cycle-consistent adversarial domain adaptation." ICML, 2018.

Consistency Regularization
Attention Alignment[1] Kang, Guoliang, et al. "Deep adversarial attention alignment for unsupervised domain adaptation: the benefit of target expectation maximization." ECCV. 2018..

Discriminative Domain-Invariant Feature Learning
Through domain adaptation, we expect the learned features satisfy:• Domain-Invariant: indistinguishable from features• Discriminative: good inter-class separability and high intra-class compactness
Contrastive Adaptation Network for the Image Classification• Class-aware alignment vs. Class-agnostic alignment (previous)[1] Kang, Guoliang, et al. "Contrastive adaptation network for unsupervised domain adaptation.” CVPR. 2019.
[2] Kang, Guoliang, et al. "Contrastive adaptation network for single-and multi-source domain adaptation." IEEE TPAMI (2020).

Pixel-Level Cycle Association for Domain Adaptive Semantic Segmentation• Align semantic-consistent pixel pairs vs. Align globally (previous)[3] Kang, Guoliang, et al. "Pixel-Level Cycle Association: A New Perspective for Domain Adaptive Semantic Segmentation.” NeurIPS (2020).

• Introduction• Contrastive Adaptation Network• Pixel-Level Cycle Association• SummaryMotivation

Class-aware alignment

Contrastive Domain Discrepancy
MMD measuring conditional distribution discrepancy

Contrastive Domain Discrepancy (CDD)

Intra: The MMD distance between cross-domain distributions conditioned on the same class.
Inter: The MMD distance between cross-domain distributions conditioned on diﬀerent classes.

Contrastive Adaptation Network

Contrastive Adaptation Network• ImageNet pertained weights to initialize the backbone

Contrastive Adaptation Network• ImageNet pertained weights to initialize the backbone• Align multiple Fully-Connected layers (including the final outputs)

Contrastive Adaptation Network• ImageNet pertained weights to initialize the backbone• Align multiple Fully-Connected layers (including the final outputs)• Overall Objective

Contrastive Adaptation Network• ImageNet pertained weights to initialize the backbone• Align multiple Fully-Connected layers (including the final outputs)• Overall Objective

Generate Target Label Hypotheses
Motivation/Assumption• The data from different categories is less likely to concentrate• The peaks of target feature distribution are good representatives for the underlying categories.
Initialize with Source Centers

Iterative Refinement via Spherical K-means Clustering
Attach Target Labels

Update Target Centers

Filtering
The ambiguous target data (i.e. far from the cluster centers) and ambiguous classes (i.e. containing
few target samples around the cluster centers) are zeroed out in estimating the CDD.

Alternative Optimization• Algorithm
The loop of AO is repeated multiple times in our experiments.
Asynchronously update of the target labels and the network parameters.

Clustering
FeatureAdaptation
Random Sampling

Class-Aware SamplingExtension to Multi-Source Setting
Framework

Extension to Multi-Source Setting
Boundary Sensitive Alignment

whereandExtension to Multi-Source Setting
Boundary Sensitive Alignment

Extension to Multi-Source Setting
Boundary Sensitive Alignment

Extension to Multi-Source Setting
Clustering Ensemble

Experiment Results
Datasets
Single-Source

Multi-Source

Experiment Results
Single-SourceOﬃce-31

VisDA-2017

Experiment Results
Multi-Source
DomainNet

Failure Case Analysis

• Introduction• Contrastive Adaptation Network• Pixel-Level Cycle Association• Summary

Domain Adaptive Semantic Segmentation
Semantic Segmentation

Domain Adaptive Semantic Segmentation

source trained network

target trained network

Previous Methods
Adversarial training based method[1] Tsai, Yi-Hsuan, et al. "Learning to adapt structured output space for semantic segmentation.” CVPR. 2018.

Previous Methods
Self-training based method[1] Lian, Qing, et al. "Constructing self-motivated pyramid curriculums for cross-domain semantic segmentation: A non-adversarial approach.” ICCV. 2019.

Motivation
Drawbacks of previous methods• adversarial training based methods:
1) Align globally;

2) Not discriminative enough.

• self-training based methods:
1) Need good initialization;2) Sensitive to the noise;3) Not stable enough.

Build associations between target and source pixels, and diminish pair-wise discrepancy

Pixel-Level Cycle Association

SourceTarget
S1

T
S2

S2

T

S1

similarities w.r.t T

starting: S1, end: S2 selected by T

T selected by S1

similarities w.r.t S1

Pixel-Level Cycle Association

SourceTargetPixel-Level Cycle Association

SourceTargetS1

S1starting: S1

similarities w.r.t S1

Pixel-Level Cycle Association

SourceTargetS1TT

S1starting: S1

T selected by S1

similarities w.r.t S1

Pixel-Level Cycle Association

SourceTargetS1

TT

S1
similarities w.r.t T

starting: S1

T selected by S1

similarities w.r.t S1

Pixel-Level Cycle Association

SourceTarget
S1

T
S2
S2
T

S1
similarities w.r.t T

starting: S1, end: S2 selected by T

T selected by S1

similarities w.r.t S1

Pixel-Level Cycle Association

SourceTarget
S1

TS2
S2
T

S1
similarities w.r.t T

starting: S1, end: S2 selected by T

T selected by S1

similarities w.r.t S1

Pixel-Level Cycle Association

SourceTarget
S1

T
S2
S2
T

S1
similarities w.r.t T

starting: S1, end: S2 selected by T

T selected by S1

similarities w.r.t S1

Pixel-Level Cycle Association

Similarity between features

Association loss

Contrast NormalizationGradient Diffusion via Spatial Aggregation

Spatial Aggregation

Gradient Diﬀusion

Gradient Diffusion via Spatial Aggregation

Spatial Aggregation

Gradient Diﬀusion

Multi-Layer Association
SourceBackbone

HeaderTargetsharedsharedBackbone

HeaderObjective

source-only
source+target
Cross-domain association loss

Adaptive LSR regularizer

where[1] Maxim Berman, et al. The Lovász-Softmax Loss: A Tractable Surrogate for the Optimization of the Intersection-Over-Union
Measure in Neural Networks, CVPR 2018

Experiment Results
Datasets

SourceSynthetic Images (SYNTHIA/GTAV)

TargetReal-World Images (Cityscapes)

Experiment Results
Ablation Study

Comparison with previous SOTA

Experiment Results

Cityscapes

Source-only

Our PLCA• Introduction• Contrastive Adaptation Network• Pixel-Level Cycle Association• SummarySummary• Without considering the discriminative ability of features, the adapted features
would be sub-optimal for the downstream task.

• Class-aware alignment helps avoid the misalignment and improve thegeneralization ability of features.

• In the semantic segmentation task, taking the pixel-wise discrepancy into
consideration is beneficial.

• In future, how to automatically optimize the discrepancy/alignment metric is worth
investigating.

Thank you for listening !