Adversarial Support Alignment
Authors: Shangyuan Tong
Timur Garipov
Yang Zhang
Shiyu Chang
Tommi S. Jaakkola
Venue: ICLR 2022 Spotlight (2022)
Keywords: support alignment
distribution alignment
optimal transport
domain adaptation

Abstract: We study the problem of aligning the supports of distributions. Compared to the existing work on distribution alignment, support alignment does not require the densities to be matched. We propose symmetric support difference as a divergence measure to quantify the mismatch between supports. We show that select discriminators (e.g. discriminator trained for Jensen-Shannon divergence) are able to map support differences as support differences in their one-dimensional output space. Following this result, our method aligns supports by minimizing a symmetrized relaxed optimal transport cost in the discriminator 1D space via an adversarial process. Furthermore, we show that our approach can be viewed as a limit of existing notions of alignment by increasing transportation assignment tolerance. We quantitatively evaluate the method across domain adaptation tasks with shifts in label distributions. Our experiments show that the proposed method is more robust against these shifts than other alignment-based baselines.

Published as a conference paper at ICLR 2022
ADVERSARIAL SUPPORT ALIGNMENT
Shangyuan Tong' MIT CSAIL
Timur Garipov MIT CSAIL
Yang Zhang
Shiyu Chang UC Santa Barbara
Tommi Jaakkola MIT CSAIL
MIT-IBM Watson AI Lab
ABSTRACT
We study the problem of aligning the supports of distributions. Compared to the existing work on distribution alignment, support alignment does not require the densities to be matched. We propose symmetric support difference as divergence measure quantify the mismatch between supports. We show that select discrimi- nators (e.g. discriminator trained for Jensen- -hannnon divergence) are able to map support differences as support differences their one-dimensional output space. Following this result, our method aligns supports by minimizing symmetrized relaxed optimal transport cost in the discriminator ID space via an adversarial process. Furthermore, existing notions of alignment by increasing transportation assignment tolerance. We quantitatively evaluate the method across domain adaptation tasks with shifts in label distributions. Our experiments show that the proposed method is more robust against these shifts than other alignment-bane baaseed baselines.
show that our approach be viewed as a limit of
INTRODUCTION
Learning tasks often involve estimating properties of distributions from samples or aligning such characteristics across domains. We can align full distributions (adversarial domain alignment), certain statistics (canonical correlation analysis), or the support of distributions (this paper). Much of the recent work has focused on full distributional alignment, for good reasons. In domain adaptation, motivated by theoretical results (Ben-David et al., 2007; 2010), a series of papers (Ajakan et al., 2014; Ganin & Lempitsky, 2015; Ganin al., 2016; Tzeng al., 2017; Shen et al., 2018; Pei al., 2015 8; Zhao et al., 2018; Li et al., 201 8a; Wang et al., 2021; ; Kumar et al., 2018) seek to align distributions of representations between domains, and utilize a shared classifier on the aligned representation space.
Alignment in distributions implies alignment in supports. However, when there are additional objec- tives/constraints satisfy, the minimizer for distribution alignment objective does necessarily minimize a support alignment objective. Example in Figure demonstrates the qualitative distinction between two minimizers when distribution alignment objective prefers to keep supports unaligned even if support alignment is achievable. Recent works (Zhao al., 2019; Li al., 2020; Tan al., 2020; Wu et al., 2019b; Tachet des Combes al., 2020) have demonstrated that a shift in label distributions between source and target leads to characterizable performance drop when the representations are forced into a distribution alignment. The error bound in Johansson al. (2019) suggests aligning the supports of representations instead.
achievable. The distribution alignment
In this paper, we focus on distribution support as the key characteristic to align. We introduce a support divergence measure the support mismatch and algorithms to optimize such alignment. We also position our approach in the spectrum of other alignment methods. Our contributions are as follows (all proofs can be found in Appendix A):
1. In Section 2. measure the differences between supports of distributions, Building on the Hausdorff distance, we introduce a novel support divergence better suited for optimization, which refer symmetric support difference (SSD) divergence.
First two authory contrihuted equally. Correspondence to Shangyuan Tong (sytong8csail .mit wwt.:saaammmiim..mmmmm.m...e...e edu). We provide the code reproducing experiment results https:://githhieee http:///tttooooo///n//iio//////t

---

Published as a conference paper at ICLR 2022
2. In Section 2.2, identify an important property of the discriminator trained for divergence: support differences in the original space of interest are "pre- served" as support differences in the one-dimensional discriminator output space.
3. In Section 3, we present our practical algorithm for support alignment, Adversurial Support Aligument (ASA). Essentially, based on the analysis presented Section 2.2, our solution is to align supports in the discriminator ID space, which is computationally efficient.
4. In Section 4. we place different notions of aligmment distribution alignment, relaxed distribution alignment and support alignment- within coherent spectrum from the point of view of optimal transport, characterizing their relationships, both theoretically in terms of their objectives and practically terms of their algorithms.
5. In Section 5, we demonstrate the effectiveness of support alignment in practice for domain adaptation setting. Compared to other alignment-based baselines, our proposed method is more robust against shifts in label distributions.
3.0
3.0
3.0
p(a)
2.5
2.5
2.5
('rr
Density
2.0
2.0
2.0
1.5
1.5
1.5
1.0
1.0
1.0
0.5
0.5
0.5
0.0
0.0
0.0
-3
-2
0
0.0
0.2
0.4
0.6
0.8 1.0
1.2
0.0
0.2
0.4
0.6
0.8
1.0
2
(a) Initialization
(b) Distribution alignment
(c) Support alignment
(p,p, d²)= 11.12 =1.12
Dw (p. g²) 2- 10
Dw q²) = 5- 10
DA 9²) = 14.9
DA 4
10-4
DA 4 < 1 10
Figure 1: Illustration of differences between the final configurations of distribution alignment and support alignment procedures. p(z) is a fixed Beta distribution p(x) = Beta(z 2, with support []: (x) is "shifted" Beta distribution q (z) = Beta( 0 4) parameterized by 0 with support [0, + 1]. Panel (a) shows the initial contiguration with 6mm = -3. Panel (b) shows the result by distribution alignment. Panel (c) shows the result by support alignment. We report Wasserstein
distance Dw (p, r²) (7) and SSD divergence D (p. q²) (1).
2 SSD DIVERGENCE AND SUPPORT ALIGNMENT
Notation. We consider an Euclidean space X R equipped with Borel sigma algebra B and metric d: : X x X - To. (e.g. Fuclidean distance). Let P be the set of prohability measures on (X, B). For E P. the support of p is denoted by supp(p) and is defined such that p(X)= = 1. f:p denotes the pushforward measure of p induced by a measurable mapping f. With slight abuse of notation, we use p(x) and (t) to denote the densities of measures p and fEP evaluated at æ and t respectively, implicitly assuming that the measures are absolutely continuous. The distance between point E X' and subset Y X' is defined as d(x, Y) inf yer d(x, y). The symmetric difference of two sets A and B is defined as A A B = (A B) U (B
the smallest closed set X X
A).
2.1 DIFFERENCE BETWEEN SUPPORTS
To align the supports of distributions, we first need to evaluate how different they are. Similar to distribution divergences like Jensen- -ensennShannon divergence, we introduce a notion of support divergence. support divergence between two distributions in P a function Ds(:.)): P x P + R satisfying: 1) Pp(p,0 q) 2 0 for all p.q E 2) (s(p. g) 0 iff supp(p) supp(g).
While a distribution divergence is sensitive to both density and support differences, a support divergence only needs to detect mismatches in supports, which are subsets of the metric space X.
PI is not technically divergence on the space of distributions, since Ds(p. 2) does not imply p q-
2

---

Published as a conference paper at ICLR 2022
An example of a distance between subsets of metric space is the Hausdorff distance: dy (x, max(sup,eex d(x, suuuppyey yey d(y, X)). Since it depends only on the greatest distance between a point and set, minimizing this objective for alignment only provides signal to a single point. To make the optimization less sparse, we consider all points that violate the support alignment criterion and introduce symmetric support difference (SSD) divergence:
Da (p 9) = Ezzp (d(x, supp(a))) + Eazn supps))pp()))
Proposition 2.1. SSD divergence DA (p. q) is a support divergence.
We note that our proposed SSD divergence is closely related to Chamfer distance/divergencee (CD) (Fan al., 2017: Nguyen al., 2021) and Relaxed Word Mover's s Distance (RWMD) (Kusner et al., 2015). While both CD and RWMD are stated for discrete points (see Section 6 for further comments), SSD divergence is general difference measure between arbitrary (discrete or continuous) distributions. This distinction, albeit small, is important in our theoretical analysis (Sections 2.2, 4.1).
2.2 SUPPORT ALIGNMENT IN ONE-DIMENSIONAL SPACE
Goodfellow al. (2014) showed that the log-loss discriminator f [0, 1]. trained distinguish samples from distributions p and q (sup, Ezep [log fxx) + Ezng [log(1 f()))) can be used to estimate the Jensen-Shannn divergence between p and q. The closed form maximizer f is
p(z) p(x) + (x=)
(x)
Va E supp(p) U Supp(())
Note that for a point x & supp(p) U supp(q) the value of (*x) can be set to an arbitrary value in
[0, 1], since the log-loss does not depend on f(x) for such æ. The form of the optimal discriminator (2) gives rise to our main theorem below, which characterizes the ability of the log-loss discriminator to identify support misalignment.
Theorem 2.1. Let and q be the distributions with densities satisfying
1
1 C < < < <C, Vz E supp(ą).
C < < C, Vo supp(p);
(3)
Let I be the optimal discriminator (2). Then, (p 0 and only Df ('r)) =0
The idea of the proof is to show that the extreme values (0 and 1) of (x) can only be attained in E supp(p) A supp(g). Assumption (3) guarantees that (xx) cannot approach neither in
the intersection of the supports supp(p) O supp(q), i.e. the values {f (z) x E supp(p) n supp(g)}
are separated from the extreme values 0 and 1.
We conclude this section with two technical remarks on Theorem 2.1.
Remark 2.1.1. The result of Theorem 2.1 does necessarily hold for other types of diseriminators. For instance, the dual Wasserstein discriminator (Arjovsky et al., 2017; Gulrajanii et al., 2017) does not always highlight the support difference in the original space as support difference in the discriminator output space. This observation is formaly stated in the following proposition.
Let füv be the maximizer of supp-L(/)s If the Lipschitz constant. There exist p and q with supp(p) # supp(g) but supp(Sivpp) supp(fi,g))
Proposition 2.2.
[f where L(-) is
Remark 2.1.2. In practice the discriminator is typically parameterized f(x) (((())) where X' - IR is realized by a deep neural network and (xx) = (1 + =--- is the sigmoid function. The optimization problem for g is
inf Expp [log(1 e =((s))] Exnn [og(1 + ea(+)) 9
(4)
and the optimal solution is g" (z) = log p(z) log (æ)). Naturally the result of Theorem 2.1 holds for g" since g" (xx) --((((x)) (z)) and is bijective mapping from R J {-oo, oo) to [0, 1).
3

---

Published as a conference paper at ICLR 2022
3 ADVERSARIAL SUPPORT ALIGNMENT
We consider distributions p and @ parameterized by pº.9² The log-loss discriminator g optimized for (4) is parameterized by t:: g Our analysis in Section 2.2 already suggests an algorithm. Namely, we can optimize by minimizing Da (g( pp g =²==") while optimizing & by (4). This adversarial game is analogous to the setup of the existing distribution alignment algorithms'.
In practice, rather than having direct access to p², .q" which is unavailable, we are often given i.i. samples [a {=]) 1--1--111-- They form discrete distributions (x) = E 6(x M
(z)
M Ein 8(x-xx), and [g" [tt)(()) & Σi 0(t- (xf"), 19 (+))
Li i-1 (x)))
Since nd "i and "" are discrete distributions, they have supports (x,)) and (x:))
respectively. SSD divergence between discrete distributions g np0 and gb ;000 is
N
M
D0 :D0 g =00)) = N E d (0))) + M Σ d ( (x (5)
1
Effect of mini-batch training. When training on large datasets, we need to rely on stochastic opti-
mization with mini- batches. We denote the mini-batches (of same size, as in common practice) from p" and q as x" = {xa Jis1 and x { j respectively. By minimizing Da (g (x*). g (Ir)), we only consider the mini-batch support distance rather than the population support distance (5). We observe that in practice the described algorithm brings the distributions to state closer to distribution alignment rather than support alignment (see Appendix D.5 for details). The problem in the typically small batch size. The algorithm actually tries enforce support alignment for all possible pairs of mini-batches, which is a much stricter constraint than population support alignment.
To address the issue mentioned above, without working with much larger batch size, we create two "history buffers": h², storing the previous ID discriminator outputs of (at most) n samples stores the values of the
from p", and similar buffer h2 for q". Specifically, h = Rov. (z { previous n samples Foid, mapped by their corresponding past "versions" of the discriminator g was. We minimize Da (v?, v where of concat(h*, (h,, h.".".g (z*)), v concat(h", go (z²)):
72 172
DA DA (2 v8)
Σ
d 09)
+
Σ d(ug, uP) 2=1
(6)
+ m
Note that (-)..)) between two sets of 1D samples can be efficiently calculated since d(u,,, u²) and d(o7, e²) are simply 1-nearest neighbor distances in ID. Moreover the history buffers store only the scalar values from the previous batches. These values are only considered in nearest neighbor assignment but do not directly provide gradient signal for optimization. Thus, the computation overhead of including a long history buffer is very light. We present our full algorithm, Adversarial Support Alignment (ASA), in Algorithm
SPECTRUM OF NOTIONS OF ALIGNMENT
In this section, we take a closer look into our work and different existing notions of alignment that have been proposed the literature, especially their formulations from the optimal transport perspective. We show that proposed support alignment framework is limit of existing notions of alignment, both in terms of theory and algorithm, by increasing transportation assignment tolerance.
4.1 THEORETICAL CONNECTIONS
Distribution alignment. Wasserstein distance is commonly used objective for distribution align- ment. In our analysis, we focus on the Wasserstein- distance:
Dw (p. q) = inf (ee,))y/d(x, v)].
(7)
FFollwwin existing adversarial distribution alignment methods, e.g. (Goodfellow et al., 2014), we use single update of db per 1 update of 0. While theoretical analysis for both distribution alignment and support alignment (ours) assume optimal discriminators, training with single update of vp is computationally cheap and effective.
4

---

Published as a conference paper at ICLR 2022
Algorithm 1 Our proposed ASA algorithm. (maximum history buffer size), we use n 1000.
1: for number of training steps do
2: Sample mini-batches [= ~ p². {a]]= ~ q².
3: Perform optimization step on vp using stochastic gradient
log(1 + exp(=e" (xf))) log(1 + exp(o
4: uP concat(h', (x²)) v + concat (49, (x7)) Tpppp - arg min, d(of, 2g)). 6: Perform optimization step on 9 using stochastic gradient
5: Tp-ig + arg min, d(e,, (2),) 2)),
og TH- d( ) d(op
V
Σ
7:
UPDATEHISTORYY(I; (x))))))))))) UPDATEHISTORY (hP, (xP)))))
8:
end for
where [(p, q) is the set of all measures on X' X' with marginals of p and q, respectively. The value of Dw (p. q) is the minimal transportation cost for transporting probability mass from q. The transportation cost is zero and only if p = q, meaning the distributions are aligned.
Relaxed distribution alignment. Wu et al. (2019b) proposed modified Wasserstein distance to achieve asymmetricalll--rlaxed distribution alignment, namely 3-admissible Wasserstein distance:
D (p. = inf [d(x,, v)l,
(8)
>ETa(P:9)
where gp) is the set of all measures on X x X' such that fr( Ja(a,, y)dz S (1 + B)q(y). Vy. With the relaxed marginal constraints, one could choose trans- portation plan y which transports probability mass from p to a modified distribution q' rather than the original distribution y as long as o' satisfies the constraint g'(x) S (1 + 3){(z), Va. Therefore, D (p.q) is zero if and only if p(x) (1 (x)) Yz. In (Wu et al., 2019b), / is normally set of distribution alignment, and is shown that Di (p. q) (p.g).) We can extend D (p, q) to symmetric version, which we term 31 admassiiiee Wasserstein distance:
p(x), Vi and
to positive finite number to achieve the asymmetric-relaxatee
D Di: (p. 9) + DW (q.
The aforementioned property of 3-admissible Wasserstein distance implies that Der (p. 9) 0 if and only if p(x) < (1 + B1 (x)) Yo and ((x)) < (1 32)p(z), Vr, in which case we call and "(8,, ali-nedd with 31 and B2 controlling the transportation assignment tolerances.
Support alignment. The term Ep [(x ((([()))p()))) in (1) represents the average distance from samples in p to the support of q. From the optimal transport perspective, this value the minimal transportation cost of transporting the probability mass of p into the support of q. We show that SSD divergence can be considered as transportation cost in the limit of infinite assignment tolerance.
Proposition 4.1. Di (p, q) limm -1
=Dx(p.0) q)-
We now have completed the spectrum of alignment objectives defined within the optimal transport framework. The following proposition establishes the relationship within the spectrum. Proposition 4.2. Let p and q be two distributions P. Then,
1. Dw (p. q) implies Dih (p. q) for all finite 31, B2
2. Di
92 "p.0>pppp))))p for some finite 31, 32 > 0 implies Da(p. q) 0.
3. The converse of statements and 2 are false.
In addition to the result presented in Theorem 2.1, we can show that the log-loss discriminator can also "preserve" the existing notions of alignment.
Proposition 4.3. Let f be the optimal discriminator (2r given distributions p and q. Then,
1. (p 0 iff Dw (J" 2D, ) 1 :g) 0:
2. Dia (p. q) 0 iff 8 :0) = 0.
5

---

Published as a conference paper at ICLR 2022
4.2 ALGORITHMIC CONNECTIONS
The result of Proposition 4.3 suggests methods similar to our ASA algorithm presented in Section can achieve different notions of alignment by minimizing objectives discussed in Section 4.1 between the ID pushforward distributions. We consider the setup used in Section 3 but without history buffers to simplify the analysis, as their usage orthogonal to our discussion in this section.
Recall that we work with mini-batch setting, where { J==1 and {a: J'=1 are sampled from p and respectively, and g is the adversarial log-loss discriminator. We denote the corresponding ID outputs from the log-loss discriminator by o" {of { (m and o² (defined similarly).
Distribution alignment. We adapt (7) for {o f==1 and {o/
Dw (0p, .00)
inf yET(o",0") .09) 1m
nrdd( ,9),
(10)
where I' o²) is the set of m x m doubly stochastic matrices. Since o" and of are sets of ID samples with the same size, it can be shown (Rabin et al., 2011) that the optimal >" corresponds to an assignment π², which pairs points in the sorting order and can be computed efficiently by sorting both sets o" and o". The transportation cost zero if and only there exists an invertible 1-to-1 assignment "'" such that of ox (t) GAN training algorithms proposed in (Deshpande et al., 2018; 2019) utilize the above sorting procedure to estimate the maximum sliced Wasserstein distance.
Relaxed distribution alignment. Similarly, we can adapt (8):
Di (00 -0)== inf 1 Σ d=1
(11)
7EIn(o",on ,00) 171
where [a o4) is the set of m x m matrices with non-negative real entries, such that Σj= Yr
1, Vi and Di 76 S V.. The optimization goal in (11) find a "soft-assignment" > which describes the transportation of probability mass from points of in o" to points of in o". The parameter controls the set of admissible assignments rg which is similar its role discussed in Section 4.1: with transportation assignment tolerance 3, the total mass of points in O transported to each of the points of cannot exceed 1 + B. We refer to such assignments as (3 + 1)-to-1 1 assignment. The transportation cost is zero and only if there exists such an assignment between 0" and d"
It can be shown (see Appendix C) that for integer value of 8, the set of minimizers of (11) must contain a ftard-assignmmnt transportation plan, which assigns each point of to exactly one point oz. Then (1 + 3) gives the upper bound on the number of points of that can be transported to given point og. This hard assignment problem can be solved quasi-linearly with worst
time complexity O ((8 + 1)m²) (Bonneel & Coeurjolly, 2019), which, combined with Proposition 4.3, can lead to new algorithms for relaxed distribution alignment besides those proposed in Wu et al. (2019b).
Support alignment. When 3 = co, the sum Σm- Yij is unconstrained for all j. and each point of can be assigned to any of the points of. The optimal solution is simply 1-nearest neighbor assignment, or to follow the above terminology, co-to-1 assignment.
EXPERIMENTS
Problem setting. We evaluate our proposed ASA method in the setting of unsupervised domain adaptation (UDA). The goal of UDA algorithms is to train and "adapt" a classification model M + y from source domain distribution px,y to target domain distribution qx.y given the access to a labeled source dataset {x,, v J ~ Px,y and an unlabeled target dataset (x)) L ~ qx-
common approach for UDA is to represent M as C Fd: classifier C+ 2 + y and a feature extractor Fx : X + Z, and train C and FD by minimizing: 1) classification loss fals source examples; 2) alignment loss Dalign measuring discrepancy between p2z = F :px and g = F pax:
N²
min 6.8
Np Σ
+ X Datign (1F) (F" (29)))))
(12)

---

Published as a conference paper at ICLR 2022
Tanget
dloce '3'
closs '''
close '3'
-18
-as GB 05 11
14
20
25
(a) No DA (avg acc: 63%)
(b) DANN (avg acc: 75%) Dw (pz: 12) 0.07
(c) ASA-abs (avg ace: 94%) Dw (PZ. 92) 0.59
Dw (pg, 92) 0.78
(a 02) = 0.10
Da (2) = 0.02
Da (P(22 92) = 0.03
Figure 2: Visualization of learned 2D embeddings on 3-class USPS~MMNH with label distribution shift. In source domain, all classes have equal probability 3. The target probabilities of classes '3', '5', '9" are (23%, 65%, 12%]. Each panel shows 2 level sets (outer one approximates the support) of the kernel density estimates of embeddings in source (filled regions) and target domains (solid/dashed lines). We report the average class accuracy of the target domain, Dw and Dz. between embeddings.
In practice Dalign is an estimate of a divergence measure via an adversarial discriminator 9². Choices of Palign include I-divergences (Ganin et al., 2016; Nowozin et al., 2016) and Wasserstein distance (Arjovsky et al., 2017) to enforce distribution alignment and versions of re- re-weighted/relaxed distribu- tion divergences (Wu et al., 2019b; Tachet des Combes et al., 2020) to enforce relaxed distribution alignment. For support alignment, we apply the proposed ASA method as the alignment subroutine in (12) with log-loss discriminator gb (4) and Datign computed (6).
Task specifications. We consider 3 UDA tasks: USPS LUSPS-MMNN -MISTT STL.,CIFAR, and VisDA-2017, and 2 versions of ASA: ASA-sq. ASA-abs corresponding to squared and absolute distances respec- tively for d(-, -) in (6). We compare ASA with: No DA (no domain adaptation). DANN (Ganin et al., 2016) (distribution aligmment with JS divergence), VADA (Shu et al., 2018) (distribution alignment with virtual adversarial training), IWDAN, IWCDAN (Tachet des Combes et al., 2020) (relaxed distribution alignment via importance weighting) sDANN-B (Wu al., 2019b) (relaxed/ Sellaeed/Saadmisssiie admissibble IS divergence via re-weighting). Please refer to Appendix D for full experimental details.
To evaluate the robustness of the methods, we simulate label distribution shift by subsampling source and target dataset, so that source has balanced label distribution and target label distribution follows the power law qy (y(y) xx a (yy) where a a random permutation of class labels (I...., K) and a controls the severity of the shift (a = 0 means balanced label distribution). For each task, we generate 5 random permutations a for different shift levels a E {0, 1, 1.3, Essentially we transform each (source, target) dataset pair to 5 x 4 = 20 tasks of different difficulty levels, since classes not equally difficult and different permutations can give them different weights.
Evaluation metrics. We choose the average (per-)class accuracy and minimum (per-)class accuracy on the target test set as evaluation metrics. Under the average class accuracy metric, all classes are treated as equally important (despite the unequal representation during training for a > 0), and the minimum class accuracy focuses on model's worst within-class performance. In order to account for the variability of task difficulties across random permutations of target labels, we report robust statistics, median and 25-75 percentile interval, across 5 runs.
Illustrative example. First we consider a simplified setting to intuitively understand and directly analyze the behavior of our proposed support alignment method in domain adaptation under label distribution shift. We consider a 3-class USPS -MNIST problem by selecting subset of examples corresponding to digits '3', '5' and '9', and use a feature extractor network with 2D output space. We introduce label distribution shift as described above with a = 1.5, i.e. the probabilities of classes in the target domain are 12% 23%, and 65%. We compare No DA, DANN, ASA-abs by their average target classification accuracy, Wasserstein distance Dw (p%, 98) and SSD divergence DA (r& 9%) between the learned embeddings of source and target domain. We apply a global affine transformation to each embedding space in order to have comparable distances between different spaces: we center the embeddings so that their average is O and re-scale them so that their average norm is 1. The
7

---

Published as a conference paper at ICLR 2022
Table Average and minimum class accuracy (%) on USPS~*MNIST with different levels of shifts in label distributions (higher a implies more severe imbalance). We report median (the main number), and 25 (subscript) and 75 (superscript) percentiles across runs.
0.0
a 1.0
1.5
a: 2.0
Algorithm
average
min
average
min
average
min
average
min
No DA
71.9 72-8
20.3 17.8
72.9 0
25.8 81.8
71.3 FT.B
27.5 37.3
71.3 13.0
16.6 26.5
DANN
97.8 97.8
96.0 96.1
84.6 83.5
25.1 36.9
70.0 71-2
01.1 01.5
57.8
60.4
00.9 01
76.7
52.0
VADA
96.0 98.0
97.9
96.2
96.3
95.9
88.2 88.1
48.9
47.8
78.2 70.1
06.6
02.4
61.9
56.3
01.4 00.8
IWDAN
97.5 97.4
95.7
95.7
95.7
92.8
81.3
67.1
86.5
80.3
15.2
04.2
74.4
70.0
07.3
06.3
IWCDAN
98.0 07.9
98.11
96.6
06.4
96.9
96.7
03.33
97.3
85.1
933.0
65-3
91.3
93..
90..5
66.5
64.1
74.5
77.5
82.3
77.8
22.2
02.7
45.4
sDANN-4
87.4 96-7
90-0 05.6
05.6
94.9 94.9
94.7
87-7 85.7
$0.1 86.8
85.5
50-3 21.6
83.1 81.5
81.3
39.3
37.9
56.2
ASA-Sq
93.7 93.33
89.2 89.4
92.3
83.5 80.4
90.9
69.9 $2.0
87 2 89.3
62.5
91.5
KO.6
ASA-abs
94.1 93.8
88.9 87.0
92.8
93.3
89.3
78.9 65.1
92.9 92.5 90.9
82.4 74.5
86.4
90.4 89.2
90.7
68.4 67.5
73.0
Table 2: Results on CIFRR Same setup and reporting metrics Table 1.
0.0
1.0
1.5
2.0 min
Algorithm
average
min
average
min
average
min
average
No DA
69.9 69.8
49.8
68.8 68.3
47.2 18.3
66.8 67-3
46.0 45.8
65.8 64.8
43.7 41.6
45.3
DANN
75.3 74.9
54.6
$4.2
69.9
68.6
44.8 45.1
64.9
87-1
34.9
33.9
63.3
57.4
27.0 28.5
VADA
76.7 76.6
76.7
56.9
53.5
68.3
71.0 70.6
70.0
47.7
44.0
48
66.1 65.4
66.5
35.7
33.3
39.3
64.7 63.2
60.2
25.5
25.2
28
IWDAN
69.9 69.9
50.5
50.6
47.9
68.7
68.8
69.
45.8
50.5
34.8
67.1
85.9
44.7
40.1
44.8
64.4
63.6
64.9
36.8
37.9
34.5
IWCDAN
70.1 10.1
47.8
49.3
69.4
69.4
47.1
51.3
66.1
85.0
39.9
40.5
64.5 65.1
37.0 10
SDANN-4
71.8 F2.A
52.1 $2.4
71.1
70.4
71.7
49.9
48.1
69.4 4 68.7
48.6
49.0
43.5
66. 4 66.2
39.0
33.6
ASA-Sq
71.7 Z1:9
52.9
46.7
70.7
71.0
70.4
51.6
46.8
52.7
69.2 2
45.6
43.3
68.1
68.5
67
44.7
39.8
45.9
ASA-abs
71.6 71:2
49.0 48.4
$3.5
70.9
71.0
70.5
19.2 47.3
50.0
69.6
69.6
69.3
43.2 12.1
49.5
67.8
68.3
66.6
10.9
35.4
49.0
results are shown in Figure and Table D.5. Compared No DA, both DANN and ASA achieve support aligmment. DANN enforces distribution alignment, and thus places some target embeddings into regions corresponding to the wrong class. In comparison, ASA does not enforce distribution alignment and maintains good class correspondence across the source and target embeddings.
Main results. The results of the main experimental evaluations are shown in Tables 2, 3. Without any alignment, source only training struggles relatively to adapt to the target domain. Nonetheless, its performance across the imbalance levels remains robust, since the training procedure the same. Agreeing with the observation and theoretical results from previous work (Zhao al., 2019; Li et al., 2020; Tan et al., 2020; Wu et al., 2019b; Tachet des Combes et al., 2020), distribution alignment methods (DANN and VADA) perform well when there is shift but suffer otherwise, whereas relaxed distribution alignment methods (IWDAN, IWCDAN and sDANN-3) show more resilience to shifts. On all tasks with positive a, we observe that is common for the existing methods to achieve good class average accuracies while suffering significantly on some individual classes. These results suggest that the often-ignored but important min-accuracy metric our support alignment methods (ASA-sq and ASA-abs) are the most robust ones against the shifts, while still being competitive in the more balanced settings (a in the more imbalanced and difficult tasks (a = 1.5 or 2) for almost all categories on all datasets. Please refer to Appendix D for ablation studies and additional comparisons.
be very challenging. Finally,
1). We achieve best results
6 RELATED WORK
Distribution alignment. Apart from the works, e.g. (Ajakan et al., 2014: Ganin et al., 2016; Ganin & Lempitsky, 2015; Pei et al., 2018; Zhao et al., 2018; Long et al., 2018; Tachet des Combes et al.,
8

---

Published as a conference paper at ICLR 2022
Table 3: Results on VisDA17. Same setup and reporting metrics Table 1.
0.0
0 1.0
1.5
a 2.0
Algorithm
average
min
average
min
average
min
average
min
No DA
49.5 50-5
22.2 22.2
50.2 40.2
21.2 21.7
47.1 47-6
18.6 23-2
45.3 45.2
19.5 14.4
DANN
75.4 76.2
36.7 40-9
64.1 65-1
25.0 20.3
52.1
52-3
11.5
12-4
43.1
44.3
03.6 14:3
39.1
VADA
75.3 74.8
40.5 30.7
65.1 64.6 61.2
22.8 21.7
53.0
$1.6
54.2
14.8 13.7
43.9
40.9
08.5 05.0
IWDAN
73.2
72.0
31.7
22.8
64.4 61.1 61.1
12.1 05.0
51.3
51.0
04.6
02.1
45.1 41.7
04.6 01.2
IWCDAN
71.6
25.2
70.6
27.6 28-0
60.6 61.0
60-2
01.1 11
49.7 61.9
02.2
95-1
38.3 46.2
97.3
00.6 01.7
sDANN-4
72.4 T3-3
37.8 32.3
68.4 66.2
29-4 26.6
57.2
56.8
23.9 18.6
50.7 51-7
18.6 20.0
ASA-Sq
64.9
65.0
63.7
35.7 32.1
61.8
60.8
31.4
31.4
20.4
57.8
56.3
55.5
26.7
32.1
17.3
51.9
50.8
18.3 21
ASA-abs
64.8 64.5
65.0
40.6 36.0
41.9
62.0 60.5
27.3 16.7
29.7
57.1
58.4
56.2
26.0
13.9
52.5
51.9
56.6
19.7
22.2
17.7
2020; Li et al., 2018b; Tzeng et al., 2017: Shen et al., 2018; Kumar et al.. 2018; Li et al., 2018a;
Wang et al., 2021: Goodfellow et al., 2014; Arjovsky et al., 2017: Gulrajani et al., 2017; Mao et al., 2017; Radford et al., 2015; Salimans et al., 2018; Genevay et al., 201 8; Wu et al., 2019a; Deshpande et al., 2018; 2019), that do distribution alignment, there are also papers (Long et al., 2015; 2017; Peng et al., 2019; Sun et al., 2016; Sun & Saenko, 2016) focusing on aligning some characteristics of the distribution, such as first or second moments. Our work is concerned with different problem. support alignment, which is novel objective in this line of work. In terms of methodology, our of the discriminator output space to work with easier optimization in ID is inspired by line of work (Salimans et al., 2018; Genevay et al., 2018; Wu et al., 201 9a; Deshpande al., 2018 8; 2019) sliced Wasserstein distance based models. Our result in Proposition 4.3 also provides theoretical insight on the practical effectiveness of ID OT in (Deshpande al., 2019).
Relaxed distribution alignment. In Section 4, we have already covered in detail the connections between our work and (Wu et al., 2019b). Balaji al. (2020) introduced relaxed distribution alignment with a different focus, aiming to be insensitive to outliers. Chamfer distance/divergencce (CD) is used compute similarity between images/3D point clouds (Fan et al., 2017: Nguyen et al., 2021). For text data, Kusner et al. (2015) presented Relaxed Word Mover's Distance (RWMD) to prune candidates of similar documents. CD and RWMD are essentially the same as (5) with d(...) being the Fuclidean distance. They are computed by finding the nearest neighbor assignments. Our subroutine of calculating the support distance in the ID discriminator output space is done similarly by finding nearest neighbors within the current batch and history buffers.
Support estimation. There exists a series of work, e.g. (Schölkopf et al., 2001; Hoffmann, 2007; Tax & Duin, 2004; Knorr et al., 2000; Chalapathy et al., 2017; Ruff et al., 2018; Perera et al., 2019; Deecke et al., 2018; Zenati et al., 2018), on novelty/anomaly detection problem, which can be casted support estimation. We consider fundamentally different problem setting. Our goal is to align the supports and our approach does not directly estimate the supports. Instead, we implicitly learn the relationships between supports (density ratio to be specific) via discriminator.
CONCLUSION AND FUTURE WORK
In this paper, we studied the problem of aligning the supports of distributions. We formalized its theoretical connections with existing alignment notions and demonstrated the effectiveness of the approach in domain adaptation. We believe that our methodology opens possibilities for the design of more nuanced and structured alignment constraints, suitable for various use cases. One natural extension is support containment, achievable with only one term in (1). This approach is fitting for partial domain adaptation, where some source domain classes do not appear in the target domain. Another interesting direction is unsupervised domain transfer, where support aligmment more desired than existing distribution alignment methods due mode imbalance (Binkowski et al., 2019).

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
The computational experiments presented in this paper were performed on "Satori" cluster developed as collaboration between MIT and IBM. We used Weights & Biases (Biewald, 2020) for experiment tracking and visualizations to develop insights for this paper.
TJ acknowledges support from MIT-IBM Watson AI Lab, from Singapore DSO, and MIT-DSTA Singapore collaboration. We thank Xiang Fu and all anonymous reviewers for their helpful comments regarding the paper's s writing and presentation.
REFERENCES
Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, and Mario Marchand. Domain-
adversarial neural networks. arXiv preprint arXiv: 14/2.4446 2014. 8
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214-223. PMLR, 2017. 3. 7.9
Yogesh Balaji, Rama Chellappa, and Soheil Feizi. Robust optimal transport with applications in generative modeling and domain adaptation. Advances in Neural Information Processing Systems Foundation (NeurIPS), 2020. 9
Shai Ben-David, John Blitzer, Koby Crammer, Fernando Pereira, et al. Analysis of representations for domain adaptation. Advances in neural information processing systems, 19:137, 2007.
Shai Ben-David. John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning. 79(1)1151-175, 2010.
Dimitris Bertsimas and John N Tsitsiklis. Introduction to linear optimization, volume 6. Athena Scientific Belmont, MA, 1997. 21
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL httph::///wwwwwwwt wandb. com/. Software available from wandb.com.
Mikolaj Binkowski, Devon Hjelm, and Aaron Courville. Batch weight for domain adaptation with mass shift. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1844-1853, 2019.
Nicolas Bonneel and David Coeurjolly. Spot: sliced partial optimal transport. ACM Transactions Graphics (TOG), 38( 1):1-13, 2019.
Eric Budish, Yeon-Koo Che, Fuhito Kojima, and Paul Milgrom. Implementing random assignments: A generalization of the hirkhoff-von neumann theorem. In 2009 Cowles Summer Conference, 2009.
21
Raghavendra Chalapathy, Aditya Krishna Menon, and Sanjay Chawla. Robust, deep and inductive anomaly detection. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 36-51. Springer, 2017.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215-223. JMLR Workshop and Conference Proceedings, 2011. 22
Lucas Deecke, Robert Vandermeulen, Lukas Ruff, Stephan Mandt, and Marius Kloft. Image anomaly
detection with generative adversarial networks. In Joint european conference on machine learning and knowledge discovery in databases, pp. 3-17. Springer, 2018. 9
Ishan Deshpande, Ziyu Zhang, and Alexander G Schwing. Generative modeling using the sliced wasserstein distance. In Proceedings of the IEEE conference on computer vision and pattern
recognition, pp. 3483-3491, 2018. 6, 9, 20
Ishan Deshpande, Vuan-Ting Hu, Ruoyu Sun, Ayis Pyrros, Nasir Siddiqui, Sanmi Koyejo, Zhizhen Zhao, David Forsyth, and Alexander G Schwing. Max-sliced wasserstein distance and its use for gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10648-10658 2019. 6, 20, 23
10

---

Published as a conference paper at ICLR 2022
Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 605-613, 2017. 3,9
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pp. 1180-1189. PMLR, 2015. 1,8 8
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. aovaariadversariall training of neural networks. The journal of machine learning research, 0:2096-20030 2016. 1, 7,
Aude Genevay, Gabriel Peyré, and Marco Cuturi. Learning generative models with sinkhorn di- vergences. In International Conference Artificial Intelligence and Statistics, pp. 1608-1617.
PMLR, 2018. 9
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint 1406,2611, 2014. 3,4,9
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of wasserstein gans. In Proceedings of the 31 International Conference Neural Information Processing Systems, pp. 5769-5777, 2017. 3, 9, 23
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016. 25
Heiko Hoffmann. Kernel pea for novelty detection. Pattern recognition, 40(3):863-874 2007. 9
Jonathan J. Hull. A database for handwritten text recognition research. IEEE Transactions on pattern analysis and machine intelligence, 16(5):550-554, 1994. 22
Fredrik D Johansson, David Sontag, and Rajesh Ranganath. Support and invertibility in domain- invariant representations. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 527-536. PMLR, 2019. 1
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1rXiv:14/2.6900 2014. 22
Edwin M Knorr, Raymond Tgg, and Vladimir Tucakov. Distance- based outliers: algorithms and applications. The VLDB Journal, 8(3):237-253, 2000.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009. 22
Abhishek Kumar, Prasanna Sattigeri, Kahini Wadhawan, Leonid Karlinsky, Rogerio Feris, Bill
Freeman, and Gregory Wornell. Co-regularized alignment for unsupervised domain adaptation. Advances in Neural Information Processing Systems, 31 31:9345-9355 2018. 1,9
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. From word embeddings to document distances. In International conference on machine learning, pp. 957-966. PMLR, 2015. 3, 9
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2 2324, 1998. 22
Bo Li, Yezhen Wang. Tong Che, Shanghang Zhang, Sicheng Zhao, Pengfei Xu, Wei Zhou, Yoshua Bengio, and Kurt Keutzer. Rethinking distributional matching based domain adaptation. arXiv preprint arXiv:2006. 13352, 2020. 1,
Haoliang Li, Sinno Jialin Pan, Shiqi Wang. and Alex C Kot. Domain generalization with adver- sarial feature learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5400-5409, 201 8a. 1,9
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 624 639, 2018b.
11

---

Published as a conference paper at ICLR 2022
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pp. 97-105. PMLR, 2015. 9
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. Deep transfer learning with joint adaptation networks. In International conference on machine learming, pp. 2208-2217. PMLR, 2017. 9
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, pp. 1640-1650, 2018.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang. and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp.
2017. 9
Trung Nguyen, Quang-Hiew Pham, Tam Le, Tung Pham, Nhat Ho, and Binh-Son Hua. Point-set distances for learning representations of 3d point clouds. arXiv preprint arXiv:2102.04014, 2021. 3,
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Proceedings of the 30th International Conference
Neural Information Processing Systems, pp. 271-279, 2016. 7
Zhongyi Pei, Zhangjie Cao, Mingsheng Long, and Jianmin Wang. Multi-adverssrial domain adap- tation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. 1,
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge, 2017. 25
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1406-1415, 2019. 9
Pramuditha Perera, Ramesh Nallapati, and Bing Xiang. Oegan: One-class novelty detection using gans with constrained latent representations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2898-2906, 2019. 9
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends in Machine Learming. 11(5-6)33556077, 2019. 21
Julien Rabin, Gabriel Peyré, Julie Delon, and Mare Bernot. Wasserstein barycenter and its application to texture mixing. In International Conference on Scale Space and Variational Methods in
Computer Vision, pp. 435-446. Springer, 2011. 6, 20
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint (.XVv:1511.064344 2015. 9
Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In International conference on machine learning, pp. 4393 4402. PMLR, 2018. 9
Tim Salimans, Han Zhang, Alec Radford, and Dimitris Metaxas. Improving GANs using optimal transport. In International Conference on Learning Representations, 2018. URL https://
openreview. nwt fortm//orumme wtt.foommmmndnunnaaaa
Bernhard Schölkopf, John C Platt, John Shawe-Taylor, Alex J Smola, and Robert C Williamson. Estimating the support of digh-dimensional distribution. Neural computation, 13(7):1443-1411,
2001. 9
Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. 1,9
12

---

Published as a conference paper at ICLR 2022
Rui Shu, Hung Bui, Hirokazu Narui, and Stefano Ermon. A DIRT-t approach to unsupervised
domain adaptation. In International Conference on Learning Representations, 2018. URL
https: openreview .et. Eor-m?2---HHll-----M 7, 22, 24
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In
European conference on computer vision, pp. 443-450. Springer, 2016.
Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016. 9
Remi Tachet des Combes, Han Zhao, Yu-Xiang Wang, and Geoffrey Gordon. Domain adaptation
with conditional distribution matching and generalized label shift. Advances in Neural Information Processing Systems, 33, 2020. 1, 7, 8, 22
Shuhan Tan, Xingchao Peng, and Kate Saenko. Class-imbalanced domain adaptation: An empirical odyssey. In European Conference on Computer Vision, pp. 585-602. Springer, 2020.
David MJ Tax and Robert PW Duin. Support vector data description. Machine learning. 54(1):45-66, 2004. 9
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7167-7176, 2017. 1,9
Jing Wang, Jiahong Chen, Jianzhe Lin, Leonid Sigal, and Clarence W de Silva. Discriminative feature alignment: Improving transferability of unsupervised domain adaptation by gaussian-guided latent alignment. Pattern Recognition, 116:107943, 2021.
Jiqing Wu, Zhiwu Huang, Dinesh Acharya, Wen Li, Janine Thoma, Danda Pani Paudel, and Luc Van Gool. Sliced wasserstein generative models. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), June 2019a. 9, 20
Yifan Wu, Ezra Winston, Divyansh Kaushik, and Zachary Lipton. Domain adaptation with asymmetriialll--relxxdd distribution alignment. In International Conference Machine Learning, pp. 6872-6881. PMLR, 2019b. 5, 7, 9,
Houssam Zenati, Manon Romain, Bhuan-Sheng Foo, Bruno Lecouat, and Vijay Chandrasekhar, Adversarially learned anomaly detection. In 2018 IEEE International conference on data mining (ICDM), pp. 727-736. IEEE, 2018. 9
Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura, Joao P Costeira, and Geoffrey J Gordon. Adversarial multiple source domain adaptation. In Advances in Neural Information Processing Systems, pp. 8568 8579, 2018. 1,8
Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. In International Conference on Machine Learning, pp.
7523-7532. PMLR, 2019. 1,8
13

---

Published as a conference paper at ICLR 2022
A PROOFS OF THE THEORETICAL RESULTS
A.1 PROOF OF PROPOSITION 2.1
Da (p. () 2 for p.q E P:
Since d(-,.) :) 2 0, for all p.q.
SD(p, q) == Ez-pl Ex-pld(z, supp(a)) Enop
inf (r-supp(a)
d(x, u) 0,
(13)
which makes SDp 9) SD(g.)) 0.
2) (p(p.) = 0f and only if supp(p) supp(g):
With statement 1, (2 q) = 0 and only if SD q) and SD(,.p) = O.
Then,
SD(p. n) = = Pd = = ((u)))(),suup(()) > 0}) 0.
This is equivalent to
Ye supp(p). d(z,supp(()) 0.
Thus, supp(p) S supp(g), and similarly, supp(g) S supp(p), which makes supp(p) supp(g)-
A.2 ASSUMPTION AND PROOF THEOREM 2.1
A.2.1 COMMENTS ON ASSUMPTION (3)
Assumption (3) is restrictive. Indeed, distributions satisfying Assumption (3) include:
uniform p(z) Ux; b));
truncated normal;
p(z) of the form
p(x) = Zp e Ep(w), x E supp(p);
x # supp(p))
with non-negative energy (unnormalized log-density) function Ep : X + [0, oo);
mixture of any distributions satisfying Assumption (3), for instance the distributions shown in Figure A.1 top-left are mixtures of truncated normal distributions on 2].
Starting from arbitrary density Po(z)) with bounded support can derive density p(z) satisfying Assumption (3) via density clipping and re-normalization
p()) αc clip (po(x), L (..))))
for some C' > 1.
A.2.2 PROOF OF THEOREM 2.1
First, we show that DA (p. q) = 0 implies (ff
pP, fP. = 0.
(p.p. q) = 0 implies (p) = supp(g). Then for any mapping f : X + R, we have supp((fp) = supp(fig), which implies supp(ff :P) supp(s* :g). Thus, DA (f" pP, f ;g) 0.
Now, we prove that DD (J" sP. f" gg)== implies DA (p, q) by contradiction.
14

---

Published as a conference paper at ICLR 2022
Dz(f" ;P, FP. 0 implies the following:
,,~,"; [d(t,supp(f) {q)] 0, Ee-f's" ;9 |d(t, supp(f =P]] 0.
1) Suppose En-pld(x; supp())) > O. This is only possible if p(fx) x E supp(p) \ supp(a))) > 0.
Since x E supp(p) supp(1) implies p(a) > 0, ((x) = O, and for any x supp(p) supp(g).
p(x) > 0, (x)) = 0 if and only if (x) = p())+(() 1, we have:
p(a)
((1)) = - Pp(( x E supp(p) \ supp(a))) > 0,
and therefore 1 E supp(f';p)
For real number o 00:0< < < FT consider the probability of the event (1 - a, 1] C [0, 1] under distribution f" f":9:
P p ((1 a, 1) = PP E az 1]]).
By assumption (3),p(r) p(x) < C and g(x) > 0 implies ((x)) j. therefore for x g(x)) 0 have
x
p(z)
C
1
f(x) p(z) + ((x) p
< <1 a
C+
c²+1
This means that fr,, ((1 a, 1)) 0, i.c. supp(f" :9) (1 a. 1| 2.
To summarize, starting from the assumption that (tcppl((r,supp(() we showed that
E supp(f" ;p). pp())))()))))))))))))))))))))( > 0;
supp(/";a) 0 a, 1 2.
P, =9) E sp Suttssupp[[" :q))] (()) d(1, (0))) > Because D2(l" f
(f())(11))) a > 0, which contradicts with the given "P 0, we have Ezpp [d(,,, supp(a)) = 0.
2) Similarly, it can be shown Thus, (f :P, :g) = 0 implies (2, q) = O.
= 0.
A.3 PROOF OF PROPOSITION 2.2
Consider I-dimensional Euclidean space R. Let supp(p) = - with p [ 11
and p[[1, 2)) 7. Let supp(1) = [-2, -1 [1, [ 2] with 2(122.11) 1. a(1 2
1.
and q(1,2)) 2 2- The supports of p and q consist of disjoint closed intervals, and we assume uniform
distribution within each of these intervals, i.e. p has density p(x) 3. Va E : (1:(()) = 4. Vx
[1,2] and has densitiy g(z) 4 Vr E [-2, (1] = 1. Va E [ [ ()((()) = 2. E [1,2]. Clearly, supp(p) # supp(g)-
The optimal dual Wasserstein discriminator fiv the maximizer of Bx~p[[(x)] - [5
sup f:Lip(N)1
Thus, füy is the maximizer of
sup 1 {(3 3 { f(z)dx+ + { f(0)do- f(x)dr- - ( / f(0)dx - S. slo)az)
( 4 which simplifies to
sup 1 4 AL (-55 f(x)dx+ +2 2 S² Ma)do - S. S(o)aa)
Since the optimization objective and the constraint are invariant to replacing the function fx) with its symmetric reflection g(x) f(-x), if f' is optimal solution, then there exists symmetric
15

---

Published as a conference paper at ICLR 2022
maximizer (z) (z) (-x) since für (2) = (-x) and Lip(fi) & Lip(f") $ 1. Thus, supp(firp? = supp(fi;;9 as (w) = (-x) for [1.
Note that one can easily "extend" the above proof to discrete distributions, by replacing the disjoint segments [-2, - -11, 1 2 2 [1, 2] with points {-1}, {0), {1}.
A.4 PROOF OF PROPOSITION 4.1
From (8), we have
(p. = lim (w.(,.) lim 9-000 yEl's(pa))
inf ((an))>ld(a v)l.
8-x0c
where lim T (p, 9) is the set of all measures 7 on X' x A' such that 7(1, v)dy = p(z), Vo and Ta y)dr v lim + B)g(s), Vy.
The set of inequalities
/ 11 W)d [ax $
< lim (1 + B)g(v)
Vy
can be simplified to
/ (x,,))xx= O.
Vy such that ((v) 0.
To put it together, we have
Dw (p.
inf (((,)) (p.2)
where 1'o0 (p. is the set of all measures 7 X X' such that [1,(2, y)dy p(x), Va and I (ax, y)dz = 0, Vy such that q(v) = 0. In other words, we seek the coupling (z, y) which defines a transportation plan such that total mass transported from given point x is equal to p(x)), and only constraint on the destination points is that no probability mass can be transported to points where q(y) = 0, i.. y & supp(q).
Let y* (I) denote a function such that y" (x) E supp(g), Vo; d(x,y = inf d(r, y)-
yEsupp())
We can see that 7 given by
the optimal coupling. Indeed, 7"h satisfies the constraints 7"' To.. and the cost of any other transportation cost 7 E Poo is at least that of yrh (since y (x) is defined as closest point in supp(q) given point x).
Thus.
d(r, v)
(p.q))
inf B(a,))~/dd(x y)] E(aa))y [d(x, y)] Expp inf ((Euppp(g)
7el' (p.g)
The last equation implies (p. SD(p, 9) (SD(..) defined in (13)). Then, Dio. (p.9)= = lim
02 (p,4) = Di (p. (a () = DD D
P1002-000
A.5 PROOF OF PROPOSITION 4.2
1. g) 0 implies q. which is equivalent
p(r)
Vr supp(p) U supp(g).
q(z)
Then clearly, for all finite 31, 32 > 0 it satisfies
p(x)
< + 31,
Va E supp(p) supp(()).
(14)
1 32
(x))
Thus, D 82 (p. 0 for all finite
31, 82 > 0.
16

---

Published as a conference paper at ICLR 2022
2. " 9) 0 for some finite 31, B2 > 0 means that (14) is satisfied. This implies that Ve E supp(p), 2 E supp(g) and Vx E supp(g), x E supp(p), which makes supp(p) = supp(g). Thus, DA (p. 0.
3. The converse of statements and 2 are false:
(a) For all finite 31, 92 > 0, let supp(p) = supp(() = f x2). Let p(x1) = p(a2)
1 1/2 and q(x)) + /2 and ( = (1-18)/2 where
1
8' min $2, 1 I 31
Then, it can be easily checked that (14) is satisfied, which makes Di (p.9) = 0. However, since 3' + 0, p * q and thus Dw (v.4) 7) + O.
(b) Similar to (a), let supp(p) supp(q) = (x1, x2). Let p(x1) = ((x2) = = and p(z2) ((x1) However,
for some 0. Since supp(p) supp(g). DA (p.9) 9) = 0.
lim p(z1) lim e40 g(21))
0,
e10
1 8
and, thus, for any finite 32 > we can choose e > 0 such that
p(an)
((<1)
1 1 1 B2 D..00 B2 (p, 2) #0.
Therefore, (14) is not satisfied and DWy
A.6 PROOF OF PROPOSITION 4.3
Using (2), we first establish connection between the pushforward distributions f* :p and f 19.
Proposition A.I. Let f be the optimal log-loss discriminator (2) between p and q. Then,
[[* (1) [f*)pl(t) [f =t, At E supp(f (P) supp(f" ;9).
(15)
Proof. For any point E supp(s* :p) Usupp([" ;4), the values of the densities
(fx - < <x)< <(+
( p(z) da
["'p((()= = lim
lim
e!u
28
e10
28
P. (1z 1t 1- (z)
lim (())=ee ((x) dr
[()
lim
=40
<40
28
Note that for all x - < (x) < t + 8 we have
p(z)
- 8 < p(x) + ((x)
< <t+e,
which implies
(t (t-:)(((z) + ((x)) < p(z) < (t (t+ (t+))((z)) + +(x)).
Since these inequalities hold for all x :l - (x) ++ e, the similar relationship holds for the integrals:
(t =)= J
(p(z) + q(x)) dx
(0) p(x) do <
(1 (1+ /
[ (p(x) + g(x)) dr.
17

---

Published as a conference paper at ICLR 2022
The ratio [f" (0)==((tt((()))))))))))))))))))( sp((t) + [f (t)) can be expressed
(t) If =P((()) + If (g](t)
lim J p(z) dr
240 [tol-eerr (p(x) + ((x)) dx
Using the inequality above we observe that
(entess//()))te))(()))) P(x) dr
< t tt,
(ell-eerr( (p(x) q(x)) dx
for & O, and taking the limit + we obtain
(()
[f =1())( (f*x)))))))
x(z)
0.4
v(x))
Pusiifersart dessity
Dr 1000
Deneity
0.2
( s0
0.3
0.01
$1.0
2.0
1.5
1.0
-05
0.0
0.5
1.0
2.0
0.0
0.1
62
0.3
04
0.5
0.6
0.7
0.8
0.9
1.0
1.0
0.9
(*(x)
0.8
1.
((x))
42 ((a)
0.3
0.3
0.2
35
0.0
-20
-1.5
-1.0
-0.
0.0
0.5
1.0
1.5
2.0
Figure A.1: Visual illustration of the statement of Proposition A.I. The top-left panel shows two example PDFs p(z), 4(x) on closed interval []. The bottom-left panel shows the optimal discriminator function (*) pxx)/(p(x)) + {(x)) as a function of on [-2,2]. The top-right panel shows the PDFs [ (t). (f) of the pushforward distributions f'P f* induced by
the discriminator mapping 1 maps [- to [0 and [/] [f","] *9]] are defined on [0. 1].
Consider point x1 E 2) The value (*1x)) characterizes the ratio of densities p(x1)/(p(z))
((x1)) at 21. For another point x2 mapped to the same value f (xz) f*(x1) 11,2, the ratio of densities (p(()) + ((x2)) is the same as ((z1))(p((e)) + ((x1)). All points mapped to 11 share the same ratio of the densities p(x)/(((p))) + q(x)). This fact implies that the ratio of the pushforward densities [f (1)((1,2)) ( + ((1,)) 6122 must be the same as the ratio of densities p(æ1). ((=1)/(p(z1) (p(x1) q(x1)) = f12 at 21 (or 22). The pushforward PDFs [f" (t). [f" g9((t) satisfy property [f" (P)(( =t) + vf*"va(t)) = for all t supp(f" :P) Usupp(f" :9).
Comment. Intuitively this proposition states the following. If for some x E X we have fr) =
[0, 1). t directly corresponds the ratio of densities not only in the original space i = ((x)((p(r))
q(x)), but also in the ID discriminator output space = [f ([(())(((* (P((t) + [f", gl()))
We also provide an intuitive example in Figure A.
Proof of Proposition& 4.3, statement #1.
If Dw q) 0 then p q, then f :P 1" g-- Thus, Dw (J" =P, "0) 0.
18

---

Published as a conference paper at ICLR 2022
=: D (frfffpPPp p. (9) = O, then f0 f=;9.
Consider probability of event { 1 > } under distribution f" pP..
F/'p (1)1>> 2) - fu(r())) 1
p(x) dz,
where I[-] is the indicator function (I[c) equal to when the condition c satisfied, and equal to p(a) and p(x) + g(x) > 0. Therefore, otherwise). For all x p(x) > O, we have that f* p(x)-9(2) the expression above can be re- written as
pr (111125) - -f
p(z) p(x)+ [ (()) 2
1 p(x) do /Ip((s) - ((x) 0p((1) da.
Similarly, the probability of event
{t t > } under distribution f= is
(f0) 1> ) [TW(s) - ((0) > 0/q(2)) dr.
IP prr
f" xP= f" implies that
pr (fe/ee t> 11 = pp (fr/e> 1)
or equivalently
/ (1p(x) - q(x) > 0)(p()) q(x))) de = O.
Note, that the function I[p(æ) - q(x) (f((2)) (())) is non-negative for any x. This means that the integral can be zero only if the function is zero everywhere implying that for any x either [[p(x) (xx) > 0] = or p(x) ((x) In other words.
p(x) S ((x), v..
Using the fact the both densities p(x) and (zx) must sum up 1, we conclude that p q and Dw (p,4) 4) = O.
Proof of Proposition 4.3, statement #2.
Note that by (2) and (15), we have
f(xx== p(z) + ((x) =t= (() (ft + (f)))))))*
p(x)
Vr supp(p) U supp(g).
=: Suppose 02 "2 0. Then supp(p) = supp(g) = S (by Proposition 4.2) and
W (p,4)
supp(f" :p) = supp(f ;9) T by (Theorem 2.1). Moreover, D (p. g) implies
1
p(x)
<
< <1 Br, ((x)
E.
1 B2
Since
p(x)
p(x)
(*) p(x) + q(x))
9(2)
Va E S,
p(x)
((x)
the inequalities above are equivalent to
1 2 3
1 B
(z)
<
Ya E
+ B
19

---

Published as a conference paper at ICLR 2022
Combined with Proposition A.1, the above implies that (f))
1+ 1
te,,
2+ 32
(f ( ((((t)
(()(t))
+ B1'
equivalently
[tt)
gg(( $ B1,
Vt ET
1 1+ 3/
Therefore, Di Pr (f';,, f4g)) =0
similarly, when 00 (S" ;P, =0] = -0,
supp(f";P) = sf ;9) =T
supp(p) supp(g) S.
Moreover,
(tt
11+ B1,
W
1 32
[())
1
(())((())
1+ 31 2 32
S
S
E
2 + B2
[ft:p(t) + ggt
$
1 31
ff S
Vo S.
2 82
2 31
1
< ((x)
< 1 B1,
Va E S.
1 B2
((x)
Therefore, Di 82 (p.9)) =0
B COMMENT ON "SLICED" SSD DIVERGENCE
Recent works (Deshpande al., 2018: 2019) have proposed perform optimal transport (OT)-based distribution alignment by reducing the OT problem (7) in the original, potentially high-dimensional space, to that between iD distributions. Specifically, Deshpande et al. (2018) consider the sliced Wasserstein distance (Rabin et al., 2011):
{ DD zP, f 29) d8, -g--1
Dsw (p. 9)
(16)
where gh-1 (0- R" |0 = 1} is unit sphere in R" and 1 ID linear projection (x) (0,z). It is known that Dsw is a valid distribution divergence: for any & q there exists linear slicing function f 0 E S--1 which identifies difference in the distributions, i.c. f "Pp * f #g CCalmér-Wold theorem). By reducing the original OT problem to that in ID space, Wu et al. (2019a) and Deshpande et al. (2019) develop efficient practical methods for distribution alignment based on fast algorithms for the 1D OT problem.
Unfortunately, the straight- forward extension of SSD divergence (1) to 1D linearly sliced version does not provide valid support divergence.
Proposition B.1. There exist two distributions p and q P, such that supp(p) * supp(g) but supp(f" :)) = supp(f" :g), (z) = (0, x)) with 0 E 3n-1
Proof. Consider 2-dimensional Euclidean space R² and let supp(p) {(x, w) + 3² 2} and supp(g) = ((r, y) S x²² +2² < 2}. Then, f (x) = (0, x)) with E S! su +p) s p4) = |- 2].
This counterexample is shown Figure B.
20

---

Published as a conference paper at ICLR 2022
p(x)
((x)
2
2
1
0
0
-2
2
-2
2
Figure B.1: Visualization of example distributions for Proposition B.1
C DISCUSSION OF "SOFT" AND "HARD" ASSIGNMENTS WITH ID DISCRETE DISTRIBUTIONS
In Section 4.2 considered the "soft-assignment" relaxed OT problem (11) and claimed that for integer 3, the set of minimizers (11) must contain a "hard-assignment" transportation plan, meaning Tiy {0, 1}, Vi.j. Below we justify this claim.
Note that for = the OT problem (11) is the standard OT problem for Wasserstein- I distance (10), since the inequality constraints Σ - Vij S Vjj can only be satisfied equalities. For this problem, is known (e.g. see Peyré et al. (2019) Proposition 2.1) that the set of optimal "softtassignmmnt" contains "hard-assignment" represented by normalized permutation matrix. This fact can be proven using the Birkhoff-von Neumann theorem. The Birkhoff-von Neumann theorem states that the set of doubly stochastic matrices
P PER RaX":
Psj 2.Vi,
P Pay=11
j=1
is exactly the set of all finite convex combinations of permutation matrices. In the context of the linear program (11) with / = O, the Birkhoff'vvon Neumann theorem means that all extreme points of the polyhedron (o .0) hard-assignment matrices. Therefore, by the fundamental theorem of linear programming (Bertsimas & Tsitsiklis, 1997), the minimum of the objective is reached at "hard-assae assiinmes"mmm matrix.
We argue that a similar result holds for the case of integer 3 > O. In this case, the matrices in [s ,o)) can not be associated with the doubly stochastic matrices, since constraints on of the marginals of are relaxed to inequality constraints. Because of that, the Birkhoff- -on Neumann the- orem can not be applied. However, Budish al. (2009) provide a generalization Neumann theorem (Theorem 1 in (Budish et al., 2009)) which applies to the cases where the equality constraints are replaced with integer-valued inequality constraints (recall that we consider integer B). Using this generalized result, our claim can be proven by performing the following steps.
the Birkhoff von
Clearly, the polyhedron (o of) contains all aasund-esigmmen"" matrices and all their finite convex combinations. The result proven in (Budish et al., 2009) implics that each element of I (o of)) can be represented as a finite convex combination of hhard-assignment matrices. Thus, the polyhedron I [o( o²) is exactly the set of all finite convex combinations of "hard-assignment" matrices and all extreme points of the polyhedron are "hard-assignment" matrices. Finally, by analogy with the case of O, invoke the fundamental theorem of the linear programming and conclude that the minimum of the objective (11) is reached at corresponding to "hard-assignment" matrix.
21

---

Published as a conference paper at ICLR 2022
D EXPERIMENT DETAILS
D.1 USPS TO MNIST EXPERIMENT SPECIFICATIONS
We USPS (Hull, 1994) and MNIST (LeCun al., 1998) datasets for this adaptation problem.
Following Tachet des Combes et al. (2020) we use LeNet-like (LeCun et al., 1998) architecture for the feature extractor with the 500-dimensional feature representation. The classifier consists of single linear layer. The discriminator is implemented by 3-layer MLP with 512 hidden units and leaky-ReLU activation.
We train all methods for 65 000 steps with batch size 64. We train the feature extractor, the classifier, and the discriminator with SGD (learning rate 0.02, momentum 0.9, weight decay 5 10-4). We perform a single discriminator update per update of the feature extractor and the classifier. After the first 30 000 steps we linearly anneal the feature extractor's and classifier's learning rates for 000 steps to the final value 2 10
The feature extractor's loss is given by weighted combination of the cross-entropy classification loss on the labeled source example and domain alignment loss computed from the discriminator's signal (recall that different method use different forms of the alignment loss). The weight for the classification term is constant and set to Adts = 1. We introduce schedule for the alignment weight Aalign For all alignment methods we linearly increase Aalign from 0 to 1.0 during the first 10000 steps.
For ASA we use history buffers of size 1000.
D.2 STL TO CIFAR EXPERIMENT SPECIFICATIONS
We use STL (Coates et al., 2011) and CIFAR- 10 (Krizhevsky, 2009) for this adaptation task. STL and CIFAR-I0 are both 10-class classification problems. There are 9 common classes between the two datasets. Following Shu et al. (2018) we create a 9-class classification problem by selecting the subsets of examples of the 9 common classes.
For the feature extractor, we adapt the deep CNN architecture of Shu et al. (2018). The feature representation is 192-dimensional vector. The classifier consists of single linear layer. The discriminator implemented by a 3-layer MI. with 512 hidden units and leaky-Rel.U activation.
We train all methods for 40 000 steps with batch size 64. We train the feature extractor, the classifier, and the discriminator with ADAM (Kingma & Ba, 2014) (learning rate 0.001, 81 0.5, B2 0.999, no weight decay). We perform single discriminator update per update of the feature extractor and the classifier.
The weight for the classification loss term is constant and set Asts For all alignment methods we use constant alignment weight Aalign 0.1.
For ASA we use history buffers size 1000.
Conditional entropy loss. Following (Shu et al., 2018) we use auxiliary conditional entropy loss on target examples for domain adaptation methods. For classifier C --> y and a feature extractor F : X + Z where classifier outputs the distribution over class labels {1...., K}
ERKK
K
[c](:)), 2 O, Σ [=) =1,
the conditional entropy loss on target examples N given by
N₂
K
Lest = A
(F((((2))) log (F*
(17)
No
Aen is the weight of the conditional entropy loss in the total training objective. This loss acts as additional regularization of the embeddings of the unlabeled target examples: minimization of the conditional entropy pushes target embeddings away from the classifier's decision boundary.
For all domain adapation methods we use the conditional entropy loss (17) on target examples with the weight Xen 0.1.
22

---

Published as a conference paper at ICLR 2022
D.3 EXTENDED EXPERIMENTAL RESULTS ON STL TO CIFAR
Effect of conditional entropy loss. In order to quantify the improvements of the support alignment objective and the conditional entropy objective in separation, we conduct an ablation study. In addition to the results reported in Section 5, we evaluate all domain adaptation methods (except VADA which uses the conditional entropy
the original implementation) on STI.-CIFAR task without the conditional entropy loss (Deni = 0). The results of the ablation study are presented in Table D. We observe that the effect of the auxiliary conditional entropy is essentially the same for all methods across all imbalance levels: with Aent 0.1 the accuracy either improves (especially the average class accuracy) or roughly stays on the same level. The relative ranking of distribution aligmment, relaxed distribution alignment, and support alignment methods is the same with both Amn = and Acat = 1. The results demonstrate that the benefits of support alignment approach and conditional entropy are orthogonal.
Table D.1: Results of ablation experiments of the effect of auxiliary conditional entropy loss on STL data. Same setup and reporting metrics as Table 1.
a 0.0
a 1.0
at 1.5
a: = 2.0
Algorithm
Aen
average
average
average
min
average
min
DANN
0.0
74.6 7
51.5 10.3
68.4 67.0
43.2 41.2
65.7 62.8
35.5 36-2
62.5 60.0
27.5 27
DANN
0.1
75.3
75-4
54.6
06.8
69.9 70.1
44.8
45.1
64.9 67-1
36.8 34.9
63.3 64.
27.0 21.2
33.9
IWDAN
0.0
70.4
70-7
47.2
48.8
68.6 08.8
43.6
13.2
66.7
66.0
44.7
20.3
63.9 02.9
36.5 37.3
IWDAN
0.1
69.9
69.9
70-7
50.5
47.9
50.4
68.7 7 69
45.8
34.8
50.5
67.1 67.3
44.7
44.8
20.4
64.4 64.8
36.8 37.9
IWCDAN
0.0
70.1
70.0
50.5
48.1
68.6 68.2
44.2
11:2
66.0 65.9
45.0
43.7
63.8 62.3
37.3 33.6
IWCDAN
0.1
70.1
70.1
47.8 41.4
69.4 09.4
47.1
46.3
66.1 67-2
39.9 40.5
64.5 G3.9
37.0
<DANN-4
0.0
69.4
70.0
46.5
49
69.6 6917
49.1
49.2
68.0 68.6
48.2
48.8
66.3 06
40.7 42.0
sDANN44
0.1
72.1 71.8
71.7
52.1
52.1
71.1 70.4
71.7
49.9
$1.8
48.1
70.0 69.4
68.7
48.6
43.5
40.0
66.4 66.2
67.6
39.0 33.6
ASA-sq
0.0
69.9 70-3
48.0 30-1
68.8 68.9
47.3
49.3
68.1
6R.7
45.4
47 8
65.7 66.4
43.6 45.0
ASA-sq
0.1
71.7
71.7
52.9
$3..
46.7
70.7 21.0
51.6
92.7
46.8
69.2
89:3
60-2
45.6
$2.0
68.1 08
44.7 45.9
ASA-abs
0.0
69.8 68.9
45.7 45.3
68.4 08.4
44.3 34.0
67.9
46.6 a0.4
66.3 63.
41.6 40.3
67.0
ASA-abs
0.1
71.6
71-7
49.0
33.5
70.9
71.0
49.2
50.0
69.6
69.9
43.2
48.5
67. 68-2
40.9
49.0
Comparison with optimal transport based baselines.
We provide additional experimental results comparing our method with OT-based methods for domain adaptation. We implement two OT-based methods which we describe below.
The first method is variant of the max-sliced Wasserstein distance (which was proposed for GAN training by Deshpande et al. (2019)) for domain adaptation. In the table below we refer to this method as DANN- In our implementation DANN- DANN-OT minimizes the Wasserstein distance between the pushforward distributions g Sa induced by the optimal log-loss discriminator g" (4). As discussed in Section 4.2 (paragraph "Distribution alignment") the computation of the Wasserstein distance between ID distributions can be implemented efficiently via sorting.
The second method is an OT-based variant of DANN which uses a dual Wasserstein dis- criminator instead of the log-loss discriminator. In the table below refer to this method as DANN-WGP. This method minimizes the Wasserstein distance in its dual Kantorovich form. We train the discriminator with the Wasserstein dual objective and gradient penalty proposed to enforce Lipshitz-norm constraint (Gulrajani et al., 2017).
We present the evaluation results of the OT-based methods STL-CIFAR domain adaptation task in the Table D.2. Note that the OT-based methods aim to enforce distribution alignment constraints. We observe that the bT-based methods follow the same trend DANN: they deliver improved accuracy compared to No DA in the balanced setting, but suffer in the imbalanced settings (a > 0) due to their distribution alignment nature.
We would also like to make a comment on bT-based relaxed distribution alignment. Wu et al. (2019b) propose method WDANN-B which minimizes the dual form of the asymmetrically--elaxdd
23

---

Published as a conference paper at ICLR 2022
Table D.2: Results of comparison with optimal transport based methods on STLA data. Same setup and reporting metrics as Table 1.
a 0.0
a 1.0
a 1.5
a 2.0
Algorithm
average
min
average
min
average
min
average
min
No DA
69.9 70-0
49.8 50-6
68.8 68.3
47.2 45-2
66.8 06.3
46.0 47-0
65.8 66-7
43.7 44.6
DANN
75.3 15-4
54.6
56.6
69.9 70.1
44.8 45.1
64.9 87 1
34.9 36.8
63.3 64.8
27.0
28.5
DANN-OT
76.0 76.0
15.8
55.2
86.6
64.3
67.7
68.9
67.1
43.0
36.5
43.7
64.5 65
60.9
34.4 34.6
20-3
61.3 64.4
62.0
24.3
23.2
26.5
DANN-WGP
74.8 To-1
53.5
63.3
67.7
67.9
38.6 34.4
63.3
27.0 26.3
59.0
21.9 18.6
sDANN-4
71.8 FR.1
52.1 S2.
71.1 T1-7
49.9 46.1
69.4 68.7
48.6 43.5
66.4 66.2
39.0 33.6
ASA-sq
71.7
21.9
71.7
52.9
$3.4
16.7
70.7
70.4
71.0
51.6
52.7
16.8
69.2
69.3
60
45.6
52.0
43.3
68.1
44.7
39.8
45.9
ASA-abs
71.6 T1-7
49.0
03.5
70.9
70.8
49.2 47.3
69.6
69.6
43.2 48.5
67.8 66.6
40.9 36.4
Wasserstein distance. However, they observe that sDANN-3 outperforms WDANN-B in experiments. Hence, we use sDANN-3 relaxed distribution alignment baseline in experiments.
Effect of alignment weight. We provide additional experimental results comparing ASA with DANN and VADA across different values of the alignment loss weight Aalign STL->CIFAR task. The results are shown in Table D.3. DANN with a higher alignment weight Aalign 1.0 performs better in the balanced (a 0) setting and worse in the imbalanced (a >> 0) setting compared to a lower weight Aalign 0.1, as the distribution alignment constraint is enforced stricter. VADA optimizes a combination of distribution alignment VAT (virtual adversarial training) objectives (Shu et al., 2018), and we observe the same trend: with lower alignment weight Aalign = 0.01, VADA performs worse in the balanced setting and better in the imbalanced setting compared weight Aulign 0.1. Weight Aaiggn = 0.1 is a middle ground between having poor performance in the imbalanced setting (Aalign = 1.0) and not sufficiently enforcing distribution alignment (Aalign = 0.01).
higher
The role of VAT (similarly to that of conditional entropy loss) is orthogonal to alignment objectives. Thus, we provide additional evaluations of combining support alignment and VAT (the "ASA-sq + VAT entry in Table D.3) with alignment weight Xalign 1.0. The good performance of such combination shows that:
One could improve our current support alignment performance by using auxiliary objectives.
Support alignment based method performs qualitatively different from distribution alignment based method, since the performance holds with stricter support alignment while distri- bution alignment needs to loosen the constraints considerably reduce the performance degradation in the imbalanced setting.
Table D.3: Results of comparison of ASA with DANN and VADA across different values of the alignment loss weight Aatggn on STL->CIFR data. Same setup and reporting metrics as Table
0.0
1.0
1.5
2.0
Algorithm
Dalign
average
min
average
min
average
min
average
min
DANN
0.01
72.3 12
7
49.5 50.8
70.6 Aa.
48.9 51:3
68.5
46.1 36.3
65.9 06.0
36.7 39.4
DANN
0.1
75.3
75.4
56.6
69.9 6x6
44.8 45.1
64.9 87
34.9 86.8
63.3 6L.
27.0 28.5
DANN
1.0
77.2 27-3
59.1 58.5
66.3 99.8
37.9 41.8
62.8 03
27.5
28.0
58.7
59.7
18.5 20.5
VADA
0.01
74.4
54.2 82.6
71.7 21.2
51.6
$5.0
69.5
68.4
47.5 10.0
65.9
37.2
35.3
VADA
0.1
76.7
76-7
56.9 68.3
70.6
47.7 48.8
66.1 65.4
35.7
30-3
63.2 00.2
25.5 25.2
ASA-sq
0.1
71.7 71
52.9 63-4
70.7 21.0
51.6 52.7
69.2
69
45.6 62
68.1 07
44.7 45.9
ASA-sq + VAT
1.0
74.2 74.3
52.2 97.
72.2 71.
53.5 $3.9
70.6 70.8
48.9 2
67.4 67.T
43.0 46.0
24

---

Published as a conference paper at ICLR 2022
D.4 VISDA- 17 EXPERIMENT SPECIFICATIONS
We use train and validation sets of the VisDA-17 challenge (Peng et al., 2017).
For the feature extractor we ResNet-50 He al. (2016) architecture with modified output size of the final linear layer. The feature representation is 256-dimensional vector. We use the weights from pre-trained ResNet-50 model (torchvision model hub) for all layers except the final linear layer. The classifier consists of a single linear layer. The discriminator is implemented by a 3-layer MLP with 1024 hidden units and leaky-Red ReUL activation.
We train all methods for 50 000 steps with batch size 36. We train the feature extractor, the classifier, and the discriminator with SGD. For the feature extractor we use learning rate 0.001, momentum 0.9, weight decay 0.001. For the classifier we use learning rate 0.01, momentum 0.9, weight decay 0.001. For the discriminator we use learning rate 0.005, momentum 0.9, weight decay 0.001. We perform a single discriminator update per update of the feature extractor and the classifier. We linearly anneal the feature extractor's 's and classifier's learning rate throughout the training (50 steps. By the end of the training the learning rates of the feature extractor and the classifier are decreased by a factor of 0.05.
The weight for the classification term is constant and set Adts = 1. We introduce schedule for the alignment weight Aaignn For all alignment methods we linearly increase Aulign from 0 to 0.1 during the first 10000 steps. For all methods with the weight Aem = 0.1.
auxiliary conditional entropy loss target examples
For ASA use history buffers of size 1000.
D.5 HISTORY SIZE EFFECT AND EVALUATION OF SUPPORT DISTANCE
History size effect. To quantify the effects of mini-bated training mentioned in Section 3, we explore different sizes of history buffers on USPS SEPSS-MNE MNIST task with the label distribution shift a 1.5. The results are presented in Figure D.1 and Table D.4. Figure D.2 shows the distributions of outputs of the learned discriminator at the end the training. While without any alignment objectives neither the densities nor the supports of gº +P& and 90 44% are aligned, both alignment methods approximately satisfy their respective alignment constraints. Compared with DANN results, ASA with small history size performs similarly to distribution alignment, while all history sizes enough for support alignment. We also observe the correlation between distribution distance and target accuracy: under label distribution shifts, the better distribution alignment is achieved, the more target accuracy suffers. Note that with too big history buffers (e.g. n 5000), we observe a sudden drop in performance and increases in distances. We hypothesize that this could be caused by the fact that the history buffer stores discriminator output values from the past steps while the discriminator parameters constantly evolve during training. As result, for a large history buffer, the older items might no longer accurately represent the current pushforward distribution as they become outdated.
Minimun cless aceuray (96
ASA
DANN
No DA
Distribution distanee
80
10P
Support distence
10
10
10
200
10
0
1000
500
1000
2000
5000
100
500
1000
2000
3000
100
500
1000 2008
5000
ASA history size
ASA
history size
ASA history size
Figure D.I: Evaluation of history size effect for ASA on MNIST-+USPS with the label distribu- tion shift (a = 1.5). The panels show (left to right): minimum class accuracy on target test set; Wasserstein distance Dw (g =P2 =92 between the pushforward distributions of source and target representations induced by the discriminator; SSD divergence DA (
"22.99 t (gg) 2 between the push- forward distributions. In each panel the dashed lines show the respective quantities for "No DA" and DANN methods.
25

---

Published as a conference paper at ICLR 2022
Direct evaluation of support distance. In order directly evaluate the ability of ASA (with history buffers) to enforce support alignment, we consider the setting of the illustrative experiment described in Section 5 (3-class USPS -PNISS adaptation with 2D feature extractor, methods No DA, DANN, and ASA-abs (with different history buffer sizes). For each method we consider the embedding space of the learned feature extractor at the end of training and compute Wasserstein distance Dw (p2- 92) and SSD divergence Da (p2 92) between the embeddings of source and target domain (note that we compute the distances in the original embedding space directly without projecting data to ID with the discriminator). To ensure meaningful comparison of the distances between different embedding spaces, we apply global affine transformation for each embedding space: we center the embeddings so that their average is O and re-scale them so that their average norm The results of this evaluation are shown Table D.5. We observe that, compared to no alignment and distribution alignment (DANN) methods, ASA aligns the supports without necessarily aligning the distributions (in this imbalanced setting, distribution alignment implies low adaptation accuracy).
1.5). We compare
Table D.4: Analysis of effect history size parameter for ASA USPS MMPSSMNIIST with class label distribution shift corresponding to a = 1.5. We report distribution and support distances between the pushforward distributions nd : and g 492. well as the value of discriminator's log-loss.
Target accuracy (5)
Distribution distances
Method
History size
average
min
Dw (0" 1722.9 9
Dala" 1722.9 492)
Log-loss
No
71.28 F1:25
27.46 21
322 307.56
40.35 46.06
00.05 00.07
DANN
69.96 71-25
01.11
00.90
00.11 00.10
00.00 90:00
00.65 00.65
ASA-abs
62.75
61:
64:35
06.36
20 63
90
01.07
00:09
01:15
00.01
00.01
06:
00.57 00.58
00.55
ASA-abs
100
80.58 78
35.09
32 10
02.64
02:15
00.00 00.00
00.53 00.52
ASA-abs
500
92.02 60
76.96
70.94
06.21
05.69
00.00 00.00
00.45 00.45
ASA-abs
1000
92.54 92.99
82.41
85-43
08.06
06.19
00.01 30.02
00.41 00.41
ASA-abs
5000
$7.50 86.03
84.86
62. 19
46.98
71
29.23
24.64
25.00
00.05
00.08
00.05
00.29
00.30
00.29
0.14
Ayt
(target)
1.0
0.5
A
0.35
0.10
0.5
0.25
0.08
0.5
0.5
0.20
0.34
0.10
0.2
0.3
0.00
0.0
0.0
0.00
DA
(b) DANN
(c) ASA (7
ASA
1000)
Figure D.2: Kernel density estimates (in the discriminator output space) of g" pP2z 9dh .92 at the end of the training on USPS- -NNIST task with a 1.5. n is the size of ASA history buffers.
26

---

Published as a conference paper at ICLR 2022
Table D.5: Results of No DA, DANN, and ASA-abs (with different history sizes) 3-class USPS MNIST adaptation with 2D feature extractor and label distribution shift corresponding 1.5. We report average and minimum target class accuracy, well Wasserstein distance Dw and support divergence DA between source p2 and target 42 2D embedding distributions. We report median (the main number), and 25 (subscript) and 75 (superscript) percentiles across 5 runs.
Algorithm
History size
Accuracy (avg)
Accuracy (min)
Dw (p2. 9
Dz (p2 o
No DA
63.0 69.6 62.3
45.3 53.6 $7.9
0.78 00.75
00.84
0.10 0.10
0.10
DANN
75.6 72.4
83.7
54.8
55. 49.6
0.07 0.06
0.08
0.02 0.02
0.02
ASA-abs
0
73.9 73.4
84.1
61.8
72.
0.23 U-47
0.03
0.03
51.6
0.22
0.03
ASA-abs
100
88.5
05.
71.4
93.3
0.54 0.56
0.36
0.03 0.03
86.8
70.6
0.03
ASA-abs
500
94.5 88.7
94.7
89.0 90.3
0.59 0.64
0.03 0.03
85.1
0.55
0.03
ASA-abs
1000
91.1
93.0
85.6 80.7
86.22
0.59 0.55
0.62
0.03 0.03
01.1
0.03
ASA-abs
2000
94.0 94.7
88.6 80.2
89.4
0.62 0.66
0.03 0.03
0.58
0.03
ASA-abs
5000
82.1
68.9 65.5
70.9
0.64 0.67
0.04
27


Reviews:
[{"review_id": "ubaQ26S3J1", "reviewer": "Reviewer_nH3D", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ Strengths:\n\nIt seems that the authors propose a \"new\" divergence to align supports of distributions, and empirically show its advantages in domain adaption.\n\n+ Weakness:\n\nThe proposed SSD is closely related to the Chamfer divergence, if not the same (see ArXiv1612.00603).\n\n+ Detailed comments:\n\n1. Could the authors discuss the differences between the proposed SSD with the Chamfer divergence? (see ArXiv1612.00603, ArXiv2102.04014)? \n\n2. It seems unclear about the motivation of using the 1d-projection? Is it better to consider the original space? What are the advantages of using the 1d-projection?\n\n3. I also wonder about the advantages of using the 1d-projection by the optimal log-loss discriminator. Why is it favored over other projects? (is it suboptimal/optimal in some sense?)\n\n4. The authors emphasize considering the alignment for the supports of distributions instead of comparing distributions. Are there any reasons to consider the expectation instead of average in equation (1) for SSD? [In case, only supports are important, I wonder why SSD  needs their \"weights\" in its definition?]\n\n5. For experiments, I think the proposed method is closely related to optimal transport-based approaches for domain adaption. However, it seems that there are no such baselines yet (e.g., using standard OT, using sliced-Wasserstein, or their variants for domain adaption).\n\n", "summary_of_the_paper": "The authors propose symmetric support difference (SSD) divergence to align supports of distributions. The authors further propose to use the optimal log-loss discriminator to project supports into 1-dimensional space for SSD. The authors also illustrate the advantages of the proposed methods for domain adaptation.", "summary_of_the_review": "It seems that the proposed method (SSD) is closely related to Chamfer divergence if not the same (however, there are no discussions about their relationship yet). For experiments, in my opinion, the proposed approach is similar to optimal transport. Therefore, the authors should compare the proposed approach with other optimal transport-based approaches for domain adaption.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "i5fPdTtIHX", "reviewer": "Reviewer_BDME", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n- Proposes a novel support (pseudo-)divergence (not a true divergence because D = 0 means the supports match but possibly the distributions do not.) to measure the difference in supports between distributions.  This is an extension of the Hausdorff distance.\n\n- Theoretically proves that alignment of the support of the optimal discriminator implies alignment of the distributions and gives several insights about alternatives.\n\n- Place the support alignment goal and objectives within related relaxed and full alignment objectives.\n\n- Extensive empirical experiments under the label shift setting.\n\n**Weakness:**\n- My main concern is that one solution to support alignment is distribution alignment.  And more importantly, the algorithm (as shown by the mini-batch part) seems to naturally collapse to distribution alignment (as evidenced by the mini-batch issue).  However, the label shift application suggests that a better solution is to align the supports but NOT align the whole distribution (even though that is one solution). Thus, it seems an explicit regularization or something should be added to the objective rather than an ad-hoc history buffer.  One suggestion might be a transportation cost regularization to the transformation (see below).\n\n- Another issue is that the support alignment method is only evaluated using the extrinsic task of domain adaptation.  The paper does not include results on evaluating the support alignment explicitly.  Even a simulated experiment may help to evaluate the algorithm with respect to support alignment directly.\n\n**Minor Weaknesses:**\n- The history buffer idea seems to be entirely heuristic.  Can you explain why this might be important theoretically? Or why without it there may be a collapse to distribution alignment?  My intuition is that the objective will naturally move towards shrinking these values towards 1/2 as mentioned below in comments but maybe there is something else going on.\n\n- The idea of \"constraints\" being an issue for distribution alignment seems a little odd (as in the first figure).  The solution seems to be to increase the model capacity rather than align the support of the distribution. This seems to be a fairly weak high-level motivation for support alignment (at least in the introduction). The shift in label distributions is a stronger motivation in my view and should be the primary example.\n\n- Proposition 2.2 is a bit confusing to understand.  Maybe a little illustration would be helpful to show the densities of $\\tau_p$ and $\\tau_q$ and the resulting ratio of densities for $\\tau$ along with the $x$ and $x'$ that was mentioned in the explanation.\n\n- The paper should make a clearer distinction between \"continuous\" support alignment (which seems to be the overall benefit) and discrete support alignment (which seems to be the case in the algorithm which uses samples).\n\n\n**Other comments or questions**\n- It is interesting that full distribution alignment corresponds to the case when the optimal discriminator is just 1/2.  Thus, it seems that concentrating all mass near 0.5 is really distribution alignment, while \"spreading\" out the mass of the $\\tau$ predicted probability distributions (except for 0 and 1) means that the supports are aligned.  Is this a correct intuition/interpretation?  Is this also why mini-batch training actually finds this solution (as it is the most stable solution maybe)---i.e., we can always improve the objective by shrinking towards 1/2, which is the full distribution alignment case?\n\n- If the above interpretation is correct, then it would seem that you could actually enforce support alignment but NOT distribution alignment by requiring $\\tau$ to have high entropy.\n\n- It seems that another way to avoid this \"collapsing\" phenomena would be to add a transportation cost term to the transformations.  Thus, we would seek to move the points as little as possible to align the support of the distributions. This would also be much simpler and more grounded than the history idea. Have you considered this idea?\n\n- Proposition 4.3 seems quite strong.  Is statement 1 related to max sliced wasserstein distance (i.e., joint Wasserstein is 0 if and only if max sliced Wasserstein is 0)? \n\n- The result tables should be shown after the description of the methods, metrics, etc.\n\n- I might suggest rewriting with label shift as the core issue as this seems to be the most realisitic example and iis strongly supported by your experiments.  Other settings are not demonstrated by your experiments and are not well-motivated.\n\n", "summary_of_the_paper": "Motivated by label shift between source and target domains, the paper proposes a novel support alignment task with a novel support divergence.  They provide theoretical results for these and build up theory that leads to a simple adversarial algorithm for support alignment using a discriminator and nearest neighbor algorithms in 1D.\nThe paper provides extensive empirical results comparing to common domain adaptation methods.\n", "summary_of_the_review": "Overall, the paper presents novel ideas and tasks and places the work well within existing literature.  This could have significant impact on unsupervised domain alignment and more generally on thinking about alignment problems. The main weakness is the potential for the proposed algorithm to collapse to standard distribution alignment and the lack of explicit evaluation of support alignment.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fUB56uwS9Vq", "reviewer": "Reviewer_ukYK", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nOverall a great paper. Very easy to follow, the algorithm seem theoretically grounded and simple and very well motivated. It's a very important domain, and the story is pretty clear in that regard. Uses well-known tools (e.g., GAN discriminator), so should be easy to implement. Experiments are well-motivated and support the approach.\n\nWeaknesses (not really) / comments:\nOne thing that would be good in this work is to see this applied to unsupervised domain translation (e.g., converting SVHN to MNIST). There are a number of works that attempt to address this problem (e.g., Binkowski 2019), but they come with a great deal of complexity. Have you tried this approach?\n\nI feel like d(.,.) could have been clarified better earlier, at least to provide some examples to avoid the reader wondering about what it was until the experiment section.\n\nBinkowski 2019: Batch weight for domain adaptation with mass shift", "summary_of_the_paper": "This paper introduces a method of aligning the support of one distribution to another. This is important as for domain adaptation problems, distributional alignment can come with a lot of problems (such as moving the mass of one class to another when there are differences in representation of different classes).", "summary_of_the_review": "The paper is very clear, well motivated, the approach is simple, and problem is important.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SkV32zeqZV4", "reviewer": "Reviewer_FsUp", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n* Rreducing divergence measures between multi-dimensional distributions to unidimensional ones, whether it concerns the support alignment (Theorem 2.1) or the Wasserstein type divergences (Proposition 4.3).\n* The interest in the minimum accuracy per class, which is a metric that is rarely considered in the domain adaptation literature to the best of my knowledge, but that is important in real-world applications.\n\n# Weaknesses\n\n* There is no conclusion summarizing the main findings of the paper at least.\n\n* Experimental part:\n\n    * It is hard to tell how much of the success can be attributed to the developments in the theory, especially that apart from VADA, other methods do not use a conditional entropy loss in their original implementations, whereas this latter loss is added with a coefficient $\\lambda_{\\text{ent}} = 0.1$. Also, the expression of this unsupervised loss should be mentioned in the supplementary material (or at least a reference to its expression, e.g. from [1]). I am not advocating for the idea that an algorithm should surpass all of the state-of-the-art methods, but it is instructive for the community to see if the proposed theory offers the same order of performance.\n    * The domain discriminator has a single update per 1 update of feature extractor and classifier. This does not go in line with the theoretical findings that only concern the optimal discriminator $f^*(x)$, i.e. after convergence. Such a detail needs to be mentioned in the main paper.\n\n# Other comments\n* In Proposition 2.3, a counter-example is provided concerning the equivalence between aligning the supports in the original spaces and their 1D push-forward counterparts induced by a domain discriminator. While the establishment of the result concerning the JS divergence discriminator relies on densities, the provided counter-example concerns discrete distributions. It would be better to provide a counter-example with distributions having densities.\n* Appendix A: it should be $f$ rather than $f_W^*$ in the operand of the $\\sup_{f:Lip(f)\\leq 1}$ operator. \n## Typos:\n* Before Remark 2.1.1: \"techincal\" --> \"technical\"\n* Paragraph **Setup**: \"optimizied\" --> \"optimized\"\n\n# Questions\n* In Proposition B.1, after slicing the distributions by $\\theta \\in \\mathbb S^1$, are the supports of the push-forward distributions equal? Because $f^\\theta \\\\#q$ will have no supports on a segment of length $2$, centered at zero.\n* I am not sure if I understand the notation $q^Y(y) \\propto \\sigma(y)^{-\\alpha}$. Does it imply that for example, we can have $q^Y(1) \\propto K^{-\\alpha}$ ?\n* Provided that I understood correctly the previous point, when $\\alpha$ is very large, it means that some class is almost absent, leading to the partial DA setting. Did you test the approach in this regime?\n* Did you notice any acceleration of the algorithm via using the strongly convex squared distance rather than the absolute one ?\n* When you mention \"all alignment methods\", do you refer to only **ASA-sq** and **ASA-abs**, or to all of the other approaches as well? If the latter case holds, why not use the default parameters of the methods (e.g. $\\lambda_{\\text{align}} = 0.01, 1$ respectively for VADA and DANN, instead of the fixed value 0.1) ?\n* Is it possible to establish theoretical guarantees on the convergence of the empirical support divergence to its true counterpart? and can this aspect be related to the observation that large buffer sizes imply a drop in performance?\n\n# References\n[1] Shu, R., Bui, H. H., Narui, H., & Ermon, S. (2018). A dirt-t approach to unsupervised domain adaptation. *arXiv preprint arXiv:1802.08735*.", "summary_of_the_paper": "The authors propose a weak alignment of distributions consisting in equalizing their supports. They justify this choice by the recent advancements in Domain Adaptation that show the drawbacks of a strict alignment of marginal distributions. A distance between supports is defined as a modification of the Hausdorff distance between sets. The support divergence is further proven to be a limit case of a previously proposed weak Wasserstein distance requiring only the boundedness of the density ratio, thus defining a continuum of relaxations of divergences.  For all of the divergences in this continuum, a theoretical result shows the possibility of reducing the original problem of computing the divergence to unidimensional distributions, obtained via a domain discriminator minimizing a Jensen-Shannon divergence. The theoretical findings are used to propose an algorithm that is tested in the class imbalanced domain adaptation setting, highlighting the interest of the approach.", "summary_of_the_review": "The theory established in the paper is interesting for the community, especially with the general result of reducing the comparison between distributions to a uni-dimensional one. However, there are some gaps between the theory and the proposed algorithm, especially concerning the use of the unsupervised conditional entropy loss on the target domain.\n\n# Post-rebuttal \nAs a result of the authors's detailed response, I update my review score to 8.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thanks for your submission to ICLR.

Three of the four reviewers are ultimately (particularly after discussion) very enthusiastic about the paper, and feel that their concerns have been adequately addressed.  The fourth reviewer has not updated his/her score but has indicated that their concerns were at least somewhat addressed.  I took a look at their review and agree that the authors have addressed these concerns sufficiently.  I am happy to recommend this paper for acceptance at this point.  Note that I really appreciate the time and effort that the authors went into adding additional results and clarifications for the reviewers.
Decision: Accept (Spotlight)