Adversarial Support Alignment

1 INTRODUCTION . Learning tasks often involve estimating properties of distributions from samples or aligning such characteristics across domains . We can align full distributions ( adversarial domain alignment ) , certain statistics ( canonical correlation analysis ) , or the support of distributions ( this paper ) . Much of the recent work has focused on full distributional alignment , for good reasons . In domain adaptation , motivated by theoretical results ( Ben-David et al. , 2007 ; 2010 ) , a series of papers ( Ajakan et al. , 2014 ; Ganin & Lempitsky , 2015 ; Ganin et al. , 2016 ; Tzeng et al. , 2017 ; Shen et al. , 2018 ; Pei et al. , 2018 ; Zhao et al. , 2018 ; Li et al. , 2018a ; Wang et al. , 2021 ; Kumar et al. , 2018 ) seek to align distributions of representations between domains , and utilize a shared classifier on the aligned representation space . Alignment in distributions implies alignment in supports . However , when there are additional objectives/constraints to satisfy , the minimizer for a distribution alignment objective does not necessarily minimize a support alignment objective . Example in Figure 1 demonstrates the qualitative distinction between two minimizers when distribution alignment is not achievable . The distribution alignment objective prefers to keep supports unaligned even if support alignment is achievable . Recent works ( Zhao et al. , 2019 ; Li et al. , 2020 ; Tan et al. , 2020 ; Wu et al. , 2019b ; Tachet des Combes et al. , 2020 ) have demonstrated that a shift in label distributions between source and target leads to a characterizable performance drop when the representations are forced into a distribution alignment . The error bound in Johansson et al . ( 2019 ) suggests aligning the supports of representations instead . In this paper , we focus on distribution support as the key characteristic to align . We introduce a support divergence to measure the support mismatch and algorithms to optimize such alignment . We also position our approach in the spectrum of other alignment methods . Our contributions are as follows ( all proofs can be found in Appendix A ) : 1 . In Section 2.1 , we measure the differences between supports of distributions . Building on the Hausdorff distance , we introduce a novel support divergence better suited for optimization , which we refer to as symmetric support difference ( SSD ) divergence . 2 . In Section 2.2 , we identify an important property of the discriminator trained for Jensen–Shannon divergence : support differences in the original space of interest are “ preserved ” as support differences in its one-dimensional output space . 3 . In Section 3 , we present our practical algorithm for support alignment , Adversarial Support Alignment ( ASA ) . Essentially , based on the analysis presented in Section 2.2 , our solution is to align supports in the discriminator 1D space , which is computationally efficient . 4 . In Section 4 , we place different notions of alignment – distribution alignment , relaxed distribution alignment and support alignment – within a coherent spectrum from the point of view of optimal transport , characterizing their relationships , both theoretically in terms of their objectives and practically in terms of their algorithms . 5 . In Section 5 , we demonstrate the theoretical effectiveness of support alignment in practice for domain adaptation setting . Compared to other alignment-based baselines , our proposed method is more robust against shifts in label distributions . 2 SSD DIVERGENCE AND SUPPORT ALIGNMENT . Notation . We consider an Euclidean space X = Rn equipped with Borel sigma algebra B and a metric d : X ×X → R ( e.g . Euclidean distance ) . Let P be the set of probability measures on ( X , B ) . For p ∈ P , the support of p is denoted by supp ( p ) and is defined as the smallest closed set X ⊆ X such that p ( X ) = 1. f♯p denotes the pushforward measure of p induced by a measurable mapping f . The distance between a point x ∈ X and a subset Y ⊆ X is defined as d ( x , Y ) = infy∈Y d ( x , y ) . The symmetric difference of two sets A and B is defined as A△B = ( A \B ) ∪ ( B \A ) . 2.1 DIFFERENCE BETWEEN SUPPORTS . To align the supports of distributions , we first evaluate how different they are . Similar to distribution divergences like Jensen–Shannon divergence and Wasserstein distance , we introduce a notion of support divergence . A support divergence1 between two distributions in P is a function DS ( · , · ) : P × P → R satisfying : 1 ) DS ( p , q ) ≥ 0 for all p , q ∈ P ; 2 ) DS ( p , q ) = 0 iff supp ( p ) = supp ( q ) . While a distribution divergence is sensitive to both density and support differences , a support divergence only needs to detect mismatches in supports , which are subsets of the metric space X . An example of a distance between subsets of a metric space is the Hausdorff distance : dH ( X , Y ) = max { supx∈X d ( x , Y ) , supy∈Y d ( y , X ) } . Since it depends only on the greatest distance between a point and a set , minimizing this objective for alignment only provides signal to a single point . To make the optimization less sparse , we consider all points that violate the support alignment criterion and introduce symmetric support difference ( SSD ) divergence : D△ ( p , q ) = Ep [ d ( x , supp ( q ) ) ] + Eq [ d ( x , supp ( p ) ) ] . ( 1 ) Proposition 2.1 . SSD divergence D△ ( p , q ) is a support divergence . 1It is not technically a divergence on the space of distributions , since DS ( p , q ) = 0 does not imply p = q . 2.2 SUPPORT ALIGNMENT IN ONE-DIMENSIONAL SPACE . Goodfellow et al . ( 2014 ) showed that the log-loss discriminator f : X → [ 0 , 1 ] , trained to distinguish samples from distributions p and q ( supf Ex∼p [ log f ( x ) ] + Ex∼q [ log ( 1− f ( x ) ) ] ) can be used to estimate the Jensen–Shannon divergence between p and q . The closed form maximizer f∗ is f∗ ( x ) = p ( x ) p ( x ) + q ( x ) , ∀x ∈ supp ( p ) ∪ supp ( q ) . ( 2 ) Note that for a point x /∈ supp ( p ) ∪ supp ( q ) the value of f∗ ( x ) can be set to an arbitrary value in [ 0 , 1 ] , since the log-loss does not depend on f ( x ) for such x . Using ( 2 ) , we can establish a connection between the pushforward distributions f∗♯p and f ∗ ♯q . Proposition 2.2 . Let f∗ be the optimal log-loss discriminator ( 2 ) between p and q , and let τp and τq be the probability density functions of the pushforward measures f∗♯p and f ∗ ♯q respectively 2 . Then , τp ( a ) τp ( a ) + τq ( a ) = a , ∀a ∈ supp ( τp ) ∪ supp ( τq ) . ( 3 ) Intuitively this proposition states the following . If the optimal discriminator f∗ maps x ∈ X to a ∈ [ 0 , 1 ] , i.e . f∗ ( x ) = a , a directly corresponds to the ratio of densities in not only the original space a = p ( x ) / ( p ( x ) + q ( x ) ) , but also the 1D discriminator output space a = τp ( a ) / ( τp ( a ) + τq ( a ) ) , which leads to the fact that p ( x ) / ( p ( x ) + q ( x ) ) = f∗ ( x ) = a = τp ( a ) / ( τp ( a ) + τq ( a ) ) . Using Proposition 2.2 , we can prove our main theorem , which characterizes the ability of the log-loss discriminator to identify support misalignment . Theorem 2.1 . Let f∗ be the optimal discriminator ( 2 ) between p and q . Then , D△ ( p , q ) = 0 if and only if D△ ( f∗♯p , f∗♯q ) = 0 . Proof sketch . We observe that for any point x ∈ supp ( p ) \ supp ( q ) we have f∗ ( x ) = 1 and , by Proposition 2.2 , τp ( 1 ) > 0 and τq ( 1 ) = 0 . Repeating the same argument in reverse direction we see that supp ( p ) \ supp ( q ) ̸= ∅ if and only if 1 ∈ supp ( f∗♯p ) \ supp ( f∗♯q ) . Similarly , supp ( q ) \ supp ( p ) ̸= ∅ if and only if 0 ∈ supp ( f∗♯q ) \ supp ( f∗♯p ) . In other words , f∗ outputs the extreme values 1 and 0 if and only if the supports of p and q are misaligned and each of this extreme value belongs to the support of only one of the pushforward distributions supp ( f∗♯p ) or supp ( f ∗ ♯q ) . We conclude this section with two technical remarks on Theorem 2.1 . Remark 2.1.1 . The result of Theorem 2.1 does not necessarily hold for other types of discriminator . For instance , the dual Wasserstein discriminator ( Arjovsky et al. , 2017 ; Gulrajani et al. , 2017 ) does not always highlight the support difference in the original space as a support difference in the discriminator output space . This observation is formaly stated in the following proposition . Proposition 2.3 . Let f⋆W be the optimal solution of supf : Lip ( f ) ≤1 Ex∼p [ f ( x ) ] − Ex∼q [ f ( x ) ] , where Lip ( f ) is the Lipschitz constant of f . There exist distributions p and q such that supp ( p ) ̸= supp ( q ) but supp ( f⋆W ♯p ) = supp ( f ⋆ W ♯q ) . Remark 2.1.2 . We note that in practice the discriminator is typically parameterized as f ( x ) = σ ( g ( x ) ) , where g : X → R is realized by a deep neural network and σ ( x ) = ( 1 + e−x ) −1 is the sigmoid function . The optimization problem for g is inf g Ex∼p [ log ( 1 + e−g ( x ) ) ] + Ex∼q [ log ( 1 + eg ( x ) ) ] , ( 4 ) and the optimal solution is g∗ ( x ) = log p ( x ) − log q ( x ) . Naturally the result of Theorem 2.1 holds for discriminator g∗ , since g∗ ( x ) = σ−1 ( f∗ ( x ) ) and σ is a bijective mapping from ( −∞ , ∞ ) to [ 0 , 1 ] . The main difference of g∗ from f∗ is that in the case of supp ( p ) ̸= supp ( q ) , optimization of the log-loss makes the values g∗ ( x ) infinite ( ±∞ ) for x ∈ supp ( p ) △ supp ( q ) . 2In order to simplify the argument , we assume that all distributions p , q , f∗♯p , f ∗ ♯q have PDFs .

Summary: The authors propose a weak alignment of distributions consisting in equalizing their supports. They justify this choice by the recent advancements in Domain Adaptation that show the drawbacks of a strict alignment of marginal distributions. A distance between supports is defined as a modification of the Hausdorff distance between sets. The support divergence is further proven to be a limit case of a previously proposed weak Wasserstein distance requiring only the boundedness of the density ratio, thus defining a continuum of relaxations of divergences.  For all of the divergences in this continuum, a theoretical result shows the possibility of reducing the original problem of computing the divergence to unidimensional distributions, obtained via a domain discriminator minimizing a Jensen-Shannon divergence. The theoretical findings are used to propose an algorithm that is tested in the class imbalanced domain adaptation setting, highlighting the interest of the approach.