Outlier Robust Optimal Transport

1 INTRODUCTION . Optimal transport is a fundamental problem in applied mathematics . In its original form ( Monge , 1781 ) , the problem entails finding the minimum cost way to transport mass from a prescribed probability distribution µ on X to another prescribed distribution ν on X . Kantorovich ( 1942 ) relaxed Monge ’ s formulation of the optimal transport problem to obtain the Kantorovich formulation : OT ( µ , ν ) , min Π∈F ( µ , ν ) E ( X1 , X2 ) ∼Π [ c ( X1 , X2 ) ] , ( 1.1 ) where F ( µ , ν ) is the set of couplings between µ and ν ( probability distributions on X × X whose marginals are µ and ν ) and c is a cost function , where we typically assume c ( x , y ) ≥ 0 and c ( x , x ) = 0 . Compared to other notions of distance between probability distributions , optimal transport uniquely depends on the geometry of the sample space . Recent advancements in optimization for optimal transport ( Cuturi , 2013 ; Solomon et al. , 2015 ; Genevay et al. , 2016 ; Seguy et al. , 2018 ) enabled its broad adaptation in machine learning applications where geometry of the data is important . See ( Peyré & Cuturi , 2018 ) for a survey . Optimal transport has found applications in natural language processing ( Kusner et al. , 2015 ; Huang et al. , 2016 ; Alvarez-Melis & Jaakkola , 2018 ; Yurochkin et al. , 2019 ) , generative modeling ( Arjovsky et al. , 2017 ) , clustering ( Ho et al. , 2017 ) , domain adaptation ( Courty et al. , 2014 ; 2017 ) , large-scale Bayesian modeling ( Srivastava et al. , 2018 ) , and many other domains . Many applications use OT as a loss in an optimization problem of the form : θ ∈ arg minθ∈Θ OT ( µn , νθ ) , ( 1.2 ) where { νθ } θ∈Θ is a collection of parametric models , µn is the empirical distribution of the samples . Such estimators are called minimum Kantorovich estimators ( MKE ) ( Bassetti et al. , 2006 ) . They are popular alternatives to likelihood-based estimators , especially in generative modeling . For example , when OT ( · , · ) is the Wasserstein-1 distance and νθ is a generator parameterized by a neural network with weights θ , equation 1.2 corresponds to the Wasserstein GAN ( Arjovsky et al. , 2017 ) . One drawback of optimal transport is its sensitivity to outliers . Because all the mass in µ must be transported to ν , a small fraction of outliers can have an outsized impact on the optimal transport problem . For statistics and machine learning applications in which the data is corrupted or noisy , this is a major issue . For example , the poor performance of Wasserstein GANs in the presence of outliers was noted in the recent works on outlier-robust generative learning with f -divergence GANs ( Chao et al. , 2018 ; Wu et al. , 2020 ) . The problem of outlier-robustness in MKE has not been studied , with the exception of two concurrent works ( Staerman et al. , 2020 ; Balaji et al. , 2020 ) . In this paper , we propose a modification of OT to address its sensitivity to outliers . Our formulation can be used as a loss in equation 1.2 so that it is robust to a small fraction of outliers in the data . To keep things simple , we consider the -contamination model ( Huber & Ronchetti , 2009 ) . Let νθ0 be a member of a parametric model { νθ : θ ∈ Θ } and let µ = ( 1− ) νθ0 + ν̃ , where µ is the data-generating distribution , > 0 is the fraction of outliers , and ν̃ is the distribution of the outliers . Although the fraction of outliers is capped at , the value of the outliers is arbitrary , so the outliers may have an arbitrarily large impact on the optimal transport problem . Our goal is to modify the optimal transport problem so that it is more robust to outliers . We have in mind the downstream application of learning θ0 from ( samples from ) µ in the -contamination model . Our main contributions are as follows : 1 . We propose a robust OT formulation that is suitable for statistical estimation in the - contamination model using MKE . 2 . We show that our formulation is equivalent to the original OT problem with a clipped transport cost . This connection enables us to leverage the voluminous literature on computational optimal transport to develop efficient algorithm to perform MKE robust to outliers . 3 . Our formulation enables a new application of optimal transport : outlier detection in data . 2 PROBLEM FORMULATION . 2.1 ROBUST OT FOR MKE . To promote outlier-robustness in MKE , we need to allow the corresponding OT problem to ignore the outliers in the data distribution µ . The -contamination model imposes a cap on the fraction of outliers , so it is not hard to see that ‖µ − νθ0‖TV ≤ , where ‖ · ‖TV is the total-variation norm defined as ‖µ‖TV = ∫ 1 2 |µ ( dx ) | . This suggests we solve a TV-constrained/regularized version of equation 1.2 . The constrained version min θ∈Θ , µ̃ OT ( µ̃ , νθ ) subject to ‖µ− µ̃‖TV ≤ suffers from identification issues . In particular , it can not distinguish between “ clean ” distributions within TV distance of νθ0 . This makes it unsuitable as a loss function for statistical estimation , because it can not lead to a consistent estimator . However , its regularized counterpart min θ∈Θ , s OT ( µ+ s , νθ ) + λ‖s‖TV , ( 2.1 ) where λ > 0 is a regularization parameter , does not suffer from this issue . In the rest of this paper , we work with the TV-regularized formulation equation 2.1 . The main idea of our formulation is to allow for modifications of µ , while penalizing their magnitude and ensuring that the modified µ is still a probability measure . Below we formulate this intuition in an optimization problem titled ROBOT ( ROBust Optimal Transport ) : Formulation 1 : ROBOT ( µ , ν ) = minΠ∈F+ ( Rd×Rd ) s∈F ( Rd ) ∫ C ( x , y ) Π ( dx , dy ) + λ‖s‖TV subject to ∫ B×Rd Π ( dx , dy ) = ∫ B ( µ ( dx ) + s ( dx ) ) ≥ 0 ∀ B ∈ B ( Rd ) ( Borel σ-algebra ) ∫ Rd×C Π ( dx , dy ) = ∫ C ν ( dy ) ∀ C ∈ B ( Rd ) ∫ s ( dx ) = 0 . ( 2.2 ) Here F ( Rd ) denotes the set of all signed measures with finite total variation on Rd , F+ ( Rd × Rd ) is the set of all measures with finite total variation on Rd × Rd . The first and the last constraints ensure that µ + s is a valid probability measure , while λ‖s‖TV penalizes the amount of modifications in µ . It is worth noting that we can identify exact locations of outliers in µ by inspecting µ+ s , i.e . if µ ( x ) + s ( x ) = 0 , then x got eliminated and is an outlier . ROBOT , unlike classical OT , guarantees that an adversarially picked outliers can not increase the distance arbitrarily . Let µ̃ = ( 1− ) µ+ µc , i.e . µ̃ is µ contaminated with outliers from µc , and let ν be an arbitrary measure ( in MKE , µ̃ is the contaminated data and ν is the model we learn ) . Adversary can arbitrarily increase OT ( µ̃ , ν ) by manipulating the outlier distribution µc . For ROBOT we have the following bound : Theorem 2.1 . Let µ̃ = ( 1− ) µ+ µc for some ∈ [ 0 , 1 ) , then ROBOT ( µ̃ , ν ) ≤ ( OT ( µ , ν ) + λ ‖µ− µc‖TV ) ∧ λ‖µ̃− ν‖TV ∧OT ( µ̃ , ν ) . ( 2.3 ) This bound has two key takeaways : since TV norm of any two distributions is bounded by 1 , adversary can not increase ROBOT ( µ̃ , ν ) arbitrarily ; in the absence of outliers , ROBOT is bounded by classical OT . See Appendix C for the proof . Related work We note connection between equation 2.2 and unbalanced OT ( UOT ) ( Chizat. , 2017 ; Chizat et al. , 2018 ) . UOT is typically formulated by replacing TV norm with KL ( µ + s|µ ) and adding an analogous term for ν. Chizat et al . ( 2018 ) studied entropy regularized UOT with various divergences penalizing marginal violations . Optimization problems similar to equation 2.2 have also been considered outside of the ML literature ( Piccoli & Rossi , 2014 ; Liero et al. , 2018 ) . We are unaware of prior applications of UOT to outlier-robustness , but it was studied in the concurrent work of Balaji et al . ( 2020 ) . Another relevant variation of OT is partial OT ( Figalli , 2010 ; Caffarelli & McCann , 2010 ) . It may also be considered for outlier-robustness , but it has a drawback of forcing mass destruction rather than adjusting marginals to ignore outliers when they are present . A concurrent work by Staerman et al . ( 2020 ) took a different path : they replaced the expectation in the Wasserstein-1 dual with a median-of-means to promote robustness . It is unclear what is the corresponding primal , making it hard to interpret as an optimal transport problem . A major challenge with the aforementioned methods , including our Formulation 1 , is the difficulty of the optimization problem . This is especially the case for MKEs , where a transport problem has to be solved in every iteration to obtain the gradient of the model parameters . Chizat et al . ( 2018 ) proposed a Sinkhorn-like algorithm for entropy regularized UOT , but it is not amenable to stochastic optimization . Balaji et al . ( 2020 ) proposed a stochastic optimization algorithm based on the UOT dual , but it requires two additional neural networks ( total of four including dual potentials ) to parameterize modified marginal distributions ( i.e. , µ + s and analogous one for ν ) . Optimizing with a median-of-means in the objective function as in ( Staerman et al. , 2020 ) is also challenging . The key contribution of our work is a formulation equivalent to equation 2.2 , which is easily compatible with the large body of classical OT optimization techniques ( Cuturi , 2013 ; Solomon et al. , 2015 ; Genevay et al. , 2016 ; Seguy et al. , 2018 ) . More efficient equivalent formulation At a first glance , there are two issues with equation 2.2 : it appears asymmetric and it is unclear if it can be optimized efficiently . Below we present an equivalent formulation that is free of these issues : Formulation 2 : ROBOT ( µ , ν ) = minΠ∈F+ ( Rd×Rd ) ∫ Cλ ( x , y ) Π ( dx , dy ) subject to ∫ B×Rd Π ( dx , dy ) = ∫ B µ ( dx ) ∀ B ∈ B ( Rd ) ∫ Rd×C Π ( dx , dy ) = ∫ C ν ( dy ) ∀ C ∈ B ( Rd ) , ( 2.4 ) where Cλ is the truncated cost function defined as Cλ ( x , y ) = C ( x , y ) ∧ 2λ . Looking at equation 2.4 , it is not apparent that it adds robustness to MKE , but it is symmetric , easy to combine with entropic regularization by simply truncating the cost , and benefits from stochastic optimization algorithms ( Genevay et al. , 2016 ; Seguy et al. , 2018 ) . This formulation also has a distant relation to the idea of loss truncation for achieving robustness ( Shen & Sanghavi , 2019 ) . Pele & Werman ( 2009 ) considered the Earth Mover Distance ( discrete OT ) with truncated cost to achieve computational improvements ; they also mentioned its potential to promote robustness against outlier noise but did not explore this direction . In Section 3 , we establish equivalence between the two ROBOT formulations , equation 2.2 and equation 2.4 . This equivalence allows us to obtain an efficient algorithm based on equation 2.4 for robust MKE . We also provide a simple procedure for computing optimal s in equation 2.2 from the solution of equation 2.4 , enabling a new OT application : outlier detection . We verify the effectiveness of robust MKE and outlier detection in our experiments in Section 4 . Before presenting the equivalence proof , we formulate the discrete analogs of the two ROBOT formulations for their practical value .

Summary: The authors propose to address the robustness over outliers for optimal transport (OT). They propose a new formulation based on penalizing the contaminated probability measures by a signed measure (which shares a close relation with unbalanced OT). The authors further derive an equivalent formulation by adjusting the cost matrix for the corresponding standard OT. Empirically, the authors evaluate their proposed approach on a toy example of robust mean estimation and outlier detection for data collection.