Conditional Generative Quantile Networks via Optimal Transport and Convex Potentials

1 INTRODUCTION . A rudimentary result in probability theory states that applying the cumulative distribution function ( CDF ) to the underlying continuous random variable results in a uniformly random sample from the unit interval . Correspondingly , passing a uniformly random sample from the unit interval through the inverse CDF , a.k.a . the univariate quantile function , results in a random sample from the desired CDF , a simple fact that has been exploited in simulating many distributions ( Devroye , 1986 ) . Quantile functions also play a pivotal role in measuring financial risks ( Rüschendorf , 2013 ) and in building regression functions at various levels ( Koenker & Gilbert , 1978 ) and robust statistical estimators under model mis-specification ( Koenker , 2005 ) . The pin-ball loss in quantile regression has also been used in training supervised classifiers ( Huang et al. , 2014 ) and more recently in building unsupervised generative models ( Ostrovski et al. , 2018 ) . The perspective to treat quantile functions as a natural mathematical object for modern generative models is very appealing ( Parzen , 1979 ) . Traditional quantile regressors are trained at fixed quantile levels , and the entire quantile function may be obtained through interpolation or smoothing , which , however , suffers from the so-called quantile crossing problem , namely that the monotonicity property of the quantile function is often violated due to training quantile regressors at different levels independently . Despite this nuance , the quantile approach , in its the univariate form , has been successfully applied in many applications ( Koenker , 2005 ) and more recently in probabilistic forecasting ( e.g . Gasthaus et al. , 2019 ; Gouttes et al. , 2021 ) . However , the pinball loss , while effective for univariate datasets , does not extend immediately to multivariate responses , a setting of paramount practical importance and is our focus in this work . While in principle it is possible to apply quantile regression to each dimension , as in Ostrovski et al . ( 2018 ) , this naive approach ignores the high-order correlations between components and can lead to incorrectly learned models . Recently , optimal transport has garnered substantial interest in generative modelling ( Arjovsky et al. , 2017 ; Makkuva et al. , 2020 ; Finlay et al. , 2020 ; Huang et al. , 2021 ) , domain adaptation ( Courty et al. , 2017 ) , imaging ( Bonneel et al. , 2015 ) , semi-supervised learning ( Solomon et al. , 2014 ) , and much more ( Peyré & Cuturi , 2019 ) . The main goal of optimal transport is to learn a coupling between two distributions such that we can transport mass from one distribution to another at minimal cost . In fact , under mild assumptions , the optimal coupling between two continuous distributions can be obtained through the gradient of a convex function ( Brenier , 1991 ; McCann , 1995 ) . Since gradients of convex function are monotonic , and hence coincide with quantile function in the univariate setting . This fact motivates attempts , such as here , to construct multivariate quantile functions through the gradient of convex functions . Compared to other approaches for generative modelling , quantile networks have the added advantage of exact estimation of quantile levels . Indeed , a pointwise estimation of quantile levels removes the necessity for empirical quantile esimation of non-quantile based generative models through say MCMC ( Rasul et al. , 2021 ) . In this work , we seek to generalize generative quantile modelling to the multivariate setting and we extend the vector quantile regression of Carlier et al . ( 2016 ) to conditional generative quantile modelling . Our main contributions are summarized as follows , • We construct a dual formulation of correlation maximization and extend its usage to generative quantile networks that is highly theoretically motivated . • To the best of our knowledge , we are the first work to propose conditional generative modelling using a Kantorovich dual objective . • Previous works that are based on the Kantorovich dual restrict their models to input convex neural networks ( Amos et al. , 2017 , ICNNs ) . Instead , we relax this assumption by parameterizing our model as smooth neural networks and verify the sufficiency and advantage of this relaxation . • We further argue that convex potential gradients are natural quantile networks since they automatically satisfy the non-crossing property of quantile functions . In addition to the aforementioned novelties , our proposed conditional generative quantile network is also very flexible and versatile . In particular , we conduct thorough experiments to show that it can serve as a conditioner for generative tasks and can also be used for probabilistic forecasting . 2 BACKGROUND . In this section , we recall some background on generative quantile functions and optimal transport . In the univariate case , given a cumulative distribution function ( CDF ) F ( x ) = Pr ( X ≤ x ) of random variable X , the corresponding quantile function Q is defined as : ∀u ∈ ( 0 , 1 ) , Q ( u ) = F−1 ( u ) : = inf { x : F ( x ) ≥ u } . ( 1 ) While the CDF can be straightforwardly extended to the multivariate case , the quantile function , its inverse , is less obvious as the loss of the total ordering on the real line made it impossible to “ just invert ” F . Instead , we can think of the quantile functionQ as a mapping that transforms a uniformly random variable U to a sample from the CDF F of interest , be it univariate or not . This generative view of the quantile function turns out to be quite fruitful and allows us to learn distributions via quantile regression , through for instance the pinball loss ` u at a fixed quantile level u ∈ ( 0 , 1 ) : ` u ( q ; Y ) = u ( Y − q ) + + ( u− 1 ) ( Y − q ) − . ( 2 ) Indeed , for fixed u , minimizing the expected loss E [ ` u ( q ; Y ) ] yields exactly q = Q ( u ) for the quantifule function Q of Y ( Koenker & Gilbert , 1978 ) . Generative Quantile Networks . Now , to estimate the entire quantile function Q , we can simply replace the fixed quantile level u with a univariate random variable U ∼ Unif ( [ 0 , 1 ] ) . We can now minimize the expected loss E ` U ( Q ( U ) ; Y ) w.r.t . Q to learn the entire quantile function . The univariate quantile loss has been applied successfully recently in probabilistic time-series forecasting . For instance , Gouttes et al . ( 2021 ) proposed stacking an implicit quantile network on top of an recurrent neural network ( RNN ) to model the temporal dynamics of the covariates . By minimizing the quantile loss , exact point-wise quantiles can be inferred when forecasting . However , when we transition to the multivariate setting where U is uniform over the hypercube [ 0 , 1 ] d , we can no longer directly minimize the expected loss E ` U . A natural alternative is to minimize the Lp norm of the vector quantity in eq . ( 2 ) . For instance , the autoregressive implicit quantile networks ( AIQN ) of Ostrovski et al . ( 2018 ) amounts to adopting the L1 norm . Wen & Torkkola ( 2019 ) proposed to train a Gaussian-copula model jointly on top of a generative multivariate quantile model with the L1-norm quantile loss and account for higher-order correlations in a two-step procedure . Normalizing flows are a popular class of generative models that allow explicit evaluation of the likelihood and the inverse of Jacobian . In particular , Wang et al . ( 2019 ) parameterized their quantile function with an increasing triangular map and applied it to novelty detection . However , an extension to conditional quantile estimation was not considered . Optimal Transport . Consider two probability measures µ and ν on X ⊆ Rd and Y ⊆ Rm , respectively , and c : X × Y → [ 0 , +∞ ] a ( closed ) cost function . The Monge problem refers to finding a transport map G : X → Y that minimizes the total transport cost : C ( µ , ν ) : = inf G : X→Y { ∫ X c ( x , G ( x ) ) dµ ( x ) | G # µ = ν } , ( 3 ) where G # µ denotes the push-forward measure by G. An ubiquitous transport cost is the squared Euclidean distance c ( x , y ) = ‖x−y‖22 , yielding the ( squared ) 2-Wasserstein distance , i.e . W 2 ( µ , ν ) . A common issue with optimal transport in practice is its computation in high dimensions . Kolouri et al . ( 2019 ) , among many others , propose to employ random projections and to reduce to the univariate setting with provable convergence guarantees . However , the number of projections sampled from the unit hypersphere necessarily scales with the dimensions of the distribution . Alternatively , Kantorovich ’ s duality offers an objective that can be conveniently approximated by restricting the structure of dual potentials . Formally , the Kantorovich dual problem is given by sup ( ϕ , ψ ) ∈L1 ( µ ) ×L1 ( ν ) ∫ X ϕdµ ( x ) + ∫ Y ψdν ( y ) s.t . ∀x ∈ X , ∀y ∈ Y , ϕ ( x ) + ψ ( y ) ≤ c ( x , y ) ( 4 ) where ϕ ∈ L1 ( µ ) , ψ ∈ L1 ( ν ) can be chosen to be bounded and continuous ( Villani , 2008 ) . In fact , should the dual admit a solution , 12‖ · ‖ 2 2 − ϕ and 12‖ · ‖ 2 2 − ψ are closed convex functions and are Fenchel conjugates of each other , provided that c is the squared Euclidean distance . A byproduct of this is that the gradient map Id− ∂ϕ is an optimal coupling between µ and ν ( Villani , 2008 ) . Makkuva et al . ( 2020 ) proposed parameterizing 12‖ · ‖ 2 2 − ϕ as an Input Convex Neural Network ( ICNN ) ( Amos et al. , 2017 ) and optimize the Kantorovich dual for generative modelling tasks . Huang et al . ( 2021 ) utilized Brenier ’ s theorem to introduce Convex-Potential Flows ( CP-Flows ) in which they restrict their flow model with strong convexity . Thus , the flow is the convex gradient that is invertible numerically using a convex solver and they use Hutchinson ’ s trace estimator to estimate the logarithmic Jacobian . Although similar to ours , both works are not conditional and neither are they generative quantile networks since the underlying distribution is not learned through all quantiles jointly . Furthermore , we do not constrain our model to be convex . We show in our work that by optimizing smooth neural networks with our correlation maximization objective below , the dual admits a convex solution . 3 CONVEX POTENTIALS AS GENERATIVE QUANTILE NETWORKS . We propose a novel method for density estimation using a quantile approach that generalizes to the multivariate setting and allows for flexible task-dependent conditioning . Briefly , we first formulate the dual problem of correlation maximization as our multivariate quantile loss . Then , we construct an affine latent model with respect to the covariate embeddings that is highly versatile . In particular , any task-dependent deep feature extractor can be used to extract latent embeddings of the covariates and can be jointly trained with our quantile objective without any auxiliary regularization . We then provide theoretical guarantees of convergence to an optimal convex function .

Summary: This paper proposes a conditional quantile generative model using optimal transport. Theoretical results show that the optimal is convex and a conditional dual objective is further introduced. Existing results shows the efficacy and versatility of this method.