Conditional Generative Quantile Networks via Optimal Transport and Convex Potentials

1 INTRODUCTION . A rudimentary result in probability theory states that applying the cumulative distribution function ( CDF ) to the underlying continuous random variable results in a uniformly random sample from the unit interval . Correspondingly , passing a uniformly random sample from the unit interval through the inverse CDF , a.k.a . the univariate quantile function , results in a random sample from the desired CDF , a simple fact that has been exploited in simulating many distributions ( Devroye , 1986 ) . Quantile functions also play a pivotal role in measuring financial risks ( Rüschendorf , 2013 ) and in building regression functions at various levels ( Koenker & Gilbert , 1978 ) and robust statistical estimators under model mis-specification ( Koenker , 2005 ) . The pin-ball loss in quantile regression has also been used in training supervised classifiers ( Huang et al. , 2014 ) and more recently in building unsupervised generative models ( Ostrovski et al. , 2018 ) . The perspective to treat quantile functions as a natural mathematical object for modern generative models is very appealing ( Parzen , 1979 ) . Traditional quantile regressors are trained at fixed quantile levels , and the entire quantile function may be obtained through interpolation or smoothing , which , however , suffers from the so-called quantile crossing problem , namely that the monotonicity property of the quantile function is often violated due to training quantile regressors at different levels independently . Despite this nuance , the quantile approach , in its the univariate form , has been successfully applied in many applications ( Koenker , 2005 ) and more recently in probabilistic forecasting ( e.g . Gasthaus et al. , 2019 ; Gouttes et al. , 2021 ) . However , the pinball loss , while effective for univariate datasets , does not extend immediately to multivariate responses , a setting of paramount practical importance and is our focus in this work . While in principle it is possible to apply quantile regression to each dimension , as in Ostrovski et al . ( 2018 ) , this naive approach ignores the high-order correlations between components and can lead to incorrectly learned models . Recently , optimal transport has garnered substantial interest in generative modelling ( Arjovsky et al. , 2017 ; Makkuva et al. , 2020 ; Finlay et al. , 2020 ; Huang et al. , 2021 ) , domain adaptation ( Courty et al. , 2017 ) , imaging ( Bonneel et al. , 2015 ) , semi-supervised learning ( Solomon et al. , 2014 ) , and much more ( Peyré & Cuturi , 2019 ) . The main goal of optimal transport is to learn a coupling between two distributions such that we can transport mass from one distribution to another at minimal cost . In fact , under mild assumptions , the optimal coupling between two continuous distributions can be obtained through the gradient of a convex function ( Brenier , 1991 ; McCann , 1995 ) . Since gradients of convex function are monotonic , and hence coincide with quantile function in the univariate setting . This fact motivates attempts , such as here , to construct multivariate quantile functions through the gradient of convex functions . Compared to other approaches for generative modelling , quantile networks have the added advantage of exact estimation of quantile levels . Indeed , a pointwise estimation of quantile levels removes the necessity for empirical quantile esimation of non-quantile based generative models through say MCMC ( Rasul et al. , 2021 ) . In this work , we seek to generalize generative quantile modelling to the multivariate setting and we extend the vector quantile regression of Carlier et al . ( 2016 ) to conditional generative quantile modelling . Our main contributions are summarized as follows , • We construct a dual formulation of correlation maximization and extend its usage to generative quantile networks that is highly theoretically motivated . • To the best of our knowledge , we are the first work to propose conditional generative modelling using a Kantorovich dual objective . • Previous works that are based on the Kantorovich dual restrict their models to input convex neural networks ( Amos et al. , 2017 , ICNNs ) . Instead , we relax this assumption by parameterizing our model as smooth neural networks and verify the sufficiency and advantage of this relaxation . • We further argue that convex potential gradients are natural quantile networks since they automatically satisfy the non-crossing property of quantile functions . In addition to the aforementioned novelties , our proposed conditional generative quantile network is also very flexible and versatile . In particular , we conduct thorough experiments to show that it can serve as a conditioner for generative tasks and can also be used for probabilistic forecasting . 2 BACKGROUND . In this section , we recall some background on generative quantile functions and optimal transport . In the univariate case , given a cumulative distribution function ( CDF ) F ( x ) = Pr ( X ≤ x ) of random variable X , the corresponding quantile function Q is defined as : ∀u ∈ ( 0 , 1 ) , Q ( u ) = F−1 ( u ) : = inf { x : F ( x ) ≥ u } . ( 1 ) While the CDF can be straightforwardly extended to the multivariate case , the quantile function , its inverse , is less obvious as the loss of the total ordering on the real line made it impossible to “ just invert ” F . Instead , we can think of the quantile functionQ as a mapping that transforms a uniformly random variable U to a sample from the CDF F of interest , be it univariate or not . This generative view of the quantile function turns out to be quite fruitful and allows us to learn distributions via quantile regression , through for instance the pinball loss ` u at a fixed quantile level u ∈ ( 0 , 1 ) : ` u ( q ; Y ) = u ( Y − q ) + + ( u− 1 ) ( Y − q ) − . ( 2 ) Indeed , for fixed u , minimizing the expected loss E [ ` u ( q ; Y ) ] yields exactly q = Q ( u ) for the quantifule function Q of Y ( Koenker & Gilbert , 1978 ) . Generative Quantile Networks . Now , to estimate the entire quantile function Q , we can simply replace the fixed quantile level u with a univariate random variable U ∼ Unif ( [ 0 , 1 ] ) . We can now minimize the expected loss E ` U ( Q ( U ) ; Y ) w.r.t . Q to learn the entire quantile function . The univariate quantile loss has been applied successfully recently in probabilistic time-series forecasting . For instance , Gouttes et al . ( 2021 ) proposed stacking an implicit quantile network on top of an recurrent neural network ( RNN ) to model the temporal dynamics of the covariates . By minimizing the quantile loss , exact point-wise quantiles can be inferred when forecasting . However , when we transition to the multivariate setting where U is uniform over the hypercube [ 0 , 1 ] d , we can no longer directly minimize the expected loss E ` U . A natural alternative is to minimize the Lp norm of the vector quantity in eq . ( 2 ) . For instance , the autoregressive implicit quantile networks ( AIQN ) of Ostrovski et al . ( 2018 ) amounts to adopting the L1 norm . Wen & Torkkola ( 2019 ) proposed to train a Gaussian-copula model jointly on top of a generative multivariate quantile model with the L1-norm quantile loss and account for higher-order correlations in a two-step procedure . Normalizing flows are a popular class of generative models that allow explicit evaluation of the likelihood and the inverse of Jacobian . In particular , Wang et al . ( 2019 ) parameterized their quantile function with an increasing triangular map and applied it to novelty detection . However , an extension to conditional quantile estimation was not considered . Optimal Transport . Consider two probability measures µ and ν on X ⊆ Rd and Y ⊆ Rm , respectively , and c : X × Y → [ 0 , +∞ ] a ( closed ) cost function . The Monge problem refers to finding a transport map G : X → Y that minimizes the total transport cost : C ( µ , ν ) : = inf G : X→Y { ∫ X c ( x , G ( x ) ) dµ ( x ) | G # µ = ν } , ( 3 ) where G # µ denotes the push-forward measure by G. An ubiquitous transport cost is the squared Euclidean distance c ( x , y ) = ‖x−y‖22 , yielding the ( squared ) 2-Wasserstein distance , i.e . W 2 ( µ , ν ) . A common issue with optimal transport in practice is its computation in high dimensions . Kolouri et al . ( 2019 ) , among many others , propose to employ random projections and to reduce to the univariate setting with provable convergence guarantees . However , the number of projections sampled from the unit hypersphere necessarily scales with the dimensions of the distribution . Alternatively , Kantorovich ’ s duality offers an objective that can be conveniently approximated by restricting the structure of dual potentials . Formally , the Kantorovich dual problem is given by sup ( ϕ , ψ ) ∈L1 ( µ ) ×L1 ( ν ) ∫ X ϕdµ ( x ) + ∫ Y ψdν ( y ) s.t . ∀x ∈ X , ∀y ∈ Y , ϕ ( x ) + ψ ( y ) ≤ c ( x , y ) ( 4 ) where ϕ ∈ L1 ( µ ) , ψ ∈ L1 ( ν ) can be chosen to be bounded and continuous ( Villani , 2008 ) . In fact , should the dual admit a solution , 12‖ · ‖ 2 2 − ϕ and 12‖ · ‖ 2 2 − ψ are closed convex functions and are Fenchel conjugates of each other , provided that c is the squared Euclidean distance . A byproduct of this is that the gradient map Id− ∂ϕ is an optimal coupling between µ and ν ( Villani , 2008 ) . Makkuva et al . ( 2020 ) proposed parameterizing 12‖ · ‖ 2 2 − ϕ as an Input Convex Neural Network ( ICNN ) ( Amos et al. , 2017 ) and optimize the Kantorovich dual for generative modelling tasks . Huang et al . ( 2021 ) utilized Brenier ’ s theorem to introduce Convex-Potential Flows ( CP-Flows ) in which they restrict their flow model with strong convexity . Thus , the flow is the convex gradient that is invertible numerically using a convex solver and they use Hutchinson ’ s trace estimator to estimate the logarithmic Jacobian . Although similar to ours , both works are not conditional and neither are they generative quantile networks since the underlying distribution is not learned through all quantiles jointly . Furthermore , we do not constrain our model to be convex . We show in our work that by optimizing smooth neural networks with our correlation maximization objective below , the dual admits a convex solution . 3 CONVEX POTENTIALS AS GENERATIVE QUANTILE NETWORKS . We propose a novel method for density estimation using a quantile approach that generalizes to the multivariate setting and allows for flexible task-dependent conditioning . Briefly , we first formulate the dual problem of correlation maximization as our multivariate quantile loss . Then , we construct an affine latent model with respect to the covariate embeddings that is highly versatile . In particular , any task-dependent deep feature extractor can be used to extract latent embeddings of the covariates and can be jointly trained with our quantile objective without any auxiliary regularization . We then provide theoretical guarantees of convergence to an optimal convex function .

Summary: Quantile regression is frequently used as an alternative to conventional regression. An important advantage of quantile regression is that provides enough flexibility to capture the whole conditional distribution, rather than the conditional mean, of the response variable for given predictor variables. Standard approaches to tackle the estimation problem of quantile regression are based on the so-called "pinball loss" (quantile loss) in the univariate case, or an $L_p$-norm of the vector quantile loss in the multivariate case. This paper builds conditional generative quantile modelling to the multivariate setting. The authors propose convex potential quantile (CPQ), which is an optimal transport approach through a Kantorovih dual objective functional, which maximizes the correlation between the target distribution (dependent variable in the regression case) and the source distribution (uniform distribution over the unit hypercube). Various experiments are conducted on synthetic and real datasets, including generative modelling (MNIST and CelebA) and probabilistic forecasting.