Year: 2021

Abstract:
Quantile regression has a natural extension to generative modelling by leveraging a stronger convergence in point rather than in distribution. While the pinball quantile loss works in the scalar case, it does not have a provable extension to the vector case. In this work, we consider a quantile approach to generative modelling using optimal transport with provable guarantees. We suggest and prove that by optimizing smooth functions with respect to the dual of the correlation maximization problem, the optimum is convex almost surely and hence construct a Brenier map as our generative quantile network. Furthermore, we introduce conditional generative modelling with a Kantorovich dual objective by constructing an affine latent model with respect to the covariates. Through extensive experiments on synthetic and real datasets for conditional generative and probabilistic forecasting tasks, we demonstrate the efficacy and versatility of our theoretically motivated model as a distribution estimator and conditioner.

heading: 1 INTRODUCTION
text: A rudimentary result in probability theory states that applying the cumulative distribution function (CDF) to the underlying continuous random variable results in a uniformly random sample from the unit interval. Correspondingly, passing a uniformly random sample from the unit interval through the inverse CDF, a.k.a. the univariate quantile function, results in a random sample from the desired CDF, a simple fact that has been exploited in simulating many distributions (Devroye, 1986). Quantile functions also play a pivotal role in measuring financial risks (Rüschendorf, 2013) and in building regression functions at various levels (Koenker & Gilbert, 1978) and robust statistical estimators under model mis-specification (Koenker, 2005). The pin-ball loss in quantile regression has also been used in training supervised classifiers (Huang et al., 2014) and more recently in building unsupervised generative models (Ostrovski et al., 2018).
The perspective to treat quantile functions as a natural mathematical object for modern generative models is very appealing (Parzen, 1979). Traditional quantile regressors are trained at fixed quantile levels, and the entire quantile function may be obtained through interpolation or smoothing, which, however, suffers from the so-called quantile crossing problem, namely that the monotonicity property of the quantile function is often violated due to training quantile regressors at different levels independently. Despite this nuance, the quantile approach, in its the univariate form, has been successfully applied in many applications (Koenker, 2005) and more recently in probabilistic forecasting (e.g. Gasthaus et al., 2019; Gouttes et al., 2021). However, the pinball loss, while effective for univariate datasets, does not extend immediately to multivariate responses, a setting of paramount practical importance and is our focus in this work. While in principle it is possible to apply quantile regression to each dimension, as in Ostrovski et al. (2018), this naive approach ignores the high-order correlations between components and can lead to incorrectly learned models.
Recently, optimal transport has garnered substantial interest in generative modelling (Arjovsky et al., 2017; Makkuva et al., 2020; Finlay et al., 2020; Huang et al., 2021), domain adaptation (Courty et al., 2017), imaging (Bonneel et al., 2015), semi-supervised learning (Solomon et al., 2014), and much more (Peyré & Cuturi, 2019). The main goal of optimal transport is to learn a coupling between two distributions such that we can transport mass from one distribution to another at minimal cost. In fact, under mild assumptions, the optimal coupling between two continuous distributions can be obtained through the gradient of a convex function (Brenier, 1991; McCann, 1995). Since gradients
of convex function are monotonic, and hence coincide with quantile function in the univariate setting. This fact motivates attempts, such as here, to construct multivariate quantile functions through the gradient of convex functions. Compared to other approaches for generative modelling, quantile networks have the added advantage of exact estimation of quantile levels. Indeed, a pointwise estimation of quantile levels removes the necessity for empirical quantile esimation of non-quantile based generative models through say MCMC (Rasul et al., 2021).
In this work, we seek to generalize generative quantile modelling to the multivariate setting and we extend the vector quantile regression of Carlier et al. (2016) to conditional generative quantile modelling. Our main contributions are summarized as follows,
• We construct a dual formulation of correlation maximization and extend its usage to generative quantile networks that is highly theoretically motivated.
• To the best of our knowledge, we are the first work to propose conditional generative modelling using a Kantorovich dual objective.
• Previous works that are based on the Kantorovich dual restrict their models to input convex neural networks (Amos et al., 2017, ICNNs). Instead, we relax this assumption by parameterizing our model as smooth neural networks and verify the sufficiency and advantage of this relaxation.
• We further argue that convex potential gradients are natural quantile networks since they automatically satisfy the non-crossing property of quantile functions.
In addition to the aforementioned novelties, our proposed conditional generative quantile network is also very flexible and versatile. In particular, we conduct thorough experiments to show that it can serve as a conditioner for generative tasks and can also be used for probabilistic forecasting.

heading: 2 BACKGROUND
text: In this section, we recall some background on generative quantile functions and optimal transport. In the univariate case, given a cumulative distribution function (CDF) F (x) = Pr(X ≤ x) of random variable X , the corresponding quantile function Q is defined as: ∀u ∈ (0, 1),
Q(u) = F−1(u) := inf{x : F (x) ≥ u}. (1)
While the CDF can be straightforwardly extended to the multivariate case, the quantile function, its inverse, is less obvious as the loss of the total ordering on the real line made it impossible to “just invert” F . Instead, we can think of the quantile functionQ as a mapping that transforms a uniformly random variable U to a sample from the CDF F of interest, be it univariate or not. This generative view of the quantile function turns out to be quite fruitful and allows us to learn distributions via quantile regression, through for instance the pinball loss `u at a fixed quantile level u ∈ (0, 1):
`u(q;Y ) = u(Y − q)+ + (u− 1)(Y − q)−. (2)
Indeed, for fixed u, minimizing the expected loss E[`u(q;Y )] yields exactly q = Q(u) for the quantifule function Q of Y (Koenker & Gilbert, 1978).
Generative Quantile Networks. Now, to estimate the entire quantile function Q, we can simply replace the fixed quantile level u with a univariate random variable U ∼ Unif([0, 1]). We can now minimize the expected loss E`U (Q(U);Y ) w.r.t. Q to learn the entire quantile function. The univariate quantile loss has been applied successfully recently in probabilistic time-series forecasting. For instance, Gouttes et al. (2021) proposed stacking an implicit quantile network on top of an recurrent neural network (RNN) to model the temporal dynamics of the covariates. By minimizing the quantile loss, exact point-wise quantiles can be inferred when forecasting.
However, when we transition to the multivariate setting where U is uniform over the hypercube [0, 1]d, we can no longer directly minimize the expected loss E`U . A natural alternative is to minimize the Lp norm of the vector quantity in eq. (2). For instance, the autoregressive implicit quantile networks (AIQN) of Ostrovski et al. (2018) amounts to adopting the L1 norm. Wen & Torkkola (2019) proposed to train a Gaussian-copula model jointly on top of a generative multivariate quantile model with the L1-norm quantile loss and account for higher-order correlations in a two-step procedure. Normalizing flows are a popular class of generative models that allow explicit evaluation
of the likelihood and the inverse of Jacobian. In particular, Wang et al. (2019) parameterized their quantile function with an increasing triangular map and applied it to novelty detection. However, an extension to conditional quantile estimation was not considered.
Optimal Transport. Consider two probability measures µ and ν on X ⊆ Rd and Y ⊆ Rm, respectively, and c : X × Y → [0,+∞] a (closed) cost function. The Monge problem refers to finding a transport map G : X → Y that minimizes the total transport cost:
C(µ, ν) := inf G:X→Y {∫ X c(x,G(x))dµ(x)| G#µ = ν } , (3)
where G#µ denotes the push-forward measure by G. An ubiquitous transport cost is the squared Euclidean distance c(x, y) = ‖x−y‖22, yielding the (squared) 2-Wasserstein distance, i.e. W 2(µ, ν). A common issue with optimal transport in practice is its computation in high dimensions. Kolouri et al. (2019), among many others, propose to employ random projections and to reduce to the univariate setting with provable convergence guarantees. However, the number of projections sampled from the unit hypersphere necessarily scales with the dimensions of the distribution. Alternatively, Kantorovich’s duality offers an objective that can be conveniently approximated by restricting the structure of dual potentials. Formally, the Kantorovich dual problem is given by
sup (ϕ,ψ)∈L1(µ)×L1(ν) ∫ X ϕdµ(x) + ∫ Y ψdν(y) s.t. ∀x ∈ X ,∀y ∈ Y, ϕ(x) + ψ(y) ≤ c(x, y) (4)
where ϕ ∈ L1(µ), ψ ∈ L1(ν) can be chosen to be bounded and continuous (Villani, 2008). In fact, should the dual admit a solution, 12‖ · ‖ 2 2 − ϕ and 12‖ · ‖ 2 2 − ψ are closed convex functions and are Fenchel conjugates of each other, provided that c is the squared Euclidean distance. A byproduct of this is that the gradient map Id− ∂ϕ is an optimal coupling between µ and ν (Villani, 2008). Makkuva et al. (2020) proposed parameterizing 12‖ · ‖ 2 2 − ϕ as an Input Convex Neural Network (ICNN) (Amos et al., 2017) and optimize the Kantorovich dual for generative modelling tasks. Huang et al. (2021) utilized Brenier’s theorem to introduce Convex-Potential Flows (CP-Flows) in which they restrict their flow model with strong convexity. Thus, the flow is the convex gradient that is invertible numerically using a convex solver and they use Hutchinson’s trace estimator to estimate the logarithmic Jacobian. Although similar to ours, both works are not conditional and neither are they generative quantile networks since the underlying distribution is not learned through all quantiles jointly. Furthermore, we do not constrain our model to be convex. We show in our work that by optimizing smooth neural networks with our correlation maximization objective below, the dual admits a convex solution.

heading: 3 CONVEX POTENTIALS AS GENERATIVE QUANTILE NETWORKS
text: We propose a novel method for density estimation using a quantile approach that generalizes to the multivariate setting and allows for flexible task-dependent conditioning. Briefly, we first formulate the dual problem of correlation maximization as our multivariate quantile loss. Then, we construct an affine latent model with respect to the covariate embeddings that is highly versatile. In particular, any task-dependent deep feature extractor can be used to extract latent embeddings of the covariates and can be jointly trained with our quantile objective without any auxiliary regularization. We then provide theoretical guarantees of convergence to an optimal convex function.

heading: 3.1 MULTIVARIATE QUANTILES AS CONVEX POTENTIALS
text: Before we consider the multivariate case, let us first analyze univariate quantile functions to motivate the correlation maximization problem. Let QY (u) : [0, 1] → R be defined as an univariate quantile function of Y ∼ ν := Law(Y ). By definition, quantile functions are monotonically increasing so QY (u) will be correlated with u. A quantile function is thus a solution to the correlation maximization problem, as established in the following equality (Galichon & Henry, 2012):∫ 1
0
uQY (u)d(u) = max {E(UY ), U ∼ Unif([0, 1])} . (5)
Now, we are interested in generalizing this equality to the multivariate setting for Y ∈ Rd and U ∼ Unif([0, 1]d). For the remainder of this paper, µ and ν are d-dimensional distributions. Formally, we start with the definition of maximal correlation functionals that motivates our goal of using convex potential gradients to parameterize multivariate quantile functions. Recall that Lpd denotes the L p space with range space Rd.
Definition 1 (Maximal Correlation Functionals, Galichon & Henry 2012). A functional ρµ : L2d → R is called a maximal correlation functional with respect to distribution µ if for all Y ∈ L2d,
ρµ(Y ) := sup { E[U>Y ], U ∼ µ } . (6)
Should the distribution µ be absolutely continuous w.r.t. the Lebesgue measure, then there exists a closed convex function f : Rd → R such that Y = ∇f(U) µ−almost surely (Villani, 2008). Indeed, it is clear that under these mild assumptions on µ, we can establish a multivariate extension of (5) using the convex potential gradient of f . Furthermore, ∇f achieves an optimal coupling between µ and ν (Huang et al., 2021) such that ρµ(X) = { E[U>∇f(U)], U ∼ µ } . By Brenier’s theorem, the existence and uniqueness of the convex potential gradient map between µ and ν can be established.
Theorem 1 (Brenier 1991). If Y is a squared-integrable random vector inRd, there is a unique map of the form T = ∇f for some convex function f such that ∇f#µ = ν.
In other words, we say that∇f is a Brenier mapping between µ and ν. Now, we are ready to define the generalized quantile function as a convex potential gradient.
Definition 2 (Convex Potential Quantile, Chernozhukov et al. 2017; Hallin et al. 2021). Let f : Rd → R be a closed convex function. Assume µ is absolutely continuous on Rd w.r.t. the Lebesgue measure. Then, the Convex Potential Quantile of Y ∼ ν is defined as Y = ∇f(U) where U ∼ µ := Unif([0, 1]d), i.e. ∇f#µ = ν.
Having adopted the above convex potential quantile function as a natural generalization to the multivariate setting, we now turn to its estimation. It turns out that directly addressing the maximal correlation functional is a bit challenging. Instead, we resort to its dual problem to learn the Brenier mapping ∇f . Revisiting the primal where we have also introduced a covariate X:
sup{E(U>Y ), U ∼ µ,E(X|U) = E(X) = 0}. (7)
Here the mean-independence constraint is added to decorrelate U with the covariate X . The dual is derived as the following:
inf (ϕ,ψ,b) ∫ [0,1]d ϕdµ(U) + ∫ Rd ψdν(Y ) s.t. ϕ(U) + ψ(Y ) ≥ U>Y (8)
E(X|U) = E(X) = 0, (9)
where ϕ is smooth. We define ψ as the Legendre transformation of ϕ,
ψ(y) := sup U∈[0,1]d
{U>Y − ϕ(U)}, (10)
to reduce the problem down to optimizing only ϕ and b as we expect the optimum ϕ to satisfy ϕ∗ = ψ. By complementary slackness, the inequality in eq. (8) becomes equality for optimal U :
ϕ(U) + ψ(Y ) = U>Y. (11)
By differentiating eq. (11) w.r.t U , we arrive at Y = ∇ϕ(U). However, this alone is not sufficient for characterizing the existence and convergence of smooth function ϕ to the optimal convex potential gradient that couples µ and ν. In section 4, we provide theoretical justification and necessary conditions for existence and convergence.

heading: 3.2 CONDITIONAL GENERATIVE QUANTILE NETWORKS
text: Suppose an explanatory vector X ∈ Rm is given, and we are interested in generating samples Y given X . Indeed, the target distribution is now the joint distribution of (X,Y ), ν := Law(X,Y ).
Carlier et al. (2016) have shown that a convex function can be approximated via an affine model. Thus, we parameterize our generative quantile network as an affine latent model,
ϕX(U) = ϕ(U) + b(U) >f(X) (12)
where ϕ(U) : Rd → R and b(U) : Rd → Rn are smooth functions and f : Rm → Rn. We parameterize ϕ, and b as neural networks with smooth activation functions while f is any taskspecific mapping to yield latent vector f(X). By learning a non-linear transformation of X , highorder moments between components of X can be modelled, thereby leading to a more expressive conditional generative quantile model.
Practically, dealing with uniform samples from the unit hyper-cube is less convenient computationally, due to its bounded domain. Empirically we also found the model to produce lackluster generated samples. Instead, we pre-process the uniform samples using a bijection Φ−1 : [0, 1]d → Rd, notably the inverse CDF of a standard normal applied component-wise. Now, the support of the model is unrestricted, which is easier to handle computationally.
In the conditional setting parameterized via the affine latent model eq. (12), problem (8) from the previous section now becomes
inf (ϕ,b) ∫ Rd ϕdµd(U) + ∫ Rd sup U∈Rd {U>Y − ϕ(U)− b(U)>f(X)}dνd(Y ), (13)
subject to zero-mean decorrelation constraints in eq. (9). The constraint can be enforced by applying batch normalization to the f(X) vector without the translation term such that the empirical batch mean of f(X) is zero-centered. We also find that enforcing this constraint is necessary for numerical stability and theoretical guarantees. Moreover, the supremum is computed across the batch. As we show in the following section, the dual admits a solution such that ϕX(U) is convex almost surely with respect to U .

heading: 4 THEORETICAL ANALYSIS
text: Existing work in generative modeling that optimize the Kantorovich dual problem rely on parameterizing their network as an input convex neural network. Such a parameterization greatly limits the
architecture and expressivity of the model. We suggest that a convex parameterization of the model is not necessary when optimizing the dual problem of our correlation maximization primal. Rather, the model only needs to be smooth which can easily be achieved by using a smooth activation function. The following theorems prove that the solution to the dual problem will be convex, and the gradients of the model w.r.t. U is the generative model. Theorem 2 (Carlier et al. 2017). Assume U ∈ Rd is random vector with distribution µ, (X,Y ) ∈ Rm ×Rd is random vector with joint distribution ν. Furthermore, assume E(X|U) = E(X) = 0. If there exists smooth function ϕ : Rd → R and smooth function b : Rd → Rm such that ϕx(u) = ϕ(u) + b(u)>x is convex for Law(X)-almost every value of x such that Y = ∇ϕ(U) +∇b(U)>X , then U solves the correlation maximization problem (7).
The following theorem suggests the existence of a solution to the dual problem (13) under mild assumptions on ν. Theorem 3 (Carlier et al. 2017). Let ν be an absolutely continuous probability measure overRm× Rd with density g. Assume the support of ν is Ω̄ where Ω is an open bounded convex subset of Rm ×Rd, and g is bounded on Ω and bounded away from zero on compact subsets of Ω. Then, the dual problem admits at least one solution.
Theorem 4 (Carlier et al. (2017)). Let U ∈ Rd be a solution to (7) and let Ψ : Rm × Rd → R, ϕ : Rd → R, b : Rd → Rm be solutions to the corresponding dual problem (13). Let ϕx(t) = ϕ(t) + b(t)>x ∀(t, x) ∈ [0, 1]d × support(Law(X)). Then, ϕX(U) = ϕ∗∗X (U) and U ∈ ∂ϕ∗X(Y ) almost surely.
By the Fenchel-Moreau theorem (Rockafellar, 1970) and Theorem 4, ϕX(U) is convex almost surely. To summarize, the aforementioned theorems establish the existence of a solution to (13), and further indicate that the gradient of the optimal convex function in the dual objective solves the primal correlation maximization problem. Lastly, Brenier’s theorem establishes the foundation for parameterizing our generative model as a convex potential gradient.
Hence, the transported mass from u ∈ Rd given x ∈ Rm to y ∈ Rd is given by y = ∇ϕx(u) = ∇ϕ(u) + ∇b(u)>x. Due to convexity of ϕx w.r.t. u, ∇ϕx is comonotonic thereby automatically satisfies the non-crossing property of quantile networks. Proposition 1. If ϕx(U) : Rd → R is convex w.r.t. U ∈ Rd and Uk ≤ Vk, then [∇ϕx(U)]k ≤ [∇ϕx(V )]k for all k = 1, . . . , d.
Therefore, convex potential gradients serve as a natural parameterization of quantile networks as additional constraints to enforce the non-crossing property of quantile functions are not needed. Furthermore, we prove that any conditional quantile functionQx(u) can be reasonably approximated by the affine parameterization in eq. (12).
Proposition 2. For any continuous conditional quantile function Qx(u) we can find large n and functions f(x) : Rm → Rn, ϕ(u) : Rd → R and b(u) : Rd → Rn such that the gradient of ϕx(u) := ϕ(u) + b(u) >f(x) approximates Qx(u) uniformly over any compact region of (u, x).
In our experiments we find that setting n = m suffices to obtain reasonable results. It would be interesting to explore in future work when and how the approximation can be substantially tightened for certain classes of conditional quantiles and network architectures for parameterizing b and f .

heading: 5 EXPERIMENTS
text: In this section, we evaluate our proposed approach to conditional generative modeling on a variety of experiments. We first evaluate the efficacy of our proposed conditional generative quantile network as a general conditional generative model on synthetic data, MNIST (LeCun et al., 1998), and CelebA (Liu et al., 2015). Then, we evaluate the model as a conditional quantile network for probabilistic time-series forecasting on UCI Appliances Energy and Google Stocks data. The architecture we use for all experiments for our generative quantile network is smooth feed-forward neural networks to parameterize both ϕ and b. The activation of choice must be smooth, so we choose to use CELU activation,
CELU(x) = max(0, x) + min(0, α(exp ( x α ) − 1)). (14)
for some α > 0. Unless explicitly stated, ϕ has 3 layers while b has 2 layers. Furthermore, we define f(X) = X unless explicitly stated. For evaluation, we compute the max and mean absolute error (MaxAE and MeanAE), the 50th and 90th quantile loss (QL50 and QL90), the root mean squared error (RMSE), and the symmetric mean absolute percentage error (sMAPE). Full details on implementation details can be found in Appendix appendix B.2. The code for our method and all experiments will be made publicly available upon acceptance.

heading: 5.1 TOY EXAMPLES
text: The efficacy of our proposed method as a quantile function and conditional generative model are first examined on synthetic data. We consider fitting our model to learning a quantile curve, and then evaluate density estimation on 2D distributions with class-labels.

heading: 5.1.1 FITTING GAUSSIAN MIXTURE MODEL QUANTILE FUNCTION
text: To evaluate our method as a quantile function estimator, we generate 10, 000 time-series samples of 24 i.i.d. points following a Gaussian Mixture Model GMM(π,µ,σ) with π = [0.3, 0.4, 0.3]>,µ = [−3, 0, 3]>, and σ = [0.4, 0.4, 0.4]. We train the model for 10 epochs and provide an evaluation against IQN (Gouttes et al., 2021) and Spline Quantile RNN (Gasthaus et al., 2019). Figure 2.a shows the learned quantile function of the three methods relative to the theoretical quantile curve. Performance metrics are displayed on Table 1 in fitting the quantile function of GMM(π,µ,σ), where our CPQ consistenly outperforms the two SOTA baselines across a wide variety of metrics.

heading: 5.1.2 2D DISTRIBUTIONS WITH LABELS
text: Here, we evaluate the efficacy of our model to generate synthetic labelled 2D distributions conditionally and unconditionally. The three densities we use are Eight Gaussians, Half Moons. and Swirls. The Eight Gaussians dataset does not distinguish points based on label, thus we use this dataset to evaluate the efficacy of our method first as an unconditional density estimator. For conditional density estimation, we turn to Half Moons. and Swirls. Each point in the Half Moons dataset is assigned to one of two classes, while each point in the Swirls dataset is assigned to one of three classes. As such, ϕ : R2 → R2 for both datasets, but b maps to R2 for Half Moons and R3 for Swirls with spt(b) = R2 for both. Figure 2.b shows the generated distributions qualitatively.

heading: 5.2 CONDITIONAL DIGIT GENERATION ON MNIST
text: To further examine the ability of our method to learn distributions conditionally, we test our conditional quantile network to generate handwritten digits conditionally. The MNIST Handwritten Digits dataset contains 60, 000 gray-scale images of handwritten digits at a resolution of 28 × 28px2. We consider two scenarios; (1) use a variational auto-encoder (Kingma & Welling, 2014) for dimensionality reduction and generate on a latent space (Figure 3.b), and (2) directly generate on the image space (Figure 3.a). The first scenario maps U ∈ RD to Y ∈ RD where D is the dimension of the latent space while the second scenario involves mapping U ∈ R784 to Y ∈ [0, 1]784. Clearly, the second case is a greater challenge than the first as often we set D << 784 and the encoder and decoder can be trained to implicitly filter out noise. However, we show that by directly generating on the image space, high quality images of the corresponding digit conditioned can be obtained, albeit slightly noisier than the auto-encoder counterpart. Nevertheless, we show that our proposed method can generate high-quality samples in strongly correlated high-dimensional spaces. Moreover, using a dimensionality reduction algorithm such as VAE can improve visual quality of the generated images by implicitly learning to filter out noise.

heading: 5.3 MULTI-ATTRIBUTE GENERATION ON CELEBA
text: Unlike MNIST, CelebA samples have multiple attribute labels rather than a single class label. We consider the model’s ability to learn multi-attribute combinations and furthermore generate novel
combinations of attributes. The CelebA dataset contains 202,599 RGB images of celebrity portraits aligned and cropped to 64 × 64px2. Each image corresponds with a 40−dimensional multi-hot vector that summarizes features of the celebrity portrait such as gender, hair-color, and whether the entity is wearing eyeglasses to name a few. A VAE was employed for dimensionality reduction thus our conditional quantile network maps to the latent space. Figure 3.c shows the results of novel attribute combination generation qualitatively.

heading: 5.4 PROBABILISTIC FORECASTING ON MULTIVARIATE TIME SERIES
text: Learning the entire distribution rather than predicting point-wise trajectories of trends is valuable and practical in domains like stock market analysis and demand forecasting (Wen & Torkkola, 2019). Thus, we evaluate our method on probabilistic forecasting using the UCI Appliances Energy dataset and Google Stocks data. The UCI Appliances Energy dataset contains 28 temporal continuous valued attributes. The Google Stocks dataset contains 6 temporal features gathered from 2004 to 2019. For each dataset, we generate time-series samples of 23 contiguous time points with a prediction window of 1. We compare our method against other competitive probabilistic time-series prediction methods. We use an 2-layer LSTM as the temporal feature extractor, f , that yields latent embeddings of the temporal covariates. For consistency, DeepAR (Gregor et al., 2014), RealNVP + RNN (Rasul et al., 2021), Huber QL, and CPQ all utilize the same backbone LSTM with same hyper-parameters. Comparisons are summarized in Table 2, where our CPQ yields comparable or superior performance and indicates more robustness across datasets and metrics than baselines.
TimeGAN (Yoon et al., 2019) and RealNVP+RNN cannot be probed to extract the 50th and 90th quantile levels without empirical estimation through MCMC which is computationally infeasible. Here, it is clear the advantage of a quantile approach to distribution estimation as the exact pointwise quantile level can be extracted in a single forward-pass by exploiting a comonotonic paramaterization of the density estimator.

heading: 6 CONCLUSION
text: We have proposed a novel method for conditional generative quantile modelling. In particular, we construct a dual formulation of the correlation maximization problem subject to the zero meanindependence constraint along with an affine latent parameterization. Furthermore, we prove that by optimizing smooth neural networks, the optimum is convex almost surely. As such, we construct a Brenier map as the gradients of the affine latent model. Through experiments on synthetic data, MNIST, CelebA, and time series forecasting datasets, we demonstrate our method is competitive and versatile as a distribution estimator and conditioner. For future work, we would like to explore higher order parameterizations as well as other optimal transport formulations. Applications to natural language modelling and genomic data would also be of great interest.

Review summaries:
Quantile regression is frequently used as an alternative to conventional regression. An important advantage of quantile regression is that provides enough flexibility to capture the whole conditional distribution, rather than the conditional mean, of the response variable for given predictor variables. Standard approaches to tackle the estimation problem of quantile regression are based on the so-called "pinball loss" (quantile loss) in the univariate case, or an $L_p$-norm of the vector quantile loss in the multivariate case. This paper builds conditional generative quantile modelling to the multivariate setting. The authors propose convex potential quantile (CPQ), which is an optimal transport approach through a Kantorovih dual objective functional, which maximizes the correlation between the target distribution (dependent variable in the regression case) and the source distribution (uniform distribution over the unit hypercube). Various experiments are conducted on synthetic and real datasets, including generative modelling (MNIST and CelebA) and probabilistic forecasting.

This paper estimates conditional quantile contours of a multivariate random vector. As shown by previous papers, his problem can be reformulated as a correlation maximization problem, and its solution exists and can be represented as the differential of a potential function by Brenier’s theorem. Solution can be found by considering the dual problem, as also shown by previous papers. This paper claims to parameterize the potential function by neural networks, which leads to a new method.

This paper proposes a conditional quantile generative model using optimal transport. Theoretical results show that the optimal is convex and a conditional dual objective is further introduced. Existing results shows the efficacy and versatility of this method. 

That paper proposes a multivariate quantile network that may be used for generative purposes. Its contribution is to generalize some scalar characterization of quantile functions (correlation maximization) to the multivariate case and propose a dual formulation of the problem that is advocated as generalizing well. Experiments are very limited, although somewhat convincing.