Implicit Generative Copulas

Copulas are a powerful tool for modeling multivariate distributions as they allow to separately estimate the univariate marginal distributions and the joint dependency structure . However , known parametric copulas offer limited flexibility especially in high dimensions , while commonly used non-parametric methods suffer from the curse of dimensionality . A popular remedy is to construct a tree-based hierarchy of conditional bivariate copulas . In this paper , we propose a flexible , yet conceptually simple alternative based on implicit generative neural networks . The key challenge is to ensure marginal uniformity of the estimated copula distribution . We achieve this by learning a multivariate latent distribution with unspecified marginals but the desired dependency structure . By applying the probability integral transform , we can then obtain samples from the high-dimensional copula distribution without relying on parametric assumptions or the need to find a suitable tree structure . Experiments on synthetic and real data from finance , physics , and image generation demonstrate the performance of this approach . 1 Introduction . Many approaches are available to reliably model univariate distributions of real-valued variables . One can use various parametric distributions or employ flexible , non-parametric methods , e.g. , by using empirical distribution functions , kernel density estimation ( KDE ) , or quantile regression [ 22 ] . Modeling distributions in higher dimensions is a much harder task . Parametric approaches using , e.g. , Gaussian distributions , are common but inflexible . Many non-parametric approaches become infeasible in higher dimensions due to the curse of dimensionality [ 18 ] . Recent successes in modeling multivariate distributions have been achieved with deep neural networkbased likelihood-free implicit generative models [ 28 ] , such as Generative Adversarial Networks [ 13 ] ( GANs ) and Generative Moment Matching Networks ( GMMNs ) [ 10 ; 24 ] . Copulas are a tool to decouple the modeling of the univariate marginal distributions from modeling the high-dimensional joint dependency structure [ 19 ] . This allows to obtain high-quality marginal models with the mentioned univariate techniques . Moreover , the ( conditional ) marginals can be tailored to the problem at hand and many fields have developed sophisticated domain-specific univariate models for this task , e.g. , for probabilistic forecasts in finance [ 4 ] , weather [ 36 ] , or energy [ 17 ] . These models can then be combined with a suitable copula structure to enable the simulation of multivariate quantities , e.g. , [ 34 ; 29 ; 41 ] . In the machine learning literature copulas have been used to increase the flexibility of Bayesian networks [ 11 ] , to model multi-agent coordination in reinforcement learning [ 43 ] , or for image generation via the Vine Copula Autoencoder [ 40 ] . Another popular application of copulas is synthetic tabular data generation [ 33 ; 27 ] . Note that in many of these applications , the main goal is to sample from a multivariate distribution and the copula is used as a building block for a generative model . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . High-dimensional copula distributions are commonly modeled via parametric structures such as the Gaussian or the student-t copula [ 8 ] . [ 25 ] propose to enhance the subclass of Archimedean copulas by learning the generator functions via deep neural networks . However , the family of Archimedean copulas is of limited use in higher dimension as it makes restrictive symmetry assumptions . A more flexible , semi-parametric state-of-the art approach is to use vine copulas which are built from trees of bivariate pair-copulas [ 1 ] . These pair-copulas can again be parametric , such as the Gumbel or Clayton copula , or non-parametric , e.g. , by using bivariate kernel density estimation [ 32 ] . Directly applying implicit generative modeling to the task of estimating copula distributions is not straight-forward since copula distributions are required to have uniform marginals . Such an approach has been proposed by [ 23 ; 16 ] , but without ensuring the uniformity property , at least not for finite sample sizes when the model is not expected to exactly fit the true copula distribution . Ensuring the marginal uniformity of the learned copula distribution is important since deviations from uniformity will result in unwanted alterations of the marginal distributions in the data space . In this paper we show how to design and train implicit generative copula ( IGC ) models to match the dependency structure of given data . A key challenge is to ensure the marginal uniformity of the estimated distribution . We achieve this through first learning a latent distribution with unspecified marginals but the same dependency structure as the training data . We then obtain the desired copula model by applying the probability integral transform component-wise to the latent distribution . The IGC model and the data distribution are matched in copula space using the energy distance [ 39 ] . During training , the probability integral transform is approximated through a differentiable softrank layer which allows to use gradient-based methods . Our contributions . • We propose the first universal , non-parametric model for estimating high-dimensional copula distributions with guaranteed uniformity of the marginal distributions . • We show how flexible likelihood-free implicit generative models based on deep neural networks can be trained for this task through the use of a differentiable softrank layer . • Compared with the state-of-the-art semi-parametric vine copula approach , we demonstrate similar or improved performance for different tasks . We present the proposed copula model in Section 2 . Model training is described in Section 3 . Section 4 shows experimental results for synthetic and real data from finance , physics , and image generation . We conclude in Section 5 . 2 Proposed IGC Model . We introduce our IGC model following the schematic shown in Figure 1 . Figure 2 provides an exemplary application for a bivariate two-component Gaussian mixture data distribution . Copula Basics The vector-valued continuous random variable X ∈ RD represents the data source of our task . We denote its distribution by P and the cumulative distribution function ( cdf ) of X by FX ( x ) . Moreover , let FXd ( xd ) be the cdf of the marginal distribution of Xd , the d-th component of X , for d = 1 , . . . , D. Each sample vector x ∈ RD can be mapped to a vector u ∈ [ 0 , 1 ] D by defining as ud the value of xd with respect to the corresponding marginal cdf of Xd , i.e. , ud = FXd ( xd ) for all d = 1 , . . . , D. This operation is called the probability integral transform ( PIT ) . In the copula literature , values obtained by the PIT are called pseudo-observations [ 19 ] . The random variable U follows a so-called copula distribution PC by construction , i.e. , the distribution is defined on the unit space [ 0 , 1 ] D and its marginals are uniform . The joint cdf of U is typically called the copula , which we denote by FU ( u ) here . Sklar ’ s theorem states that such such a copula function exists for all random variables X and it holds that FX ( X ) = FU ( FX1 ( X1 ) , . . . , FXd ( XD ) ) , i.e. , any multivariate distribution can be expressed in terms of its marginals and its copula [ 38 ] . IGC Model In this work , we aim at defining a flexible , non-parametric model for copula distributions QCθ in high dimensions . Such a model can either be used to ( approximately ) represent the copula distribution PC of the unit space vectors U or the distribution P of the original data vectors X , in which case the U vectors have to be transformed component-wise with the inverse cdfs of the components Xd , i.e. , xd = F−1Xd ( ud ) . A flexible model class for learning high-dimensional probability distributions is the class of implicit generative models [ 28 ] . Here , latent random variables with a simple and known distribution are transformed via a parameterizable mapping , typically a deep neural network , to a complex distribution that approximates the distribution of the training data . A straight-forward application of this framework to model copula distributions , as done in [ 16 ] and [ 23 ] , is difficult . The output of the generative model can be guaranteed to lie in [ 0 , 1 ] D by applying an appropriate output layer , e.g. , using a sigmoid function . However , guaranteeing the desired uniformity of the marginal distributions is not straight-forward . We have first experimented with additional training losses that penalize deviations from marginal uniformity . However , a simpler approach without any hyper-parameters for weighting additional cost terms and with a guarantee of marginal uniformity is the following two-step procedure . In the first step , we model the distribution of the vector-valued latent random variable Y ∈ RD . To this end , we start with random variables Z ∈ RK from a simple base distribution that we choose as zero mean , unit variance Gaussian here . The tuneable map gθ : RK → RD with parameters θ is then used to transform the noise samples z to the latent samples y as y = gθ ( z ) . ( 1 ) We denote the resulting probability distribution of Y by Qθ and the marginal cdfs by FYd ( yd ) . In a second step , we transform the latent variables Y into unit space vectors V. To this end , we define component-wise for d = 1 , . . . , D vd = FYd ( yd ) . ( 2 ) This transformation step is analogous to going form X to U . It guarantees both that the sample values vd from the model lie in the unit interval [ 0 , 1 ] and that they are uniformly distributed , regardless of the distribution Qθ . We refer to the distribution of V as the model copula distribution and denote it by QCθ . Interpretation Note that the distributions Qθ and P might be very different . More precisely , the mapping ( 1 ) can result in arbitrary marginal distributions but an optimal model of the distribution P would have PC = QCθ . ( 3 ) Given a sufficiently rich function class for gθ any distribution of Y can be modeled with small error [ 26 ] . This statement naturally extends to universal approximation properties for the dependency structure of Y , i.e. , the unit space random vectors V. The approximation of the data copula distribution PC with our model copula distribution QCθ will thus be arbitrarily close if the generator function gθ is flexible enough and if the optimal θ is selected . 3 Training Procedure . Problem statement We aim at empirically estimating an IGC model QCθ from data , i.e. , we want to determine the optimal parameter vector θ given a set of N training samples x1 , . . . , xN from X . The underlying target is to optimally match the copula model QCθ to the ( typically unknown ) copula distribution PC from which the samples were generated . Parameter estimation We first transform the given data samples into unit space vectors in [ 0 , 1 ] D , before starting the actual training procedure . Since we typically do not know the exact marginal cdfs FXd of the components Xd , we estimate them from the given data . To this end , we use the empirical cdfs separately for each component and define for n = 1 , . . . , N and d = 1 , . . . , D , und = 1 N N∑ i=1 1 [ xid ≤ xnd ] . ( 4 ) Here , 1 [ A ] is the indicator function of event A , i.e. , it is 1 iff A is true . Note that this is an unbiased and consistent estimator for the true cdf [ 42 ] . We then aim at matching the copula distribution QCθ to the unit space vector samples u 1 , . . . , uN . We base the inference of our likelihood-free model on the energy distance ED2 [ 39 ] between distributions PC and QCθ , ED2 ( PC , QCθ ) = 2E∥U−V∥ − E∥U−U′∥ − E∥V −V′∥ . ( 5 ) Here , U , U′ and V , V′ are independent copies of a random vector with distribution PC and QCθ , respectively . It holds that ED2 ( PC , QCθ ) = 0 if and only if P C = QCθ [ 39 ] . In practice , we use a sample-based approximation of the energy distance . We draw M samples v1 ( θ ) , . . . , vM ( θ ) from QCθ and minimize the loss function LED2 ( θ ) = 1 NM N∑ n=1 M∑ m=1 ∥un − vm ( θ ) ∥2 − 1 2M ( M − 1 ) M∑ m=1 M∑ m′=1 ∥vm ( θ ) − vm ′ ( θ ) ∥2 . ( 6 ) Since PC is independent of θ , the second term of ( 5 ) does not need to be included . Note that the energy distance is an instance of the more general maximum mean discrepancy ( MMD ) measure [ 15 ; 37 ] . We also experimented with MMD losses based on the Gaussian kernel but found the energy distance to be more robust as it does not require to tune the kernel bandwidth . To generate samples from our copula model QCθ , we first draw samples y 1 , . . . , yM from the implicit generative model Qθ . To obtain vectors in unit space , we then again require the cdfs FYd ( yd ) of the marginal distributions of the components Yd of Y , d = 1 , . . . , D. However , these functions are not known during model training and will likely change after each gradient step . Hence , we again use the empirical distribution functions to define for m = 1 , . . . , M and d = 1 , . . . , D , vmd ( θ ) = 1 M M∑ j=1 1 [ yjd ( θ ) ≤ y m d ( θ ) ] . ( 7 ) This provides us with an unbiased estimate of the current cdf during training . However , the indicator operation in ( 7 ) is not differentiable and hence will not allow the gradients to flow through this operation . Therefore , during training , we replace ( 7 ) with a softrank layer based on a scaled sigmoid function vmd ( θ ) = 1 M ( 0.5 + M∑ j=1 1 1 + exp ( α ( ymd ( θ ) − y j d ( θ ) ) ) ) , ( 8 ) where α is a scaling constant [ 35 ] . For sufficiently large α , ( 8 ) provides a close approximation to the empirical marginal cdfs at the current training step . A larger M will result in a finer approximation of the true marginal cdfs but comes with the increased cost for computing the ranks which requires O ( M2 ) operations for all samples . Sampling from the trained model Once we have completed training and thus have fixed θ , we have also fixed the component-wise cdfs FYd ( yd ) . Now , the operation to estimate FYd ( yd ) does not need to be differentiable anymore and we can chose any available method to estimate the univariate marginal distributions based on samples from Qθ . We choose to simply draw a very large set of samples from Qθ and then use the empirical marginal cdfs , i.e. , for each sampled value ytd we store the corresponding value utd according to ( 4 ) , t = 1 , . . . , T . Given a new realization of Yd , we obtain its approximate cdf value F̂Yd ( yd ) by interpolation of the stored values . Other more compact approximations like univariate kernel density estimation would also be feasible . In sum , we obtain a sample vi from QCθ at test time by first sampling a realization z i from N ( 0 , I ) , transform it via yi = gθ ( zi ) , and then apply the component-wise PIT approximation to obtain vi = [ F̂Y1 ( y i 1 ) , . . . , F̂Yd ( y i D ) ] . Given estimates for the component-wise marginal cdfs of the data F̂X1 , . . . , F̂Xd we can derive a sample s i in data space via si = [ F̂−1X1 ( v i 1 ) , . . . , F̂ −1 Xd ( viD ) ] .

Summary: The authors propose to model general copula models by training a neural network that samples from a latent space and maps samples to the high dimensional unit cube. Fitting the model is done by transforming the generated samples to have uniform margins by empirically computing the ranks. Then, the model is optimized by minimizing the MMD distance w.r.t true uniform data samples.