Implicit Generative Copulas
Authors: Tim Janke
Mohamed Ghanmi
Florian Steinke
Venue: NeurIPS 2021 Poster (2021)
Keywords: copulas
implicit generative models
generative neural networks

Abstract: Copulas are a powerful tool for modeling multivariate distributions as they allow to separately estimate the univariate marginal distributions and the joint dependency structure. However, known parametric copulas offer limited flexibility especially in high dimensions, while commonly used non-parametric methods suffer from the curse of dimensionality. A popular remedy is to construct a tree-based hierarchy of conditional bivariate copulas.
In this paper, we propose a flexible, yet conceptually simple alternative based on implicit generative neural networks.
The key challenge is to ensure marginal uniformity of the estimated copula distribution.
We achieve this by learning a multivariate latent distribution with unspecified marginals but the desired dependency structure.
By applying the probability integral transform, we can then obtain samples from the high-dimensional copula distribution without relying on parametric assumptions or the need to find a suitable tree structure.
Experiments on synthetic and real data from finance, physics, and image generation demonstrate the performance of this approach.

Implicit Generative Copulas
Tim Janke, Mohamed Ghanmi, Florian Steinke Energy Information Networks and Systems
TU Darmstadt, Germany
{tim. janke), (florian -aammntt
Abstract
Copulas are powerful tool for modeling multivariate distributions as they allow separately estimate the univariate marginal distributions and the joint dependency structure. However, known parametric copulas offer limited flexibility especially in high dimensions, while commonly used non-parametric methods suffer from the curse of dimensionality, A popular remedy is to construct a tree-based hierarchy conditional bivariate copulas. In this paper, propose flexible, yet conceptually simple alternative based on implicit generative neural networks. The key challenge is to ensure marginal uniformity of the estimated copula distribution. We achieve this by learning a multivariate latent distribution with unspecified marginals but the desired dependency structure. By applying the probability integral transform, we can then obtain samples from the high-dimensional copula distribution without relying on parametric assumptions Experiments on synthetic and real data from finance, physics, and image generation demonstrate the performance of this approach.
the need to find a suitable tree structure.
Introduction
Many approaches are available to reliably model univariate distributions of real-valued variables. One can use various parametric distributions or employ flexible, non-parametric methods, e.g., by using empirical distribution functions, kernel density estimation (KDE), quantile regression [22]. Modeling distributions higher dimensions is a much harder task. Parametric approaches using, e.g., Gaussian distributions, common but inflexible. Many parametric approaches become infeasible in higher dimensions due to the curse of dimensionality [18].
Recent successes in modeling multivariate distributions have been achieved with deep neural network- based likelihood-frer implicit generative models 22), such as Generative Adversarial Networks (GANs) and Generative Moment Matching Networks (GMMNs) [10 24).
Copulas tool to decouple the modeling of the univariate marginal distributions from modeling the nigh-dimensional joint dependency structure 19 This allows to obtain high-quality marginal models with the mentioned univariate techniques. Moreover, the (conditional) marginals can be tailored to the problem at hand and many fields have developed sophisticated domain-specific univariate models for this task, e.g., for probabilistic forecasts finance 4. weather (36), or energy [17]. These models can then be combined with a suitable copula structure to enable the simulation of multivariate quantities, e.g., (34] 29 4T the machine learning literature copulas have been used increase the
flexibility of Bayesian networks [1I, to model multi-agent coordination in reinforcement learning 43). or for image generation via the Vine Copula Autoencoder [40). Another popular application of copulas is synthetic tabular data generation [33 27 Note that in many of these applications, the main goal is to sample from a multivariate distribution and the copula used as a building block for generative model.
35th Conference on Neural Information Processing Systems (NcurIPS 2021), virtual.

---

High digeh-iimesional copula distributions are commonly modeled via parametric structures such as the Gaussian or the student-t copula [8. (25 propose to enhance the subclass of Archimedean copulas by learning the generator functions via deep neural networks. However, the family of Archimedean copulas is of limited use in higher dimension as makes restrictive symmetry assumptions. A more flexible, coii-paraettri state-of-the art approach is to use vine copulas which are built from trees of bivariate pair-copulas []] These pair-copulas can again be parametric, such as the Gumbel or Clayton copula, or non-parametric e.g., by using bivariate kernel density estimation [32].
Directly applying implicit generative modeling to the task of estimating copula distributions is not straight- forward since copula distributions are required to have uniform marginals. Such an approach has been proposed by [6). but without ensuring the uniformity property. at least not for finite sample sizes when the model is not expected to exactly fit the true copula distribution. Ensuring the marginal uniformity of the learned copula distribution important since deviations from uniformity will result in unwanted alterations of the marginal distributions in the data space.
In this paper we show how to design and train implicit generative copula (IGC) models to match the dependency structure of given data. A key challenge is to ensure the marginal uniformity of the estimated distribution. We achieve this through first learning latent distribution with unspecified marginals but the same dependency structure as the training data. We then obtain the desired copula model by applying the probability integral transform component-we IGC model and the data distribution are matched in copula space using the energy distance [39]. During training, the probability integral transform approximated through differentiable softrank layer which allows to use wradient-based methods.
to the latent distribution. The
Our contributions
We propose the first universal, non-parametric model for estimating high-dimensional copula distributions with guaranteed uniformity of the marginal distributions.
We show how flexible likelihood-free implicit generative models based on deep neural networks can be trained for this task through the use of differentiable softrank layer.
Compared with the state-of-the-aas semi- camimmaraettric vine copula approach, we demonstrate similar or improved performance for different tasks.
We present the proposed copula model Section 2 Model training is described in Section 3
Section 4 shows experimental results for synthetic and real data from finance, physics, and image generation. We conclude in Scction 3
2 Proposed IGC Model
We introduce our IGC model following the schematic shown in Figure [ exemplary application for a bivariate two-component Gaussian mixture data distribution.
Figure 2 provides an
Copula Basics The vector-valued continuous random variable X E TRD represents the data source of our task. We denote its distribution by P and the cumulative distribution function (edf) of X by (x). Moreover, let Fx,, (Ia) be the edf of the marginal distribution of Xa. the d-th component of X, for d......11
Each sample vector x E RDD can be mapped to a vector u E [0, 1 by defining as us the value of Ia with respect to the corresponding marginal edf of Xa. i.e., Udd Fx. (xa) for all d 1,..., D. This operation is called the probability integral transform (PIT). In the copula literature, values obtained by the PIT called pseudo-observations [99 The random variable U follows so-called copula distribution pC by construction, i.e., the distribution is defined on the unit space [0. 1]2 and its marginals
uniform. The joint edf of U is typically called the copula, which we denote by Fo (u) here. Sklar's 's theorem states that such such a copula function exists for all random variables X and holds that (X) Fu (XI)..... .., Fx, (Xp)), i.c., any multivariate distribution can be expressed in terms of its marginals and its copula (38]
IGC Model In this work, we aim at defining flexible, non-parametric model for copula distri- butions Qh in high dimensions. Such model can either be used to (approximately) represent the

---

noise
latent space
unit space
data space
Va Fr, (Y6a
M/t (tte)
Z-
Zz
Y-QQ
V-Q6
U-PC
X-P
(Pa)
Fige (Maz)
Figure I: Illustration of implicit generative copula (IGC) concept: We use generative neural network go with learnable parameters 0 to map samples from a multivariate standard Normal distribution to samples of the latent distribution Qa. The marginal edfs Fy, of this distribution then used to generate samples of the copula distribution Q6 The model trained to match copula distribution the real data pc using the energy distance.
(a) Qo
(b) Q6
(c) pC
(d)
Figure 2: Contour plots and marginal densities of all data and model distributions mentioned in Figure 1 for the example of awo-dimensional mixture of Gaussians data distribution P. Contour plots derived via KDE from of 10000 samples.
copula distribution PC of the unit space vectors U or the distribution P of the original data vectors X, in which case the U vectors have to be transformed component-vee with the inverse edfs of the components Xa. i... 2a = [xx] (ua).
A flexible model class for learning high-dimensional probability distributions is the class of implicit generative models 22 Here, latent random variables with simple and known distribution are transformed via parameterizable mapping, typically a deep neural network, to complex distribu- tion that approximates the distribution of the training data. A sraight-forwaat application this framework to model copula distributions, done in [16) and [23], is difficult. The output of the generative model can be guaranteed to lie [0, 1] D by applying an appropriate output layer, e-g., using sigmoid function. However, guaranteeing the desired uniformity of the marginal distributions not traight-forwarte We have first experimented with additional training losses that penalize deviations from marginal uniformity. However, simpler approach without any hyper-parameters for weighting additional cost terms and with a guarantee of marginal uniformity is the following two-step procedure.
In the first step, we model the distribution of the vector-valued latent random variable Y R. To this end, we start with random variables Z E I2 from simple base distribution that we choose as zero mean, unit variance Gaussian here. The tuneable map go R E. with parameters is then used to transform the noise samples z to the latent samples as
ge(z).
(1)
We denote the resulting probability distribution Y by Qo and the marginal edfs by Fy, (ya)-
In a second step, we transform the latent variables Y into unit space vectors V. To this end, we define component-wise for 1,...,D
Rd = FY,
This transformation step is analogous to going form to U. guarantees both that the sample values va from the model lie in the unit interval [0, 1] and that they are uniformly distributed, regardless of the distribution Qo. We refer to the distribution of V as the model copula distribution and denote by Qo

---

Interpretation Note that the distributions Qo and P might be very different. More precisely, the mapping ① can result in arbitrary marginal distributions but an optimal model of the distribution P would have
PC =
Given sufficiently rich function class for ge any distribution Y can be modeled with small error 226 This statement naturally extends to universal approximation properties for the dependency structure of Y, i.e., the unit space random vectors V. The approximation of the data copula distribution pC with our model copula distribution O will thus be arbitrarily close if the generator function ga is flexible enough and if the optimal is selected.
Training Procedure
Problem statement We aim empirically estimating IGC model Og from data, i.e., want to determine the optimal parameter vector 0 given set of N training samples x'....... .....x" from X. The underlying target is to optimally match the copula model Q3 to the (typically unknown) copula distribution PC from which the samples were generated.
Parameter estimation We first transform the given data samples into unit space vectors in [0, 1]?, before starting the actual training procedure. Since we typically do not know the exact marginal cdfs Fx, of the components Xa, estimate them from the given data. To this end, we use the empirical edfs separately for each component and define for I....,N N and I.
N
1 S x
(4)
2=1
Here, 1 [[4] is the indicator function of event A, i.e., is 1 iff A is true. Note that this is an unbiased and consistent estimator for the true edf (42.
We then aim at matching the copula distribution Q to the unit space vector samples u'...., u We base the inference of our likelihood free model on the energy distance 2 [39 between distributions pC and Q
EPC .Q8) - 2E VII E U'l E - V/I.
(5)
Here, U, U' and V, V are independent copies of random vector with distribution PC and Qg respectively. holds that EDa ,6)) 0 and only if PC Q [3
In practice, we use sample-base approximation of the energy distance. We draw M samples vM (0) from Q8 and minimize the loss function
M Σ Σ
M
M
(0) = NM
lu- v (0)lla 2M(M -
Σ
Σ
-0)
v'' (0) (6)
#=1 m=1
mE m'==1
Since PC is independent of 0, the second term of (5) does not need to be included. Note that the energy distance is an instance of the more general maximum mean discrepancy (MMD) measure We also experimented with MMD losses based on the Gaussian kernel but found the energy distance to be more robust as does not require to tune the kernel bandwidth.
15 37.
To generate samples from our copula model Q we first draw samples y'....,y" yM from the implicit generative model Qo. To obtain vectors in unit space, we then again require the edfs Fy, (ya) of the marginal distributions of the components Y, of Y, d = 1.... D. However, these functions are not known during model training and will likely change after each gradient step. Hence, we again use the empirical distribution functions to define for
.,. M and d = 1,..., D,
(0)
(0)
M Σ
[v) < v (0))
This provides us with an unbiased estimate of the current edf during training. However, the indicator operation in (7) is not differentiable and hence will not allow the gradients to flow through this

---

operation. Therefore, during training, we replace function
7 with softrank layer based on a scaled sigmoid
M
Ud = M
0.5
0.5
Σ
1
(8)
j=i 1+ exp(a())(
where a is scaling constant [35] For sufficiently large a, (8) provides a close approximation the empirical marginal edfs at the current training step. A larger M will result in a finer approximation of the true marginal edfs but comes with the increased cost for computing the ranks which requires ((22) operations for all samples.
Sampling from the trained model Once we have completed training and thus have fixed 0, have also fixed the component-wise edfs Fy, (ya). Now, the operation to estimate Fy, (y4) does not need be differentiable anymore and we can chose any available method estimate the univariate marginal distributions based on samples from Qe- We choose to simply draw a very large set of samples from Qe and then use the empirical marginal cdfs. i.e.. for each sampled value yy we store the corresponding value ut according to 4 f I....,' T, Given a new realization of Ya. we obtain its approximate edf value Fy, (ya) by interpolation of the stored values. Other more compact approximations like univariate kernel density estimation would also be feasible.
In sum, we obtain a sample v² from Q6 at test time by first sampling realization z' from (0. I), transform it via y = (z()),, and then apply the component-wise PIT approximation to obtain v = [n ( (wb)). Given estimates for the component-wise marginal cdfs of the data Fx,,.... ,,,, Fx, we can derive a sample s in data space via s² [xx] (xf) (uf),.... Fx, (ub)).
4 Experiments
In the following we empirically demonstrate the capabilities of IGC models on series of experiments on synthetic and real data with increasing complexity. Additional experiments can be found Appendix
Implementation For all experiments except the image generation task, we use a fully connected neural network with two layers, 100 units per layer, ReLU activation functions, and train for 500 epochs. For the image generation experiment we use a three layer, fully connected neural network with 200 neurons in each layer, and train for 100 epochs. In all cases we train with batch size of Nootch = 100 and generate M = 200 samples from the model per batch. The number of noise distributions set as K 3D and T 10 We use the Adam optimizer (00 with default parameters.
All experiments besides the training of the autoencoder models were carried out desktop PC with Intel Core 17-7700 3.60Ghz CPU and 8GB RAM. For the training of the autoencoders we used Google Colab 14. Training times for all experiments are in the range of few minutes except for the image generation task. Details are provided in Appendix B, We use tensorflow with the Keras API for the neural networks 2 77. For copula modeling, we use pyvinecopulib [31] in Python and the R
package kdecopula 30 Our code is available from https:://githttt www. htt//TimmJJaneee igg
Evaluation We evaluate the models by comparing the distance between the learned and the true copula. More specifically, we use the integrated squared error (ISE) in unit space, i.e.,
N
ISE = / J0.11 (Fu(w) Fv(w)) = - (Fu(")))-
N
1=1
where Fu the joint edf of U, the true copula, and Fv the joint edf of the learned model. Since analytical integration is not possible, we approximate the integral with an empirical sum and use the data vectors u ...,,,, N for this purpose.
The copula function Fo is not known for real data. Instead, we use the empirical cdf of the unit space data vectors in 9. i.e., for E [0, 112 we set
N
D
Fu(w)
1
EII
1 S wa].
(10)
d=1
5

---

parametric
beta
2.0
1.5
Student-t
Cleyton
Gumbel
Gaussian miture
Figure 3:
Box plot for the integrated squared error (lower is better) for 25 runs fitting bivariate copulas to training data from Student-t, Gumbel, Clayton, and Gaussian mixture copula.
Due to the curse of dimensionality this approximation of the joint edf will have coarse structure in high dimensions. This is why we only use the values at the data vectors u² u'..... u in (9)
For parametric copula models the distribution function I'y is analytic. For the non-parametric models, including IGC, we again resort to approximation. We draw 10 samples from the copula distribution model and use their empirical cdf in unit space in (9). We found that this procedure provided reliable and stable estimates for the underlying copula in dimensions D <
4.1 Synthetic data
Learning bivariate parametric copulas We first show that IGC models are able to emulate bivariate parametric copulas. We compare our IGC approach two parametric copula density estimation techniques, kernel density estimation with beta kernels (BETA)
and the transformation non-parametric
local likelihood estimator (TL.L) [12]. These are state-of-the art non- parametric estimators for pair-
copulas [30]. The compact support of the beta distribution on [0, 1 is convenient for copula density estimation. The idea the TLL approach is first transform the data using the inverse normal CDF such that the data is supported on R²2 Then the density is estimated via local regression using linear (TLLI) or cubic polynomials (TLL2) on fixed grid of points. Finally the estimated density transformed back to unit space. Additionally we report the performance of parametric model. In this approach copula family is selected from set of copulas based on the Bayesian information criterion (BIC). This set comprises the Independence, Gaussian, Student-t, Clayton, Gumbel, Frank, Joe, BBI, and BB7 copula. Parameters are then estimated via maximum likelihood.
We run experiments for data sets generated from a Student-t, a Gumbel, and a Clayton copula as well the copula resulting from two-component Gaussian mixture distribution shown in Figure 2 For each one of these, we sampled a random parameter set and then generated 1000 training samples from the resulting model. We repeated each experiment 25 times. Appendix B contains the details on the considered parameter ranges and the exact sampling procedure.
Figure 2 shows the data and the model with intermediate steps for instance of the Gaussian mixture case. The fitted copula matches the data distribution in unit space very well. FigureB]presens aggregated ISE results for the different setups. The IGC model shows comparable performance scores to TLL1 and TLL22 for the Student-t, Clayton, and Gumbel copulas, and better performance than the BETA approach. For the more complex Gaussian mixture test case, IGC shows superior performance and a much lower variance than all baseline methods. Notably, the parametric approach with BIC-based model selection results in large variance in accuracy for the Clayton and Gumbel data, most likely because the wrong copula family is selected in some cases.
Learning multivariate vine copulas We now turn to the problem of estimating copulas in dimen- sions D 2. To this end, we conduct a similar experiment as before using 5-dimensional vine copulas as data generating distribution. For each of 25 repetitions. we first sample a random tree structure using pyvinecopulib's RVineStructure wwt.rructtueeeemmmaeeeeeeme Simulate( method. Then, we randomly assign parametric pair-copulas with random parameters to each edge in the tree. The considered families of bivariate copulas are Independence, Gaussian, Student-t, Clayton, Gumbel, Frank, Joe, BB1, and BB7. Further details are available in Appendix
We generate 5000 samples from the resulting vine copula

---

model and use these to train our IGC model as well as the benchmark models. As benchmarks, we use vine copula model with TLL2 pair-copulas only and a vine copula model which can select all parametric copula families named above well as the TT.12. Additionally, we estimate parametric Gaussian copula. The parameters are estimated via maximum likelihood and the copula families are selected using the BIC. The selection of the vine structure is based on the Dissmann algorithm (9. We evaluate all models with the ISE at 10000 data points sampled from the true model.
The results of the simulation study are presented in Figure The IGC model has the lowest mean ISE (0.0697) followed by the TLL2- Vine model (0.2093), while the latter has a lower median ISE (0.0339 vs. 0.045). This is the result of some large outliers of the ISE for the vine models that do not occur for the IGC model. These are most likely caused by poorly selected vine structure. Interestingly, the results specifically imply that the data is on average better modeled by the IGC model than by the vine approach although this model class entails the true data generating model. However, recovering the true model does not seem to be an easy task.
4.2 Real data
Exchange rates Copulas are a popular tool in financial risk manage- ment as they can be used estimate the distribution of the multivariate returns over different assets [4 under the assumption of a stationary
2.0
copula. We consider a data set of size N = 5844 that contains 15 years of daily exchange rates between the US-Dollar and the Canadian Dollar, the Euro, the British Pound, the Swiss Franc, and the Japanese Yen. The data was obtained from the R package prm_data. We preprocess the data to filter out the effects of temporal dependencies. To this end, we fit an (((I)-GARCC(L.) [4) process with Student-t innovations to the time series of the daily returns. We then obtain the standardized residuals from the AR-GARCH models and transform these observations to the unit space using the empirical cdfs. Figure shows a kernel density estimate for two selected dimensions of the resulting data set in unit space, namely the US-Dollar/Euro and US-Dollar/Pound exchange rates. While being non-trivial and multi-modal, the distribution is strongly concentrated along the diagonal.
1.5
1.0
0.5
0.0
Gouss
Vin Vine(TLL2)
Figure 4: Box plot of the ISE results for 25 runs of fitting data from 5D vine copula
We then estimate Gaussian copula, vine copula, vine copula with only TLL2 pair-copulas, GMMN as proposed by [16), and our IGC model for this data. The GMMN model has the same loss function, architecture, and hyper parameters as the IGC model but uses sigmoid activation in the final layer. To ensure a test data set of appropriate size, we use a 5-fold cross-validation scheme where 20% of the data is used for training and 80% for testing. For the IGC and the GMMN model additionally use five different random initializations per fold for the neural network weights, i.e., we report means and standard deviations from 25 values for the IGC and the GMMN model and 5 values for the other methods. The results are presented in [ The IGC model clearly outperforms the Gaussian baseline and also the GMMN model. However, the vine copula models show lower average ISES. This might be due to the symmetric nature of the data which can be approximated well by Student-t and TLL poir-copulas.
MAGIC Gamma Telescopes data In order to test our approach on a more com- plex dependency structure, we consider the MAGIC (Major Atmospheric Gamma-ray Imaging Cherenkov) Telescopes data set available from the UCI repository (uttps://archive. ics uci. edu/m1/ Ratasetss/AACIC+GG Gaweeaaaaaaaaaaeaaeeeeeeaee This data was also used for benchmark- ing naramaeiet copula estimation techniques in (32. We only consider the observations classified as gamma and the 5 variables ength, Width, JConc, JM3.ong, fM3Trans. The size of the data set is N
Table ISE mean and standard deviation for exchange rate and magic data from 5 fold CV and random NN weight initializations per fold (lower is better).
exchange rates
magic
Gauss
0.9196 0.1024
0.2415 + 0.0167
Vine
0.2791 + 0.0290
0.0588 + 0.0055
Vine-TLL2
0.2457 + 0.0383
0.0588 + 0.0055
GMMN
0.5791 + 0.2638
0.08065 + 0.0286
IGC (ours)
0.3829 0.1385
0.0345 0.0105
12332. We use the empirical cdfs to transform the
7

---

(8)
(b)
(d)
Figure 5: Exemplary bivariate KDE estimates of the copula densities from the real data sets: (a) Exchange rates GBRPUSD (h) MAGIC TLength-fM3Trrnn, (c) and (d): FashionMNIST Autoencoder latent space dimensions 1-2 and 2-3
observations to the unit space. Figure 5b showcases the dependency structure of the data for two selected data dimensions. The observed structures are highly asymmetric. We again fit the data with a Gaussian copula, a vine copula with all available pair-copulas as above, a vine copula with TLL2 pair-copulas, and a GMMN model with sigmoid output layer [6] benchmarks and the same 5-fold CV strategy as before, i.e., in each fold we use 20% as training data, 80% as test data, and report the average ISE. The results are presented in Table [T] The IGC model achieves the lowest average ISE. Again the GMMN model shows a substantially larger mean ISE with a much larger standard deviation. As only TLL2 pair-copulas were selected by the BIC criterion, the scores for both vine models are the same. The Gaussian copula is clearly not suited for such complex data as the average ISE times larger than for the other approaches.
Copula Autoencoders 40 introduced the Vine Copula Autoencoder, a generative model which uses vine copulas for ex-post density estimation of the latent space of trained autoencoder. In a first step, an autoencoder is trained on data set to learn low dimensional representation of the data. After training, the encoder part of the network is used to map the training data the autoencoder's latent space representation. Next, the uni-variate marginal distributions are estimated, c.g., by using the empirical edfss kernel density estimation, and the compressed data is mapped unit space. After fitting copula model to the observations in unit space, one can sample from the fitted copula model, apply the inverse marginal edfs, and map the simulated data back to the image space using the decoder network of the autoencoder in order to generate new data samples.
In the following, ionMNIST [44] data set. We train convolutional autoencoder on the entire training data of 60000 sam- ples with a latent space of dimension 25, Details on the architecture are found Appendix B After train- ing the autoencoder, we estimate the empirical cdfs of the compressed data using the training data. See Figures and 5d]for exemplary visualizations of the resulting bivariate data densities in unit space. No- tably, these distributions are more complex than the ones from the previous experiments. We fit this data with a Gaussian copula, a vine copula with TLI.2 pair-copulas, a GMMN with sigmoid output layer [16, and an IGC model. We also test independence copula, i.c., assume independence over the latent space. Additionally we report results for a standard variational autoencoder (VAE) [1]] with the same architecture.
present results for the Fash-
Table MMD scores FashionMNIST test set (lower is better)
image
latent
copula
Indep Gauss
0.01912
0.00722
0.00373
0.00619
0.00209
0.00087
Vine
0.00674
0.00131
0.00079
GMMN
0.00392
0.00341
0.00073
IGC
0.00426
0.00114
0.00069
VAE
0.01316
Exemplary samples from all models and the test set are presented in Figure 6 Sampling with the independence copula, i.e., assuming no dependency structure, leads to images with many artifacts. The Gaussian copula produces better images, but still produces some artifacts. Samples from the vine copula or the IGC model are comparable in quality. They show smaller details and few implausible artifacts. The images generated by the VAE are blurry and show much less variation pixel intensity than the test data.

---

(a) independence copula
(b) Gaussian copula
(c) vinc copula
(d) IGC (ours)
(e) VAE
(l) test data
Figure 6: Generated image samples from different copula models on top of a pre-trained autoencoder as well as from standard variational autoencoder. The latent space has 25 dimensions.
In dimensions not feasible anymore compute and compare empirical edfs as required for ISE scoring. We thus resort to the MMD [5 for the numerical evaluation since it is commonly used as simple and robust measure for evaluating image generation models 466. We generate 10000 images from each model and compare those the 10000 test set images by computing the MMD with a Gaussian kernel.
use the bandwidths 1.5, 350, and 8 for the latent unit space, the latent data space, and the image space, respectively. The bandwidths were selected based the median heuristic proposed in [15].
The results are found in Table 2 We provide p-values for these results using the test proposed by [S] in Appendix [ The IGC and GMMN models perform better than all other models for the latent unit space distribution as well as the image space. Interestingly the GMMN model shows relatively large MMD value for the latent data space. This could caused by deviations from marginal uniformity of the learned copula distribution which alter the marginal distributions in the data space.
5 Conclusion
We have proposed the first fully non-parametric model framework for copulas higher dimensions that guarantees uniformity of the marginal distributions. The proposed IGC approach, which is based on an implicit generative step and a differentiable ranking transformation, is structurally simple. Yet, if the generator class is sufficiently complex, any copula dependency structure be modeled. The model can be well implemented with standard deep learning frameworks. For various data sets we have shown modeling performance on par above other state-of-the-art approaches, especially, the vine copula approach.
IGC models should be further investigated in various ways. First, it would be straight-forwart to condition the model on external factors, either by including such values into the input of the generator network by reparameterization of the noise distributions. This would allow to describe context- dependent changes of the dependency structure. Second, many more applications for copulas should be examined, e.g., synthetic tabular data generation [45). Third, the ranking operation could probably be sped up from (M²) to (M log M) using ideas from (3) Finally, the use of adversarial training schemes could also be investigated.

---

Acknowledgments and Disclosure of Funding
This work has been performed the context of the LOEWE center emergenCITY.
References
[1] K. Aas, C. Czado, A. Frigessi, and H. Bakken. Pair-copula constructions of multiple dependence. Insurance: Mathematics and economics, 44(2):182 198, 2009.
[2] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis,
J. Dean, M. Devin, S. Ghemawat, Goodfellow, A. Harp. G. Irving, M. Isard, Y. Jia, Joze- fowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, Shlens, Steiner, Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and
X. Zheng. TensorFlow: scree-scale machine learning on heterogencous systems, 2015. URL https://wwy wew.orffnsooroo...ooooooooooooo. org/ Software available from tensorflow.org.
[3] M. Blondel, O. Teboul, Q. Berthet, and J. Djolonga. Fast differentiable sorting and ranking. In International Conference Machine Learning, pages 950-959. PMLR, 2020.
[4] T. Bollerslev. Generalized autoregressive conditional heteroskedasticity Journal of economet- rics, 31(3):307-327, 1986.
[5] W. Bounliphone, E. Belilovsky, M. Blaschko, I. Antonoglou, and A. Gretton. A test of relative similarity for model selection in generative models. In 4th International Conference Learning Representations, ICLR 2016, 2016.
[6] A. Charpentier, J.-D. Fermanian, and O. Scaillet. The estimation of copulas: Theory and practice. Copulas: From theory to application in finance, pages 35-64, 2007.
[7] Chollet et al. Keras, https:
io, 2015.
[8] S. Demarta and A. J. McNeil. The copula and related copulas. International Statistical Review, 73(1):111-129, 2005.
[9] J. Dissmann, E. C. Brechmann, C. Czado, and D. Kurowicka. Selecting and estimating regular vine copulae and application to financial returns. Computational Statistics & Data Analysis, 59; 52-69, 2013.
[10] G. K. Dziugaite, D. M. Roy, and Z. Ghahramani. Training generative neural networks via maximum mean discrepancy optimization. In Proceedings of the Thirty-First Conference on Uncertainty in Artificial Intelligence, pages 258-267, 2015.
[11] G. Elidan. Copula bayesian networks. Advances in Neural Information Processing Systems, volume 23, 2010.
[12] G. Geenens, A. Charpentier, D. Paindaveine, et al. Probit transformation for nonparametric kernel estimation of the copula density. Bernoulli, 23(3):184--1873 2017.
[13] I. Goodfellow, J. Pouget-Abadis M. Mirza, B. Xu, D. Varde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, volume 27, 2014.
[14] Google Colaboratory. URL
com/
[15] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf. and A. Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
[16] M. Hofert, A. Prasad, and M. Zhu. Quasi-random sampling for multivariate distributions via generative neural networks. Journal of Computational and Graphical Statistics, pages 1-24, feb 2021.
10

---

[17] T. Hong, P. Pinson, S. Fan, H. Zareipour, A. Troccoli, R. J. Hyndman, et al. Probabilistic energy forecasting: Global energy forecasting competition 2014 and beyond. International Journal of Forecasting, 32(3):89-9113, 2016.
[18] J.-N. Hwang, S.-R. Lay, and A. Lippman. Nonparametric multivariate density estimation: comparative study. IEEE Transactions on Signal Processing, 42(10):2795-2810, 1994.
[19] H. Joe. Dependence modeling with copulas. CRC press, 2014.
[20] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. 3rd International Conference Learning Representations, ICLR 2015, 2015.
[21] D. P. Kingma and M. Welling. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, 2014.
[22] R. Koenker and Bassett Jr. Regression quantiles. Econometrica: Journal of the Econometric Society, pages 33-50, 1978.
[23] N. A. Letizia and M. Tonello. Segmented generative networks: Data generation in the uniform probability space. IEEE Transactions on Neural Networks and Learning Systems, 2020.
[24] Y. Li, K. Swersky, and R. Zemel. Generative moment matching networks. In International Conference on Machine Learning, pages 1718-1727. PMLR, 2015.
[25] C. K. Ling, F. Fang, and Z. Kolter. Deep archimedean copulas. In Advances in Neural Information Processing Systems, volume 33, 2020.
[26] Y. Lu and J. Lu. A universal approximation theorem of deep neural networks for expressing probability distributions. In Advances in Neural Information Processing Systems, pages 3094 3 3105, 2020.
[27] D. Meyer, T. Nagler, and R. J. Hogan. Copula-based synthetic data generation for machine learning emulators in weather and climate: application simple radiation model. Geoscientific Model Development Discussions, pages 1-21, 2021.
[28] S. Mohamed and B. Lakshminarayanan. Learning in implicit generative models. arXiv preprint arXiv:1610.03483, 2016.
[29] A. Möller, A. Lenkoski, and T. L.. Thorarinsdottir. Multivariate probabilistic forecasting using ensemble bayesian model averaging and copulas. Quarterly Journal of the Royal Meteorological Society, 139(673):822999 2013.
[30] Nagler. kdecopula: An R package for the kernel estimation of bivariate copula densities. Journal of Statistical Software, 84(7):1-22, 2018.
[31] T. Nagler and T. Vatter. pyvinccopulib. pyvinecopulib/
URL https::/ggituu..comm/teeeop//
[32] T. Nagler, C. Schellhase, and C. Czado. Nonparametric estimation of simplified vine copula models: comparison of methods. Dependence Modeling, 5(1):99-120, 2017.
[33] N. Patki, R. Wedge, and K. Veeramachaneni. The synthetic data vault. In 2016 IEEE Interna- tional Conference on Data Science and Advanced Analytics (DSAA), pages 399-410. IEFE, 2016.
[34] A. J. Patton. review of copula models for economic time series. Journal of Multivariate Analysis, 110:4-18, 2012.
[35] T. Qin, T.-Y. Liu, and H. Li. A general approximation framework for direct optimization of information retrieval measures. Information retrieval, 13(4):375-397, 2010.
[36] A. E. Raftery, T. Gneiting, Balabdaoui, and M. Polakowski. Using bayesian model averaging to calibrate forecast ensembles. Monthly weather review, 1555-1144 2005.
11

---

[37] D. Sejdinovic, B. Sriperumbudur, A. Gretton, and K. Fukumizu. Equivalence of distance-basee and rkhs-based statistics in hypothesis testing. The Annals of Statistics, pages 2263-2291, 2013.
[38] M. Sklar. Fonctions de repartition an dimensions leurs marges. Publ. inst. statist. univ. Paris, 8:229-231, 1959.
[39] G. J. Székely M. L. Rizzo, et al. Testing for equal distributions in high dimension. InterStat, (16.10):1249-127) 2004.
[40] N. Tagasovska, D. Ackerer, and T. Vatter. Copulas as high-dimensional generative models: Vine copula autoencoders. Advances in Neural Information Processing Systems, volume 32, 2019.
[41] Tastu, Pinson, and H. Madsen. Space-time trajectories of wind power generation: Parametrized precision matrices under gaussian copula approach. In Modeling and stochastic learning for forecasting in high dimensions, pages 267-296. Springer, 2015.
[42] A. W. Van der Vaart. Asymptotic statistics, volume Cambridge university press, 2000.
[43] H. Wang, L. Yu, Z. Cao, and S. Ermon. Multi-agent imitation learning with copulas. In Joint European Conference Machine Learning and Knowledge Discovery in Databases, pages 139-156, Springer, 2021.
[44] H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist; novel image dataset for benchmarking machine learning algorithms. arXiv preprint 1rXiv:1708.07747, 2017.
[45] L. Xu, M. Skoularidou, A. Cuesta-Infante, and K. Veeramachaneni. Modeling Tabular data using Conditional GAN. In Advances in Neural Information Processing Systems, 2019.
[46] Q Xu, G. Huang, Y. Yuan, C. Guo, Y. Sun, F. Wu, and K. Weinberger. An empirical study on evaluation metries of generative adversarial networks. arXiv preprint 1rXiv:1806..77 2018.
12


Reviews:
[{"review_id": "ct3bxBv7nvT", "reviewer": "Reviewer_VacV", "summary": "The paper proposes an implicit method (IGC) of estimating a copula. The suggested approach is elegant and from the presented results the method seems to work on par with existing non-parametric vine copula approaches. Previous works have made an attempt to use GANs for implicit copula estimation with difficulties, however, in IGC the authors suggest techniques to address the requirements for uniform margins with a simple two-step procedure leveraging the probability integral transform.\nThe general idea and contributions are clearly presented and I find this work valuable due to its simplicity and I believe its efficiency can be further exploited in other copula-related/inspired applications.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Since IGC is an implicit generative model, I would suggest comparing it to at least *one* such baseline (either [16] or [23] ). VAEs are considered Explicit generative models [1].\n\n2. The latent dimension used in the experiments is quite low (up to 25 which suffices for fashionMNIST), did you try IGC for more complex datasets that require larger latent spaces? Was there any limitation?\n\n3. The experiments could be extended to present this new method better. Toy examples such as Swiss Role or the multi-modal ring of Gaussians which are the difficult cases for implicit generative models. These are experiments easy to run and can be informative on the behavior of IGC (for example lack of mode collapse etc.)\n\n4. The main advantage of copulas is that we can model the marginals and the dependence separately. Is this still possible with your approach? \n\n5. Could you get some variance for the results in the tables by generating with different initializations?\n\n6. How can IGC be extended to a conditional generative model?\n\n[1] Goodfellow, Ian. \"Nips 2016 tutorial: Generative adversarial networks.\" arXiv preprint arXiv:1701.00160 (2016).\n\n\n\n------------------------------------------------------------\nUpdate after Author Feedback and Discussion\n------------------------------------------------------------\n\nI thank the authors for the clarification. My questions and concerns have been mostly addressed/answered. \nThe original score still reflects my general opinion about the paper so I am keeping it, however, I do agree with the other reviews that it is important to show an advantage of the proposed method compared to previous copula-NN hybrids whether it is on new datasets or new applications/experiments. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors could mention the potential negative impact of generating fake data.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "S9fwIyAGIpy", "reviewer": "Reviewer_qxq2", "summary": "The authors propose to model general copula models by training a neural network that samples from a latent space and maps samples to the high dimensional unit cube. Fitting the model is done by transforming the generated samples to have uniform margins by empirically computing the ranks. Then, the model is optimized by minimizing the MMD distance w.r.t true uniform data samples. ", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the paper is very well written and main results are clearly presented. However, my main concern is related to the applicability of  the proposed copula model. It seems that the proposed model can only be exploited as a generative model. For example, in order to compute CDFs which is the primary reason for learning copulas, the proposed model needs to generate samples, then empirically estimate the CDF which even with a large sample size could be not accurate in high dimensional settings.   Also,  computing the ranks seems to add errors to the training procedure which may affect  learning the model. On another front, the simulation results do not show substantial improvements over previous models (sometimes worse) especially Vine copula models. For the FashionMNIST dataset, it is probably more convenient to compare w.r.t the FID score as it is more used in image generation tasks.  \n\nMinor comments: \n1. In line 36, that’s not accurate, there are also nested Archimedean copulas which support non-symmetry. \n2. There is a repetition in line 76.\n3. Typo in line 299. \n4. Add standard deviation to the MMD scores. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, he authors adequately addressed the limitations and potential negative societal impact of their work", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 "}, "extra_text": {}}, {"review_id": "C6yAkxqsXg", "reviewer": "Reviewer_8HVZ", "summary": "The paper proposes a novel method to model \"high\"-dimensional copulas using deep neural networks.\nThey are able to train their implicit generative models in a likelihood-free way and obtain performances that are on-par with SOTA approaches.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Given the noted lack of availability of performant (>2)-dimensional copula models, I welcome this contribution.\nI think that it is a worthwhile pursuit and that the authors' approach is novel and interesting.\nAdditionally, the paper is well written and reads nicely.\n\nOne thing that bothers me is that there is no theoretical result at all.\nNo universal approximation/sample complexity/generalization bound on the ML side, no identifiability/consistency/asymptotica distribution/estimation rate on the stats side.\n\nAnother concern that I have is that I see no way of obtaining computing copula densities or probabilities in closed form, but this is true for any implicit generative model.\nSimilarly, I don't see a way to either marginalize or condition over subsets of the variables, but maybe the authors can expand on this?\n\nBelow is my section per section review.\n\n## Introduction\n\nThe introduction is good, and the literature review cites all the important papers on the topic.\nHowever, the \"contributions\" part need serious work.\n\n### Uniformity property\n\nWhy is ensuring the uniformity property in finite samples important?\nAs a copula expert/statistician foremost and ML researcher secondly, I think that for a copula model to be sensible, being approximately uniform in finite sample is enough as long as it is asymptotically so.\nThis is the case for most nonparametric copula estimators such as the bivariate kernel density estimation used in [31].\nSee e.g., [12], which powers the TLL method and can easily be generalized to higher-dimensions.\n\nNote that, if D is a function that satisfies all the requirements of a copula except the uniform margins, i.e. if $D:[0,1]^n\\\\to[0,1]$ is s.t. (1) $D(u_1, \\\\dots, u_{i-1}, 0, u\\_{i+1}, \\\\dots, u_n) = 0$, (2) $D(1, \\\\dots, 1) = 1$, and (3) D is n-increasing, then $F(x_1, \\\\dots, x_n) = D(F_1(x_1), \\\\dots, F_n(x_n))$ is a valid distribution with margins $F_1, \\\\dots, F_n even$ if $D(1, \\\\dots, 1, u_i, 1, \\\\dots, 1) \\\\neq u_i$.\n\nTherefore, since ensuring marginal uniformity is the \"key challenge\", its importance should be motivated.\nIt seems that ML people designing new copula models, whether in a generative fashion or via deep nets, focus a lot on this without really thinking about how relevant it is (if at all).\nSo why is this important and/or relevant?\nAnd if it is not, then what is this contribution really bringing as an improvement over [16].\n\nAs a side-note: since you are using the softrank layer to approximate the ECDF of the latent variables during training, the model that you really learn is only an approximation of the copula.\nAnd estimating the latent marginals after training is an example of a posteriori normalization that could be applied to other methods (e.g., [16]) as well.\n\n### No tree structures\n\nWhy is the fact that the model is free from tree structures relevant as you mention in the main contributions?\nIn the nonparametric context, trees represent a structural constraint allowing to obtain better estimation rates (e.g., bivariate, see \"Evading the curse of dimensionality in nonparametric density estimation with simplified vine copulas\").\nIn your case, since there is no structural constraint, your model will suffer from the curse of dimensionality in the sense that you will need a sample size that grows exponentially with the dimension to obtain the same error rates.\n\n### Misleading claim of a simpler framework\n\nIn my opinion, the claim of \"similar or improved performance for different tasks, with a structurally simpler framework\" is at best misleading.\nI am not saying that vine structures are \"simple\", but how is a deep neural network \"simpler\" than a nested set of trees.\nAlso, there is well established software for vines that is ready to use, and the following seems easy enough to me:\n\n```\nimport pyvinecopulib as pv\nfit = pv.Vinecop(data)\n```\n\nAnd training times for vines are order of magnitudes smaller than the ones for your method.\nThus, I would argue to either clarify what is \"simpler\" about your approach or tamp down the claim.\n\n### Likelihood-free training\n\nPlease explain and motivate why this is a good thing.\nWhat you do is essentially minimize a distance between your copula and the empirical copula.\nBut it is well known that MLEs are efficient (i.e., attaining the Cramer-Rao Lower Bound), so likelihood-based training is generally viewed as desirable if at all possible.\nTo me, it seems that the reason you are training likelihood-free is simply that you cannot obtain the density.\n\n## Proposed IGC Model\n\n### Copula basics\n\n- Why write $F_{X,d}$ instead of $F_{X_d}$?\n- Please mention whether you are concerned with discrete or continuous data (Sklar's theorem also giving unicity of the copula for continuous distributions).\n\n### ICG model\n\n- Please describe the architecture that you use in more details. This could also be done in the next section.\n\n## Training\n\n### Parameter estimation\n\n- Have you tried optimal transport-based distances (e.g., Wassertein)? It would seem like the natural thing to do in your context.\n\n## Experiments\n\n### Synthetic data\n\n- 1000 observations for a bivariate dataset is a fairly large sample already. It would be interesting to see how the methods improve from low to high samples (e.g., 100, 200, 500, 1000, 2000, 5000),\n- In the Gaussian mixture example, the only reason ICG does better than TLL is because the automatic bandwidth selection rule of the later is geared towards unimodal distributions. Also, the TLL method is generally more stable than TLL1/TLL2 for complex (e.g., multimodal) distributions.\n- Regarding your discussion about recovering the true model and ICG doing better than vines when the true model is a vine: the reason is not related to the likelihood/misspecification of the pair-copulas. It is due to the fact that the selected tree structure is incorrect (you might know that the space of vine structures is super exponential, e.g. much larger than the space of DAG structures). Basically, the Dissmann algorithm is a greedy algorithms whose heuristic aims at capturing most of the dependence in the higher trees. But your simulation setup creates models by sampling parameters irrespectively of the tree level. Thus, the structure selection algorithm is simply not designed for the model class that you sample from. I'm not saying it's bad, just that it is unsuprising. This also likely explains why the Vine(TLL2) is OK most of the time (i.e., low median ISE), but can go really wrong is a poor structure is selected (i.e., doesn't matter how well you estimate pairs if the structure is bad).\n\n### Real data\n\n- Table 1 is missing the standard errors. In particular, for MAGIC, it seems like Vine, Vine(TLL2) and IGC differences are unlikely to matter. Given that you do cross-validation, adding the standard errors should be easy enough.\n\n- Table 2, same comment, but here you say \"significantly\"...", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have not mentioned anything.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "BWuMfcsGyT0", "reviewer": "Reviewer_xntW", "summary": "The paper proposes a novel method for learning flexible cupola distributions of high dimensional data using implicit generative models, the first non-parametric method for estimating cupola distributions with a guarantee of uniform marginal distributions.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is generally well written and I found the description of the model and training procedures easy to follow. The authors provide a thorough review of related literature and put their work into context.\n\nI am not closely familiar with the literature on estimating cupolas but based on the authors discussion the method seems novel and mathematically sound. The main contribution of the work are ensuring that the cupola distributions learned by the implicit generative model has uniform marginals by using a differentiable approximation of the cumulative density function (a softrank layer) to transform the output of the generator network into $[0,1]^D$.\n\nThe method is evaluated on a number of synthetic and real-world datasets and shown to be matching or in some cases out-performing SOTA methods such as vine cupolas.\nThe weaknesses of the evaluation is that it relies on estimating CDF functions in up to 5-dimensional spaces which can be inaccurate. For fashion MNIST the authors report MMD distances as point estimates with relatively small differences between methods. It would be useful to get estimates of variability either by resampling or performing a 3-sample test as described by Bounliphone et al 2015 (A Test of Relative Similarity For Model Selection in Generative Models.)\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss some limitations related to computational scaling.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5h"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers had divergent opinions on this paper — behind the scenes, in the discussion phase, one was arguing strongly for acceptance, and one for rejection. The primary complaints were: (i) the model may have limited applicability, due to the fact that it is purely "implicit" and does not provide a density estimate, precluding many of the common uses of copula models; (ii) the empirical evaluation could have been stronger, as there is only moderate improvement relative to vine copula baselines, and for the image data such as fashion MNIST other metrics could be more appropriate. However, all reviewers agreed that the authors responded well to concerns during the discussion period. I would strongly suggest including the suggested additional results on generation for standard datasets in the appendix, as mentioned, as well as including more on motivating examples in which an implicit copula model is helpful despite lack of an explicit density.
Decision: Accept (Poster)