Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a novel method for learning flexible cupola distributions of high dimensional data using implicit generative models, the first non-parametric method for estimating cupola distributions with a guarantee of uniform marginal distributions.

main_review: The paper is generally well written and I found the description of the model and training procedures easy to follow. The authors provide a thorough review of related literature and put their work into context.

I am not closely familiar with the literature on estimating cupolas but based on the authors discussion the method seems novel and mathematically sound. The main contribution of the work are ensuring that the cupola distributions learned by the implicit generative model has uniform marginals by using a differentiable approximation of the cumulative density function (a softrank layer) to transform the output of the generator network into $[0,1]^D$.

The method is evaluated on a number of synthetic and real-world datasets and shown to be matching or in some cases out-performing SOTA methods such as vine cupolas.
The weaknesses of the evaluation is that it relies on estimating CDF functions in up to 5-dimensional spaces which can be inaccurate. For fashion MNIST the authors report MMD distances as point estimates with relatively small differences between methods. It would be useful to get estimates of variability either by resampling or performing a 3-sample test as described by Bounliphone et al 2015 (A Test of Relative Similarity For Model Selection in Generative Models.)

limitations_and_societal_impact: The authors discuss some limitations related to computational scaling.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a novel method to model "high"-dimensional copulas using deep neural networks.
They are able to train their implicit generative models in a likelihood-free way and obtain performances that are on-par with SOTA approaches.
main_review: Given the noted lack of availability of performant (>2)-dimensional copula models, I welcome this contribution.
I think that it is a worthwhile pursuit and that the authors' approach is novel and interesting.
Additionally, the paper is well written and reads nicely.

One thing that bothers me is that there is no theoretical result at all.
No universal approximation/sample complexity/generalization bound on the ML side, no identifiability/consistency/asymptotica distribution/estimation rate on the stats side.

Another concern that I have is that I see no way of obtaining computing copula densities or probabilities in closed form, but this is true for any implicit generative model.
Similarly, I don't see a way to either marginalize or condition over subsets of the variables, but maybe the authors can expand on this?

Below is my section per section review.

## Introduction

The introduction is good, and the literature review cites all the important papers on the topic.
However, the "contributions" part need serious work.

### Uniformity property

Why is ensuring the uniformity property in finite samples important?
As a copula expert/statistician foremost and ML researcher secondly, I think that for a copula model to be sensible, being approximately uniform in finite sample is enough as long as it is asymptotically so.
This is the case for most nonparametric copula estimators such as the bivariate kernel density estimation used in [31].
See e.g., [12], which powers the TLL method and can easily be generalized to higher-dimensions.

Note that, if D is a function that satisfies all the requirements of a copula except the uniform margins, i.e. if $D:[0,1]^n\\to[0,1]$ is s.t. (1) $D(u_1, \\dots, u_{i-1}, 0, u\_{i+1}, \\dots, u_n) = 0$, (2) $D(1, \\dots, 1) = 1$, and (3) D is n-increasing, then $F(x_1, \\dots, x_n) = D(F_1(x_1), \\dots, F_n(x_n))$ is a valid distribution with margins $F_1, \\dots, F_n even$ if $D(1, \\dots, 1, u_i, 1, \\dots, 1) \\neq u_i$.

Therefore, since ensuring marginal uniformity is the "key challenge", its importance should be motivated.
It seems that ML people designing new copula models, whether in a generative fashion or via deep nets, focus a lot on this without really thinking about how relevant it is (if at all).
So why is this important and/or relevant?
And if it is not, then what is this contribution really bringing as an improvement over [16].

As a side-note: since you are using the softrank layer to approximate the ECDF of the latent variables during training, the model that you really learn is only an approximation of the copula.
And estimating the latent marginals after training is an example of a posteriori normalization that could be applied to other methods (e.g., [16]) as well.

### No tree structures

Why is the fact that the model is free from tree structures relevant as you mention in the main contributions?
In the nonparametric context, trees represent a structural constraint allowing to obtain better estimation rates (e.g., bivariate, see "Evading the curse of dimensionality in nonparametric density estimation with simplified vine copulas").
In your case, since there is no structural constraint, your model will suffer from the curse of dimensionality in the sense that you will need a sample size that grows exponentially with the dimension to obtain the same error rates.

### Misleading claim of a simpler framework

In my opinion, the claim of "similar or improved performance for different tasks, with a structurally simpler framework" is at best misleading.
I am not saying that vine structures are "simple", but how is a deep neural network "simpler" than a nested set of trees.
Also, there is well established software for vines that is ready to use, and the following seems easy enough to me:

```
import pyvinecopulib as pv
fit = pv.Vinecop(data)
```

And training times for vines are order of magnitudes smaller than the ones for your method.
Thus, I would argue to either clarify what is "simpler" about your approach or tamp down the claim.

### Likelihood-free training

Please explain and motivate why this is a good thing.
What you do is essentially minimize a distance between your copula and the empirical copula.
But it is well known that MLEs are efficient (i.e., attaining the Cramer-Rao Lower Bound), so likelihood-based training is generally viewed as desirable if at all possible.
To me, it seems that the reason you are training likelihood-free is simply that you cannot obtain the density.

## Proposed IGC Model

### Copula basics

- Why write $F_{X,d}$ instead of $F_{X_d}$?
- Please mention whether you are concerned with discrete or continuous data (Sklar's theorem also giving unicity of the copula for continuous distributions).

### ICG model

- Please describe the architecture that you use in more details. This could also be done in the next section.

## Training

### Parameter estimation

- Have you tried optimal transport-based distances (e.g., Wassertein)? It would seem like the natural thing to do in your context.

## Experiments

### Synthetic data

- 1000 observations for a bivariate dataset is a fairly large sample already. It would be interesting to see how the methods improve from low to high samples (e.g., 100, 200, 500, 1000, 2000, 5000),
- In the Gaussian mixture example, the only reason ICG does better than TLL is because the automatic bandwidth selection rule of the later is geared towards unimodal distributions. Also, the TLL method is generally more stable than TLL1/TLL2 for complex (e.g., multimodal) distributions.
- Regarding your discussion about recovering the true model and ICG doing better than vines when the true model is a vine: the reason is not related to the likelihood/misspecification of the pair-copulas. It is due to the fact that the selected tree structure is incorrect (you might know that the space of vine structures is super exponential, e.g. much larger than the space of DAG structures). Basically, the Dissmann algorithm is a greedy algorithms whose heuristic aims at capturing most of the dependence in the higher trees. But your simulation setup creates models by sampling parameters irrespectively of the tree level. Thus, the structure selection algorithm is simply not designed for the model class that you sample from. I'm not saying it's bad, just that it is unsuprising. This also likely explains why the Vine(TLL2) is OK most of the time (i.e., low median ISE), but can go really wrong is a poor structure is selected (i.e., doesn't matter how well you estimate pairs if the structure is bad).

### Real data

- Table 1 is missing the standard errors. In particular, for MAGIC, it seems like Vine, Vine(TLL2) and IGC differences are unlikely to matter. Given that you do cross-validation, adding the standard errors should be easy enough.

- Table 2, same comment, but here you say "significantly"...
limitations_and_societal_impact: The authors have not mentioned anything.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes an implicit method (IGC) of estimating a copula. The suggested approach is elegant and from the presented results the method seems to work on par with existing non-parametric vine copula approaches. Previous works have made an attempt to use GANs for implicit copula estimation with difficulties, however, in IGC the authors suggest techniques to address the requirements for uniform margins with a simple two-step procedure leveraging the probability integral transform.
The general idea and contributions are clearly presented and I find this work valuable due to its simplicity and I believe its efficiency can be further exploited in other copula-related/inspired applications.
main_review: 1. Since IGC is an implicit generative model, I would suggest comparing it to at least *one* such baseline (either [16] or [23] ). VAEs are considered Explicit generative models [1].

2. The latent dimension used in the experiments is quite low (up to 25 which suffices for fashionMNIST), did you try IGC for more complex datasets that require larger latent spaces? Was there any limitation?

3. The experiments could be extended to present this new method better. Toy examples such as Swiss Role or the multi-modal ring of Gaussians which are the difficult cases for implicit generative models. These are experiments easy to run and can be informative on the behavior of IGC (for example lack of mode collapse etc.)

4. The main advantage of copulas is that we can model the marginals and the dependence separately. Is this still possible with your approach? 

5. Could you get some variance for the results in the tables by generating with different initializations?

6. How can IGC be extended to a conditional generative model?

[1] Goodfellow, Ian. "Nips 2016 tutorial: Generative adversarial networks." arXiv preprint arXiv:1701.00160 (2016).



------------------------------------------------------------
Update after Author Feedback and Discussion
------------------------------------------------------------

I thank the authors for the clarification. My questions and concerns have been mostly addressed/answered. 
The original score still reflects my general opinion about the paper so I am keeping it, however, I do agree with the other reviews that it is important to show an advantage of the proposed method compared to previous copula-NN hybrids whether it is on new datasets or new applications/experiments. 
limitations_and_societal_impact: The authors could mention the potential negative impact of generating fake data.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The authors propose to model general copula models by training a neural network that samples from a latent space and maps samples to the high dimensional unit cube. Fitting the model is done by transforming the generated samples to have uniform margins by empirically computing the ranks. Then, the model is optimized by minimizing the MMD distance w.r.t true uniform data samples. 
main_review: Overall the paper is very well written and main results are clearly presented. However, my main concern is related to the applicability of  the proposed copula model. It seems that the proposed model can only be exploited as a generative model. For example, in order to compute CDFs which is the primary reason for learning copulas, the proposed model needs to generate samples, then empirically estimate the CDF which even with a large sample size could be not accurate in high dimensional settings.   Also,  computing the ranks seems to add errors to the training procedure which may affect  learning the model. On another front, the simulation results do not show substantial improvements over previous models (sometimes worse) especially Vine copula models. For the FashionMNIST dataset, it is probably more convenient to compare w.r.t the FID score as it is more used in image generation tasks.  

Minor comments: 
1. In line 36, that’s not accurate, there are also nested Archimedean copulas which support non-symmetry. 
2. There is a repetition in line 76.
3. Typo in line 299. 
4. Add standard deviation to the MMD scores. 
limitations_and_societal_impact: Yes, he authors adequately addressed the limitations and potential negative societal impact of their work
review_title: null

Meta-review: Reviewers had divergent opinions on this paper — behind the scenes, in the discussion phase, one was arguing strongly for acceptance, and one for rejection. The primary complaints were: (i) the model may have limited applicability, due to the fact that it is purely "implicit" and does not provide a density estimate, precluding many of the common uses of copula models; (ii) the empirical evaluation could have been stronger, as there is only moderate improvement relative to vine copula baselines, and for the image data such as fashion MNIST other metrics could be more appropriate. However, all reviewers agreed that the authors responded well to concerns during the discussion period. I would strongly suggest including the suggested additional results on generation for standard datasets in the appendix, as mentioned, as well as including more on motivating examples in which an implicit copula model is helpful despite lack of an explicit density.
Decision: Accept (Poster)