Inference and Sampling for Archimax Copulas

Abstract: Understanding multivariate dependencies in both the bulk and the tails of a distribution is an important problem for many applications, such as ensuring algorithms are robust to observations that are infrequent but have devastating effects. Archimax copulas are a family of distributions endowed with a precise representation that allows simultaneous modeling of the bulk and the tails of a distribution. Rather than separating the two as is typically done in practice, incorporating additional information from the bulk may improve inference of the tails, where observations are limited. Building on the stochastic representation of Archimax copulas, we develop a non-parametric inference method and sampling algorithm. Our proposed methods, to the best of our knowledge, are the first that allow for highly flexible and scalable inference and sampling algorithms, enabling the increased use of Archimax copulas in practical settings. We experimentally compare to state-of-the-art density modeling techniques, and the results suggest that the proposed method effectively extrapolates to tails while scaling to higher dimensional data. Our findings suggest that the proposed algorithms can be used in a variety of applications where understanding the interplay between the bulk and the tails of a distribution is necessary, such as health and safety.

Reviews and author responses:

 Thank you for your careful consideration and response to my comments. I have accordingly raised my score and I recommend acceptance of this paper. 

 Thank you very much for your careful responses. I am happy with your responses to my comments. In particular, my concerns given in comments (e) and (f) have been sufficiently addressed, and therefore I increased my score to 7 (Accept).

 Thank you for your response and addressing the comments! I will leave the score the same; I do recommend acceptance of the paper.

 We thank you for the very positive review, you brought up many thoughtful comments and interesting questions. The following is our response:

**[Clarity]** Figure 1 is not referenced in the text. 

Thank you for pointing out the error. We added a reference to Figure 1 in the revision, line 101 on asymptotic dependence and line 109 on radial envelope. We also enriched Figure 1 with an example of asymmetric dependence.

**[Initialization]** Archimedean generator first set to $\varphi(x)=\exp\\{-x\\}$.

This is a very important question. We did not mention in the paper but we also considered initialization with different one-parameter families of Archimedean generators, with choice of generator based on the highest log-likelihood of transformed observation $\xi$, equations (7) and (9). The parameter for each family may be obtained from an average of inversion of pairwise Kendall tau, as per the following equation [1], for each pair:

$\tau_{\varphi,\ell} = \tau_\ell + (1-\tau_\ell)\tau_\varphi,$

where $\tau_{\varphi,\ell}$ is the Kendall's tau of the Archimax bivariate marginal, $\tau_\ell$ is the Kendall's tau of the extreme value component and $\tau_\varphi$ is the Kendall's tau of the Archimedean component. An average of inversion of pairwise Kendall tau was employed in [2], with emphasis on the Clayton generator.

Initialization with specific families of Archimedean generators might bias initialization, and thus we suggested initializing via the Archimedean generator first set to $\varphi(x)=\exp\\{-x\\}$ representing extreme value copulas and pre-processing the initial data to have extreme value dependence via block-maximas. This was also motivated by the experiment on extrapolating to extremes. 

We added this alternative initialization procedure to the revision, in Appendix A.4.1 and briefly mention it as a footnote in Section 3.3.1 which describes the initialization procedure.

**[Hierarchical Archimedean copula]**

This is a good direction for future work. The most straightforward way is to add hierarchy in the construction of the random variables $R$ and $\mathbf{W}$ as in [3]. We have added a note on this in the revision. 

**[Architecture of generative neural networks]**

This is also another excellent direction for future work, such as to specify hierarchy and time dependence. We have also added this suggestion to the future work section of the revision.

---

References

[1] P. Capéraà, A.-L. Fougères, and C. Genest. Bivariate distributions with given extreme value attractor. Journal of Multivariate Analysis, 72(1):30–49, 2000.

[2] S. Chatelain, A.-L. Fougères, and J. G. Nešlehová. Inference for Archimax copulas. The Annals of Statistics, 48(2):1025 – 1051, 2020.

[3] M. Hofert, R. Huser, and A. Prasad. Hierarchical archimax copulas. Journal of Multivariate Analysis, 167:195–211, 2018.

 We thank you for the overall positive and very detailed review. Your critique and suggestions are highly appreciated and the following is our response:

**[Quality]** more experimental results to support claims of significance and contribution.

We thank you for the suggestion, and we have added a number of additional experiments in the revision, including an application of out-of-distribution detection and another baseline of the t and skew-t copulas. We hope that these help bolster the claims of the paper by adding additional empirical evidence. 

**[Clarity]** motivating example to clarify why Archimax copulas are appropriate/ necessary in critical scenarios. 

This is a wonderful suggestion. One of the motivation for using Archimax copulas in critical scenarios is to model extreme data (e.g. very strong and rare earthquakes), where observations are rare, from a mix of moderately less extreme data (e.g. strong earthquakes), where observations are relatively more abundant. Another motivation for use of Archimax copulas is the ability to model both the bulk and the tails with one copula. We added more explanation in the revision under the experiment on extrapolating to extreme rainfall in Section 4.

**[Significance]** superiority on tasks where SOTA ML models fail

Through the original and additional experiments, we provided promising results on the task of extrapolating to extremes. Following your useful suggestion, we also demonstrate good performance on the task of out-of-distribution detection. 

**1.** complexity not mentioned, trade-off between flexibility and scalability.

There is a direct trade-off if the sizes of the supports for $R$ and $\mathbf{W}$ are fixed. This was briefly mentioned as a disadvantage in solving for $R$ and $\mathbf{W}$ as discrete random variables with fixed sizes of supports in Appendix A.3. In our case, we let them be outputs of generative neural networks. In addition, to speed up training, we use a smaller number of samples in the empirical expectations of $\varphi_\theta$ and $\ell_\theta$ and resample fewer times during training, as mentioned in Appendix A.2.1 and A.3. The computational complexity is given in terms of time taken in Appendix B.1 and B.2. All timings are with a 2.7 GHz Intel Core i7, 16GB 2133MHz LPDDR3.

**2.** ablation studies of data efficiency. 

We agree with your helpful suggestion, an ablation study with number of observations $n=200$ and $n=1000$ was given in Appendix B.1.

**3.** non-tabular data, other NN architecture. 

We thank you for the suggestion. While the primary focus of the paper is on describing the inference and sampling algorithms, extensions to non-tabular data should be straightforward. For example, the Archimax copula can be used for describing the dependencies of a graph neural network. The Gaussian copula in [1] may be replaced with the Archimax copula described in this paper. We have added these notes to the future work section of the revision.

**4.** comparison of sampled marginals to original. 

We compared the CvM distance between samples from the learned copula and the empirical copula. We also gave visual comparisons. Confirming a reasonable fit is necessary for mitigating risks due to model misspecification. The CvM and plots show the dependence being captured correctly. Since the primary goal of the copula is to model the dependency, we focused on showcasing that aspect. We note that the marginals can be transformed fairly easily, such as via deep sigmoidal flows [2] or neural spline flows [3].

**5.** experiment to increase significance of paper, e.g. out-of-distribution detection 

We thank you for this excellent suggestion. We added an experiment on out-of-distribution detection to the revision, after the paragraph on extrapolating to extreme rainfall in Section 4, with details in Appendix B.5. 

The AUC scores for outlier detection based on likelihoods computed from the Archimax copula, normalizing flow and variational autoencoder are **0.92** (higher is better), 0.82, 0.37. The F1 scores are **0.72** (higher is better), 0.48, 0.04. Please see the details and visual comparison in Appendix B.5 of the revision.

**6.** missing related work.

Thank you for pointing out this literature which we overlooked, we cited the missing related work [4] in the revision, under Section 1.1 related work.

**Reference Figure 1** in the text. 

Thank you for pointing out the error. We added a reference to Figure 1 in the revision, line 101 on asymptotic dependence and line 109 on radial envelope. We also enriched Figure 1 with an example of asymmetric dependence.

---

References

[1] Ma et al. Copula-GNN: Towards integrating representational and correlational roles of graphs in graph neural networks. In ICLR 2021.

[2] Huang et al. Neural autoregressive flows. In ICML 2018. 

[3] Durkan et al. Neural spline flows. In Neurips 2019.

[4] Bhatia et al. ExGAN: Adversarial generation of extreme samples. In AAAI 2021.

 We thank you for the very positive and constructive review, your suggestions are highly appreciated and are incorporated into a revised version:

**(i) Intuitive explanation** to why Archimax copulas are good for modeling dependencies between extreme events.

Archimax copulas were initially developed as a tool to study the behaviour of methods used to estimate the joint distribution of extreme events [1]. 

The extreme value copula that arises in the limit can be understood from the stable tail dependence function (stdf) $\ell$ and the index of regular variation of the Archimedean generator $\varphi$. Unlike extreme value copulas which emerge from the limiting distribution of extreme events, the motivation for use of Archimax copulas is to model extreme data (e.g. very strong and rare earthquakes), where observations are rare, from a mix of moderately less extreme data (e.g. strong earthquakes), where observations are relatively more abundant. For example the monthly rainfall of French Brittany did not pass the test of extreme value dependence [2, 3]. Additionally, a pre-processing step for use with extreme value copulas by computing block-maximas would reduce the amount of available observations by 6-fold. Finally, by modeling the data with an Archimax copula using the techniques in this paper, we can generate samples for further studies and simulation. We provided this explanation in Appendix A.5 and added a brief summary to the revision in Section 4, motivating the experiment on extrapolating to extreme rainfall.

**(ii) Concise statement** of the complete model up front, explaining how the deep generative model determines the stdf. 

We thank you for the very helpful suggestion and will state a concise statement of the complete model up front. The deep generative models are used to represent the distributions of the spectral random variable $\mathbf{W}$ and radial random variable $R$.
By taking an expectation, these characterize the stdf $\ell$ and the Archimedean generator $\varphi$. We added this statement at the end of Section 1.1.

---

References

[1] P. Capéraà, A.-L. Fougères, and C. Genest. Bivariate distributions with given extreme value attractor. Journal of Multivariate Analysis, 72(1):30–49, 2000.

[2] I. Kojadinovic, J. Segers, and J. Yan. Large-sample tests of extreme-value dependence for multivariate copulas. LIDAM Reprints ISBA 2011025, Université catholique de Louvain, Institute of Statistics, Biostatistics and Actuarial Sciences (ISBA), 2011. 

[3] S. Chatelain, A.-L. Fougères, and J. G. Nešlehová. Inference for Archimax copulas. The Annals of Statistics, 48(2):1025 – 1051, 2020.

 We thank you for your positive feedback, thoughtful comments and useful suggestions. We address the individual comments and suggestions below: 

**(e) [Quality]** insufficient comparison to existing flexible copulas such as skew-t copulas.

We agree that the skew-t copula is a good additional baseline for asymmetrical copulas [1, 2, 3, 4].

For the 3-dimensional dataset that models the rainfall in French Britanny, the Cramér–von Mises (CvM) distance between the Clayton - negative scaled extremal Dirichlet (C-NSD) copula and the learned copulas for Archimax, Gaussian, extreme value, t and skew-t are: **0.0003** (lower is better), 0.0005, 0.0006, 0.0008, 0.0027. 

For this scenario, the Archimax and Gaussian copulas performed better than the extreme value, t and skew-t copulas. This may be because the C-NSD copula does not exhibit extreme value dependence, i.e. it fails the test of extreme value dependence [5]. The skew-t copula might have performed worse than the t copula due to over-parameterization. In addition, maximum likelihood estimation for the skew-t copula was extremely time consuming even for three dimensions and thus intractable for higher dimensions. 

We have included the comparison to skew-t copulas in the revision, under Appendix B.4.1 which describes the rainfall experiment. 

**(f) [Significance]** popularity of Archimax copulas unsure.

We believe that the popularity of Archimax copulas has been hindered due to the lack of existing inference methods to fit parameters to observations and the lack of sampling tools [6, 7].
Recent existing work only presented methods for recovering the stable tail dependence function (stdf) given the Archimedean generator [6] or maximum likelihood estimation for few parameters in 2-3 dimensions [8, 9]. On the other hand, these works showed a better fit to data when using Archimax copulas as compared to Archimedean and extreme value copulas [6, 8, 9]. 

We hope that through the proposed methods and release of source code, more applications of Archimax copulas will follow.
Additionally, we note that extreme value copulas are already popular in a variety of environmental and financial situations, and are a major component of Archimax copulas.

**(g) [Pairwise dependence]** Archimax copulas allow different strength of dependence for pairs of variables, and it can be achieved through appropriately defining the spectral random variable $\mathbf{W}$. 

You are correct that the Archimedean copula is limited due to the symmetry assumption of the variables and that choosing an appropriate spectral random variable $\mathbf{W}$ removes the symmetry in the Archimax case. An example of a case involving asymmetry would be to define the spectral random variable $\mathbf{W}$ such that the stdf matches the stdf of the asymmetric logistic copula [10].

**(h) [Correction]** $\mathbf{X}\in[0,\infty)^d$, $\mathbf{U}=(\varphi(X_1),\cdots,\varphi(X_d))\in[0,1]^d$ follows an Archimax copula. 

We thank you for pointing out this error and we have corrected it in the revision. 
The dependence of $\mathbf{X}$ follows an Archimax copula when appropriately transformed by the marginal CDFs.
In this case $\mathbf{U}=(\varphi(X_1),\cdots,\varphi(X_d))$ follows an Archimax copula.

---

References

[1] S. Demarta and A. J. McNeil. The t copula and related copulas. International Statistical Review, 73(1):111–129, 2005. ISSN 03067734, 17515823.

[2] T. Kollo and G. Pettere. Parameter estimation and application of the multivariate skew t-copula. In Copula Theory and Its Applications, pages 289–298, 2010. Springer Berlin Heidelberg.

[3] T. Yoshiba. Maximum likelihood estimation of skew-t copulas with its applications to stock returns. Journal of Statistical Computation and Simulation, 88(13):2489–2506, 2018.

[4] M. S. Smith, Q. Gan, and R. J. Kohn. Modelling dependence using skew t copulas: Bayesian inference and applications. Journal of Applied Econometrics, 27(3):500–522, 2012.

[5] I. Kojadinovic, J. Segers, and J. Yan. Large-sample tests of extreme-value dependence for multivariate copulas. LIDAM Reprints ISBA 2011025, Université catholique de Louvain, Institute of Statistics, Biostatistics and Actuarial Sciences (ISBA), 2011.

[6] S. Chatelain, A.-L. Fougères, and J. G. Nešlehová. Inference for Archimax copulas. The Annals of Statistics, 48(2):1025 – 1051, 2020.

[7] A. Charpentier, A.-L. Fougères, C. Genest, and J.G. Nešlehová. Multivariate Archimax copulas. Journal of Multivariate Analysis, 126:118–136, 2014.

[8] A. J. McNeil and J. Nešlehová. From Archimedean to Liouville copulas. Journal of Multivariate Analysis, 101(8):1772–1790, 2010.

[9] T. Bacigál, V. Jágr, and R. Mesiar. Non-exchangeable random variables, Archimax copulas and their fitting to real data. Kybernetika, 47(4):519–531, 2011.

[10] A. Stephenson. Simulating multivariate extreme value distributions of logistic type. Extremes, 6 (1):49–59, 2003.

 This paper presents new methods for inference and sampling for Archimax copulas. Archimax copulas are a family of copulas defined through an Archimedian generator and a stable tail dependence function (stdf). In order to discuss the inference and sampling for Archimax copulas, the authors first proposed inferential and sampling methods for Archimedian generator and stdf. Then, combining these methods, the methods for inference and sampling for Archimax copulas are established. In experiments, it is seen that the proposed inferential methods for Archimedian generator and stdf show satisfactory performance. Other experiments are given to illustrate that Archimax copulas outperform, or work as well as, some existing models for a couple of real datasets. *** STRENGTHS ***

(a) [Originality] The presented inferential and sampling methods for Archimax copulas seem new. These methods are derived mainly by combining existing methods for Archimedian generators and stdfs.
 
(b) [Quality] The paper seems technically sound. Comprehensive experiments are given to assess the performance of the proposed inferential methods and compare submodels of Archimax copulas with some existing models.

(c) [Clarity] The paper is clearly written in general. Section 2 providing the background of the presented theory would be helpful for readers who are not family with copulas.

(d) [Significance] Archimax copulas are flexible models which include Archimedian copulas and extreme-value copulas as special cases. Therefore the proposed inferential and sampling methods for Archimax copulas could be useful in practice when flexible modelling is required.
 

*** WEAKNESSES ***

(e) [Quality] Apart from Archimax copulas, there exist other flexible families of copulas such skew-$t$ copulas (see, e.g., Joe [49], Section 3.17.2). The paper does not sufficiently compare the Archimax copulas with those existing copulas.

(f) [Significance] I am not sure about the popularity of Archimax copulas and the importance of the related theory this paper presents. (Nonetheless I appreciate the results of experiments which suggest the usefulness of the Archimax copulas.)
 (g) For the multivariate Archimedean copula (equation (3.4), [49]), all the pairs of variables have the same strength of dependence, and this is considered a drawback in terms of flexibility in general. How about the Archimax copulas? Is it possible to solve this problem by appropriately defining the spectral random variable $W$?

(h) In lines 107 and 108, it is mentioned that $X$ defined in (5) follows an Archimax copula. However it seems to me that $X$ takes values in $[0, \infty )^d$ in general and therefore the distribution of $X$ is not a copula which is defined on $[0,1]^d$. I think a correction is necessary to avoid this misunderstanding.

(i) I would appreciate the authors' responses to my comments (e) and (f).

 The authors have addressed the limitations and potential negative societal impacts of their work in Section 5. Depending on the authors' response to my questions (g) and (i), I might claim that the usefulness of Archimax copulas is limited.

 The authors propose scalable estimation and sampling procedures for Archimax copulas. On simulated and real data, they demonstrate that Archimax copulas fit using their procedure can model complex data with dependencies between extreme values accurately, in comparison to existing deep generative methods. The paper bridges two important areas of research: copulas and deep generative modeling. It is highly original (the first method of its kind) and technically excellent. It is also potentially very significant in its impact: modeling rare events is critical to managing risk in real world applications, and relying naively on modern deep generative approaches can potentially be very problematic. The paper is overall quite clear, though there are two areas where I struggled: first, an intuitive explanation as to why Archimax copulas are good for modeling dependencies between extreme events would be helpful to the reader; second, I’d appreciate a concise statement of the complete model up front, explaining how the deep generative model determines the stdf.  See above for suggestions. The authors clearly explain some of the method's important limitations.

 This paper proposes novel procedures for both inference and sampling of Archimax copulas. This family of copulas is important due to their ability to represent the bulk and tail of distributions simultaneously which can be suitable for healthcare and hydrology real-world data. The authors propose a 'hybrid' approach mixing copulas and neural networks to allow for more flexible estimation. In experiments, the proposed method is compared to SOTA density modeling techniques and the results suggest that their method can extrapolate to tails and scale to higher dimensions.  Strengths:
  - Originality - to the best of my knowledge this is the first work to address flexible density estimation of bulk and tail distribution with Archimax copulas.
  - Quality - the authors put a lot of effort into including a high level of technical details and experiments in the paper and appendix 
  - Clarity - since copulas are not a straightforward tool in the machine learning community, I appreciate the background overview and related work mentioned throughout the paper and supplementary. 
  - Significance - considering the tails not only bulk of the data is overlooked problem and can be very significant in many real-world applications

Weaknesses:
  - Originality - although the presented methodology is novel, it does still build on existing work regarding Archimax copulas
  - Quality - I would expect more challenging/motivating experimental results to support the claims of significance and contribution from the introduction
  - Clarity - A running example or one real data motivating example could improve/clarify why Archimax copulas are an appropriate/necessary tool in critical scenarios. This can help bring the paper closer to the ML community, make it more relevant for readers.
  - Significance - since copulas are still not widespread at the ML conferences, I feel like additional motivation for such papers is needed, either to showcase superiority on large-scale real-world datasets or find some new tasks where SOTA models fail. I also appreciate the code submission and effort of implementing everything in Python (rare for statistics methods). 
 


 1. Complexity of the proposed algorithms is not mentioned. Is there any trade-off between flexibility and scalability?
2. Are there any ablation studies regarding data efficiency? Seems like such non-parametric methods can be data-hungry
3. Will this generative network work for non-tabular data? Can there be an example of another NN architecture?
4. How do the sampled marginals compare to the original data?
5. To increase the significance/contribution of the paper, could you add an experiment for out-of-distribution detection for example? (since this is mentioned as a limitation of existing methods) maybe one of the experiments you have already can be used to present some uncertainty scores.
6. related work that is missing:  
Bhatia, Siddharth, Arjit Jain, and Bryan Hooi. "Exgan: Adversarial generation of extreme samples." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 8. 2021.

Minor:
Figure 1 is not referenced in the text

----------------------------------------------------------------------------------------------------------------------------------------------------------------
### Update after authors response:

Thank you for the thorough response and addressing all of the comments in the updated manuscript.
I am happy to accordingly raise the score. The authors have addressed the limitations and potential societal negative impact of their work.

 The authors propose an efficient inference and sampling schemes for archimax copulae based on learning of the generator and stable tail dependence functions through deep learning techniques.  *Originality*. The work is original. To my knowledge, the authors propose a new method to infer and sample from archimax copulas extending the previous work in this area.

*Clarity*. The authors go a long way to make sure the paper is accessible by a larger machine learning community. They provide necessary background info on copulae and their use in machine learning. They also provide extensive derivations in the appendix. That said, the material itself is quite dense. I have not found typos. One minor thing is that Figure 1 does not seem to be referenced anywhere in the text.

*Quality*. The authors build upon a previously developed theory and methods to derive a scheme to infer archimax copulae via the means of deep learning models which also allows for an easy sampling from. The authors provide an analysis of performance of the proposed method comparing it to other state-of-the-art methods. The details of the experiment setups are given in detail in the supplementary material. The authors provide extensive background/related work review in both the main text and the appendix.

*Significance*. Inference of the multivariate distributions from the data is a core statistical/machine learning problem. The authors propose a way to infer multivariate dependencies via archimax copulae representations of which are learned via deep learning models. The method also allows for sampling from the learned distribution. The method serves both the bulk and the tails of a distribution. With that said, a proposed method is of major significance for the field.
 As far as I understand when optimizing the presentations of the copula generator and stable tail dependence function the authors start with an initial copula generator $exp(-x)$. Have you considered other initial generators and how would this affect learning of the copula?

Have you considered estimating several copulae as in a hierarchical archimedean copula?

What do you say about modifying the architecture of the generative neural networks? The authors faithfully address limitations and possible negative impact of the proposed method.

Meta-review: The paper proposes a new method for inference and for sampling in archimax copulas. All the reviewers praised the soundess and clarity of the paper, the novetly of the ideas and the experimental results. Copulas might not be one of the core topics of the NeuRIPS community, but the reviewers pointed out that:
1) the authors did a great job at explaining copulas to the ML community, a valuable tool to model extreme events.
2) the method builds a connection between copula and deep generative modeling, and hence opens new research directions.
Hence, they all enthusiastically recommend to accept the paper, and I agree with them.

Some of the reviewers [HCqW, 5Yjr]  also supported the idea to highlight the paper (oral or spotlight presentation).

Decision: Accept