Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper focuses on understanding the tail behavior of normalizing flows through a mathematical and statistical way. Motivated by Jaini et al 2020's work on learning long-tailed distribution via triangular flows, this work proves that the marginal tailedness can be controlled by the tailedness of the marginals of the base distribution in flow-based models. Based on this theoretical insight, the authors propose a new algorithm by leveraging a data-driven permutation scheme to enable a correct tail behavior of the target distribution. 

 
main_review: Strength:

1. Solid and rigorous mathematics foundation 

The theoretical proofs are very helpful and provide a clear insight, which explains the motivation and intuition. 

2. Careful discussion about the related work and current limitations

The authors provide a good review and comparison of the existing works. Also, the limitations and future directions are helpful and insightful. 

3. Well-written and easy to follow

The paper is well-structured and easy to follow. The theoretical proof strongly connects with the experiments, which makes the papers much easier to understand. 

Weakness

 1. The novel contribution is marginal 

This work is mainly inspired by Jaini et al, 2020, who proposed to model long-tailed distribution via normalizing flows. Although the theoretical contribution is strong, the new proposed mTAFs did not show a significant improvement, as shown in Table 1, compared with vanilla and TAF.  

2. Lack of baseline methods and comparison 

The SOTA flow models and architectures are not included in the baselines.  Although it is argued by the authors in the potential work, I still believe the comparison is necessary. Many papers have shown that the affine coupling layers, such as RealNVP, have limited expressivity in handling complex distribution. Either light-tailed or heave-tailed distribution would be more challenging. So the worse performance might be due to the limited representation capability of the vanilla flows. 

3. Experiments need to be improved with large-scale and high-dimensional datasets

Currently, only synthetic toy examples are provided to demonstrate the performance. The dimensionality is also low. If the proposed algorithm is able to scale to high-dimensional problems, it would be very helpful to increase the impact. 

summary_of_the_review: The paper provides a strong theoretical insight but the experiments and baselines are weak.  The contribution seems limited with a marginal improvement compared with the current work.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes an extension to Tail-adaptive flows for learning the tail behavior of target distributions using normalizing flows. The authors propose to learn the tail behavior by learning flows that match the tail properties of the marginal distributions. They achieve this by using a source distribution consisting of marginal distributions with tail properties matching the target distribution. The tail coefficient of the source distribution is set in a data-driven manner using estimators that can estimate this tail coefficient. 
main_review: The paper is  The problem of estimating the tail behavior 

**Pros**: Modelling tail-phenomena (or rare events) in general is a challenging problem made even more difficult in higher dimensions due to the lack of any definition of heavy/light tails in higher dimension. The problem considered by the authors of capturing tail behavior by normalizing flows and limitations of the chosen architecture for flow layers imposes is an interesting and valid problem. The proposed solution that uses estimators to estimate the tail-coefficient for choice of base distribution is interesting and a nice addition to start the optimization of the $\nu$ at a favorable location and to be closer to whats needed. Overall, the paper is very well written and easy to follow. The paper develops the idea in a natural and easy to understand manner.

**Cons**: 

1)  **Motivation**: I found that the problem under consideration was not properly motivated and this issue lingers throughout the paper essentially making the paper come across as just an extension of Jaini et.al 2020. For example, it is not clear from the paper why capturing tails in variational inference paradigms is of importance. It can be shown that the error in modelling a probability density can be bounded arbitrarily well by learning the density properly on a bounded subset of the support of the density. Thus, what are the drawbacks in the model if it is unable to capture the tail phenomena present in the problem?

The authors do make statements regarding the limitations of the push-forward density given the choice of base density and the transformation map. However, I believe a more thorough discussion about the implications of these results (both in general and particularly for normalizing flows) and some recipe or ideas to alleviate these problems will help the paper tremendously.

2) **Significance**: Another weakness I believe of the paper itself is the significance itself. The main result of the paper ie that of specifying and ensuring that marginal distributions have the correct tail coefficient is a direct extension of the work of Jaini et.al 2020. In some ways, the optimization problem presented in Jaini et.al already can encompasses the correct marginal tails by optimizing over the $\nu$ vector. This weakness of significance is further amplified by the lack of strong empirical results. In the present form, the experimental results come across more as proofs-of concept rather than proving strong empirical support. Furthermore, I'd be interested to see the gain in starting the optimization with tail-coefficients estimated using the various estimators vs random initialization and letting the process figure these out. Will the first step of using estimators lead to any significant gains?

**Other comments:**
- Definition 4 it seems is a bit restrictive as well since it completely side-steps any issues with differences in tail-coefficients. For examplke different ,marginals can be heavy tailed but have different degrees of heaviness. In that case, is it correct to call the two distributions having the same tail behavior?

- It seems in the experiments that if the tail estimator estimates that some marginals are light-tailed, a normal distribution is used. However, again there can be degrees of light-tailedness (see Jaini et.al 2020) eg. uniform vs normal. In these scenarios too, a lighter tailed distribution cannot be pushed-forward to another light-tailed distribution but with higher tail coefficient with lipschitz maps. Thus, the problem of mismatched tails may still persist.
summary_of_the_review: Overall, the paper studies a pertinent and difficult problem. However, in the current form, the present manusript provides only initial proofs-of-concept for potentially interesting ideas. These ideas need to be demonstrated and explored in more detail both in theory and empirically to make the manuscript stronger.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces Marginally Tail-Adaptive Flows (mTAFs), which extend existing work on TAFs to better learn a generative model of heavy-tailed distributions. In particular, they propose a new type of normalizing flow (NF) that can learn marginals with mixed-tail behavior.
main_review: **Strengths:** I thought the paper was interesting. NFs definitely do have their limitations despite their expressivity, and I don’t think the problem of generating distributions with a mixture of both heavy- and light-tailed marginals has been considered before. The paper provides a more general definition of heavy tailedness that extends existing work and uses it to construct their mTAF method.

**Weaknesses:** That being said, I think the paper still requires a significant amount of work in order to demonstrate the efficacy of mTAF. 
- First, it’s not clear to me when you would run into situations where you want to generate distributions with mixed-tail behavior in the marginals. I understand that it would be desirable to generate distributions with heavy tails, but when do we encounter cases where we would like to do both? I think making this clear would definitely strengthen the paper, and could also guide some downstream evaluation tasks.
- The experiments were probably the weakest aspect of the paper. mTAF was only evaluated on a synthetic dataset of 16 dimensions, which seems too small (even for tabular datasets commonly used for evaluating NFs). Also, the evaluations conducted in the experiments did not clearly demonstrate the advantage of mTAF over existing methods. For example in Table 1, does mTAF capture both the light-tailed and heavy-tailed components better than TAF/the base method? (this is hard to tell with just a simple average). Additionally, it’s hard to tell the difference between mTAF and TAF in Figures 2 and 3. I think the paper would be much stronger if the authors could find some compelling use cases of the method beyond synthetic Gaussians, and demonstrate that mTAF both captures all marginals more faithfully (via likelihoods) and can generate samples properly in the tails.
- I also think a big limitation of the method is that mTAF essentially requires separating out the light-tailed marginals from the heavy-tailed marginals (the permutation step where such marginals are grouped into 2 categories). This seems particularly problematic as the real advantage of using NFs is to learn complicated dependencies between all dimensions of the data to best capture the overall density. This is also why I was asking whether there are real-world examples where such mixtures occur, and whether this kind of ``independence assumption’’ makes sense in these scenarios. It seems like mTAF is very restrictive, and I am wondering if maybe that is why it doesn’t significantly outperform TAF and the vanilla baseline.

**Questions:**
- I’m also curious if the method performs worse relative to conventional flows (e.g. MAF) when the distribution in question is only light-tailed or heavy-tailed. It seems like if the tail index estimator is correct, mTAF should return the correct “tail behavior” of each marginal and generate either a light-tailed or heavy-tailed distribution only. Is this the case? Or does mTAF do a worse job at modeling, say, the light-tailed components, etc?
- Additionally, the vanilla baseline has pretty high variance and sometimes seems to perform on par with TAF -- would the authors elaborate upon this point?

**Miscellaneous/minor typos:**
- “Allow [us] to” in Section 2.1
- “By out theory” in Section 5


summary_of_the_review: Although the paper extends an existing approach to learn generative models of distributions exhibiting mixed-tail behavior, the paper has a number of weaknesses: (1) it’s not clear when such mixed-tail behavior arises in the real-world; (2) the class of flows considered are quite restrictive (affine, coupled with a permutation that requires light-tailed and heavy-tailed marginals to be split into two consecutive blocks); and (3) the empirical results are lacking: they only provide experiments on datasets of dim=16.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper is composed of two main parts: 1) A theoretical section where the authors prove that (Lipschitz) triangular normalizing flows cannot map either heavy- or light-tailed base distributions into target distributions with different tail indices for different marginals; 2) An algorithmic section where the authors introduce a new method for modeling distributions with different tail indices for different marginals.

Both the theoretical and the algorithmic parts are a straightforward extension of the analysis and methods introduced in the paper "Tail-adaptive flows" (Jaini, 2020). 
main_review:  Strengths:
- The addressed problem is important. Vanilla normalizing flows (such as other deep density estimators) are poor at tail-estimation, which limits their applicability to many problems in science and engineering. 
- The paper is very well written and it serves as a good introduction to both normalizing flows and heavy-tailed distributions. 
- The offered solution is technically sound. 
- The theoretical analysis is sound and convincing.

Weaknesses: 
- The main weakness of the paper is its very limited novelty. All theoretical analysis and methodological improvements are relatively minor modifications of the work in (Jaini, 2020). The present paper does not contain major new ideas.
- The proposed method is somewhat inelegant as it proposes the use of a separate off-the-shelf tail estimator prior to the flow training. I do agree that it could be the right approach in many applications, however it is a rather obvious idea, not really worth of a top conference publication.
summary_of_the_review: I highly appreciate the clarity and technical soundness  of the paper. However, I cannot recommend acceptance given the very limited novelty.
review_title: null

Meta-review: This paper addresses the performance of normalizing flows in the tail of the distribution. It does this by controlling tail properties in the marginals of the high-dimensional distribution. The paper is well-motivated, and the key theoretical insight has merit. However, the general perspective and methodology appears to be incremental relative to past results. Furthermore, some concerns over correctness remain after discussion with authors. Also, clear baselines and more realistic settings are lacking in the experimental results. Thus, while the paper generally has promising ideas on a pertinent topic, it appears to be not developed enough to merit dissemination.
Decision: Reject