Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a new family of regularized Renyi divergences by infimal-convolution with integral probability metric (IPM). The authors derive its dual variational representation and provide several theoretical results about limits, interpolations, regularized worse-case regret. The authors evaluate the proposed divergence on both synthetic and real datasets, and its application on GANs.
strength_and_weaknesses: (*) Strength:
+ The authors propose a new family of regularized Renyi divergences by infimal-convolution with IPM.
+ The authors derive several theory results for the proposed divergence

(*) Weaknesses:
+ It seems that some results in the submission appear in the Birrell et al, JMLR'2022a. 
+ It is better to give definitions/explanations for some notations before using them.
+ It is unclear how to choose the hyperparameters of the proposed divergences (it seems that these parameters play important roles in applications)
clarity,_quality,_novelty_and_reproducibility: It is not easy to follow the ideas in the paper, some more explanations may be useful to improve the flows of the paper (i.e., some notations are given without definitions/explanations which makes readers hard to follow the proposed ideas).
Overall, I think the proposed divergences and their properties are important. However, the authors may need to elaborate with more details to improve the flows of the papers and highlight the new contributions of the submission (and distinguish with results in the literature)

Some concerns are as follows:
+ Equ. (1) is for Renyi divergence and Equ. (2) is for Renyi-Donsker-Varadhan variational formula. Are these equations equivalent? (If not, the authors should use different notations for the Renyi divergence and its variational formula). It is better in case the authors elaborate the relations between (1) and (2)
+ In Equ. (1), the authors should describe the meaning of the notation (P << Q)
+ It seems that the definitions of infimal-convoluation \Gamma-Renyi divergence appears in Birrell et al, JMLR'2022a. Could the authors elaborate the relation between these divergences. Are they the same? or what are the key differences between them. It is also better in case the authors distinguish the relation between the submission and Birrell et al, JMLR'2022a, and emphasize the novelty of the submission.
+ How the authors choose the \alpha nd \Gamma space for the proposed divergence in applications? especially the \Gamma space? How does the choice of \Gamma affect the proposed method in experiments?
+ In Section 3, the authors give some important choices for \Gamma. Could the authors elaborate how to compute the proposed method for these important cases?
summary_of_the_review: The proposed divergences and their properties are interesting. However, the authors may need to elaborate with more details to highlight the contribution and distinguish with results in the literature (especially Birrell et al, JMLR'2022a)
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new divergence which is a convolution between Renyi divergence and other IPM.

strength_and_weaknesses: Strength:

1. The paper proposes a new divergence which include the Renyi divergence and other IPM. It is new and have some interesting properties.
2. The author derives some properties of the new divergence and show its dual form for computation. 
3. They derive the limit formula of divergence and present some special case of the new divergence.
4. Empirical results show that the new divergence is more stable the Renyi divergence when $\alpha >1$. 

Weaknesses:

1. The derivations of mathematical results of the paper are quite straightforward from known results.
2. It is not clear for me the popularity of using new divergence in reality when $\alpha >1$. For $\alpha <1$, I do not see any advantage to have new divergence.
3. In the application on biological datasets, the best results are obtained when $\alpha = \infty$. It means that we do not need to care about other values of $\alpha \in (1,\infty)$, then why do we need to define the new divergence. 
4. In the experiment with GAN, the proposed divergence is used to improve the performance of generative model. The results are shown through the FID scores. We need to choose a particular $\alpha$ such that it obtains the lowest FID. In this case, is there any explanation of using the proposed divergence rather than searching something in a larger space will obtain a lower minimum.


clarity,_quality,_novelty_and_reproducibility: The paper is clear, well-written and easy to follow.
summary_of_the_review: A new divergence is introduced, some its properties are explored, however the real need of new divergence for applications is not well justified.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper discusses a regularized family of Renyi divergences using a variational function form. For this family of divergences the paper discusses various properties relating to limits, interpolations, data processing inequality. Examples of the importance of using the new variational Renyi divergences is demonstrated through examples (and counterexamples). Finally, the paper ends with a discussion of numerical applications: a) the variance of the convex-conjugate Renyi divergence estimators are much lower than that of estimators designed using the Renyi-Donsker-Varadhan variational formula when risk sensitive terms are detrimental to the DV-Renyi estimators; b) proposed infimal-convolution divergences are more suitable in the detection of rare sub-populations for a particular dataset; c) the proposed infimal-convolution divergences can perform better than Wasserstein GAN on the CIFAR-10 dataset; and d) the proposed divergences provides improvement on the ROTMNIST dataset. 
strength_and_weaknesses: Overall the paper is nicely written and describes well Renyi divergence and its background, and provides a good description of the paper's main results. In particular, the paper provides an interesting development to the study of Renyi divergences by introducing a new family of regularized Renyi divergences. 

Some minor comments on the strengths of the paper are:
1. The paper includes clear description of definitions and main results. The highlights of contributions on Page 2 were very helpful to understanding the paper.
2. The paper addresses Renyi divergences, an interesting and important topic, and provides interesting development to available results. The variational formulations of the Renyi divergence and the corresponding regularizations using an IPM (i.e., (6)) are very interesting. 
3. The paper considers a variety of numerical experiments, demonstrating flexibility in applications. 

Some minor comments on the weaknesses of the paper are:
1. In the first experiment (Section 6.1), it would be interesting to see if CC-Renyi also performs better than DV-Renyi when distributions are close. In the experiment shown the two Gaussians are chosen to be N(2,0.5) and N(0,1) respectively and relative variance and MSE numbers are quite extreme for DV-Renyi -- it would be interesting if CC-Renyi also improves upon DV-Renyi for scenarios where say the two Gaussians are N(0,1) and N(0.1,1), for example. 
2. The first section in Section 5 may benefit from some more explanation. 
3. It may be helpful to adjust some of the citation formatting to make the paper more readable. For example, in the very first sentence of the Introduction, the two citations are a bit confusing to the overall readability of the sentence.
4. The details to the last experiment seem to be slightly lacking; it would be helpful to add some more description to the main text. 
clarity,_quality,_novelty_and_reproducibility: Overall, the paper was clearly written (bar a few of the comments listed in the above section). The quality and clarity are good. In terms of originality, the main results in this paper builds on the work within the literature and provides interesting new results. 
summary_of_the_review: Below is a short summary of justification, for more details please refer to the above sections. 

1. The paper is clearly written and states the background and main results adequately. Main ideas such as the addition of regularization to Renyi divergences are nicely conveyed. 
2. The paper introduces interesting new results for the important field of Renyi divergences, and is closely related to the scope of ICLR.
3. Overall, no major issues were identified in the main text.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors propose a new family of Renyi divergences by infimal-convolution with integral probability metric (IPM). The paper is mainly theoretical, with theorems on different aspects of the proposed divergence (e.g., limits, interpolations, regularised worse-case regret). Experiments considered applications to training GANs, with small improvements over baselines.

Reviewers overall welcomed the development of this new family of divergences and the analyses of their theoretical properties in multiple different aspects. The synthetic experiments on reporting variances of Renyi divergence estimates show significant advantages of the proposed approach. Given that high variance of Renyi divergence estimates is a major obstacle for practical usage, I consider these results as showing promising progress towards addressing this challenge.

However, the main concerns are (1) whether the theoretical development is significant as compared with prior work (Birrell et al. JMLR 2022), and (2) the experimental results on GANs do not show significant gains.

In revision, I suggest the authors to better clarify their theoretical contribution as compared with Birrell et al. JMLR 2022, and improve their experiments, e.g., testing variances of the estimators for higher dimensional distributions.
justification_for_why_not_higher_score: There are concerns regarding whether the theoretical contribution is significantly different than Birrell et al. JMLR 2022 paper. Also the experiments on GANs do not show significant practical advantages of the associated GAN algorithm for the proposed divergence.
justification_for_why_not_lower_score: I believe the ideas and results should be of interest to a variety of ML researchers working on GANs, variational inference and differential privacy, where Renyi divergence has been applied to applications in these fields.
Decision: Accept: poster