Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a differentiable kernel-based estimator of differential entropy, named KNIFE.  
KNIFE-based estimators can be applied to both conditional (on either discrete or continuous variables) differential entropy and mutual information. In essence, KNIFE leveraged a kernel-based nonparametric likelihood estimator for the plug-in estimate of differential entropy, where the basis and covariance parameters are learned via MLE. The proposed method is validated on high-dimensional synthetic data and guiding the training of neural networks for real-world tasks, including domain adaptation and fair learning. 


main_review: Strengths:
1. Estimation of (differential) entropy and the related mutual information are topics in machine learning of fundamental importance, which drive a wide range of applications.
2. The proposed method is relatively easy to implement. 
3. Experiments on the real-world dataset have decent coverage of important applications concerning mutual information. 
4. This paper is written with clarity and is fairly easy to follow. 

Weaknesses:
1. Originality. There is not much novelty in this work. The proposed solution is a simple plug-in estimator based on adaptive kernel density estimation. The techniques used are very standard and there is no new theory developed.  
2. Lack of comparisons, the following competing estimators should be covered in the discussion or compared in experiments.
 - Plug-in estimators based on likelihood-ratio estimates, like neural estimators (or JSD estimator) and ML/least-square estimators (see [1] and reference therein)
 - Nearest-neighbor estimator [2]. 
3. Lack of in-depth discussions. Variational schemes have been proposed to address the inadequacy of plug-in estimators in such settings (likelihood estimation for complex distributions in high-dimensions is a long-standing challenge in statistics and machine learning), at least the paper should expand the discussion on that point. The experiments have focused on low-dimensional setups, so the proposed method works okay. But performance on high dimensions random variables is unknown. 

[1] Suzuki, Taiji, et al. "Approximating mutual information by maximum likelihood density ratio estimation." New challenges for feature selection in data mining and knowledge discovery. PMLR, 2008.
[2] Berrett, Thomas B., Richard J. Samworth, and Ming Yuan. "Efficient multivariate entropy estimation via $ k $-nearest neighbour distances." The Annals of Statistics 47.1 (2019): 288-318.

summary_of_the_review: The quality is okay but falls below the ICLR threshold because of the lack of originality and in-depth discussions.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an estimator $\widehat{h} _ {\text{KNIFE}}$ for differential entropy suited for applications in deep learning. A set of desirable requirements is specified all of which are satisfied by $\widehat{h} _ {\text{KNIFE}}$ but not by other commonly used estimators. Extensive experiments are performed to justify the new estimator.
main_review: The paper tackles the important problem of estimating commonly used information-theoretic quantities like differential entropy and mutual information. For a random variable $X$ with density $p$, differential entropy is $h(X) := - \int p(x) \log p(x) \, \mathrm{d}x$. Given $n$ i.i.d. samples $\\{x_i\\} _ {i=1}^n$ of $X$, a naive monte carlo estimate $-\frac{1}{n} \sum _ {i=1}^n \log p(x_i)$ cannot be calculated since $p$ is unknown. Therefore, a common solution to this problem is to estimate the density $p$ using an i.i.d. sample $\\{x'_i\\} _ {i=1}^m$ of $X$ _independent_ of $\\{x_i\\} _ {i=1}^n$. This paper uses a modification of standard kernel density estimators to estimate $p$, and therefore gets
$$
\widehat{h} _ {\text{KNIFE}} = -\frac{1}{n} \sum _ {i=1}^n \log \widehat{p} _ {\text{KNIFE}}(x _ i; \theta).
$$

It's easier to criticize, so let me do that first. The core of the paper, Section 2, where the proposed estimator is discussed is devoid of necessary details. For example, what is $\mathbf{a}$ in the proposed estimator for $\widehat{p} _ {\text{KNIFE}}$? It is unclear how $\widehat{p} _ {\text{KNIFE}}$ is estimated: what data are used (is it a subset of training data to maintain independence?, is it all training data? why?)? In the paragraph "Learning step:" of Section 2.2, why would _minimizing_ $\widehat{h} _ {\text{KNIFE}}$ be desirable? isn't the goal to get $\widehat{h} _ {\text{KNIFE}}$ close to $h(X)$? Although an interesting result, how is Theorem 1 relevant for KNIFE estimator? The estimator $\widehat{h}$ used in Theorem 1 is not $\widehat{h} _ {\text{KNIFE}}$. The selling point of the paper is the added flexibility provided by $\widehat{h} _ {\text{KNIFE}}$ over other estimators like $\widehat{h}$.

The biggest strength of the paper is the extensive experiments to justify that the proposed estimator is in fact better than other estimators on many tasks.
summary_of_the_review: The paper needs some rewriting of the technical sections, but otherwise is a good paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper provides a new approach to estimating differential entropy called KNIFE that is also applied to mutual information estimation. The authors define their estimator using a parametric model based on estimating a KDE. The authors provide some theoretical analyses of the estimator and multiple empirical experiments where the proposed estimator outperforms several other estimators.
main_review: While the empirical results are promising, ultimately this paper is incomplete. First, the authors completely ignore the many estimators of information theoretic measures that have good empirical results and strong theoretical results. These include [R1-R5]. The authors should include these in the discussion of prior work at the very least and compare the empirical results to the ones that are also differentiable.

Second, the theoretical results are very weak and may even be wrong. Minimax estimation results for estimating entropy, divergence, and mutual information have been established [R1,R2,R6]. All of these results show that lower bounds on the estimation accuracy depend on the dimension of the data as well as the smoothness of the densities. The authors' convergence rate results do take into account the density smoothness but appear to be independent of the data dimension. The authors should resolve this apparent discrepancy.

Other comments:
In Section 1.2 the authors state: "In machine learning applications, however, the use of asymptotic results is not realistically justified." I disagree with this. Asymptotic results often relate to finite sample results, i.e. many estimators with good asymptotic theory often have better empirical results as well. 

While it's nice that KNIFE adapts to new data, it seems that most nonparametric estimators would automatically adapt. Thus the lack of adaptability seems to be more a problem with parametric estimators and wouldn't be an issue with the estimators given in [R1-R5]. 

I'm somewhat skeptical about minimizing the LHS of (6). While the explanation provided makes intuitive sense, in practice, due to finite samples, it does seem like underestimating the entropy could still happen. Perhaps some kind of concentration inequality could be used to obtain a more accurate bound instead of using the LLN?

[R1] Moon et al, "Ensemble estimation of generalized mutual information with applications to genomics," IEEE Trans. on IT, 2021.
[R2] Kandasamy et al., "Nonparametric von Mises Estimators for Entropies, Divergences and Mutual Informations," NeurIPS, 2015.
[R3] Singh and Poczos, "Exponential Concentration of a Density Functional Estimator," NeurIPS, 2014.
[R4] Sricharan et al., "Ensemble Estimators for Multivariate Entropy Estimation," IEEE Trans. on IT, 2013.
[R5] Berrett et al, "Efficient multivariate entropy estimation via k-nearest neighbour distances," Annals of Statistics, 2019.
[R6] Birge and Massart, "Estimation of Integral Functionals of a Density," Annals of Statistics, 1995.

Post-rebuttal update: I have read the authors' response to my review and the other reviewers. I appreciate the revisions the authors have made up to this point, especially regarding the theoretical results. However, I do believe that comparisons to other estimators should be done before I can recommend publication. In their comment, the authors claim that some of these estimators are not well-suited for the proposed use-cases. However, many of these estimators are based on plug-in approaches similar to the KNIFE estimator. Thus I believe that they can be compared to as well.

I have thus raised my score to be marginal, leaning towards reject. 

summary_of_the_review: The authors do not mention nor compare to other important estimators of differential entropy and mutual information. Furthermore, there are important unanswered questions about the theoretical results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes an entropy / mutual information estimator that is suitable for representation learning, by extending the Parzen-Rosenblatt estimator with learnable centroids, bandwidth matrices and coefficients.  The authors prove consistency, and demonstrate that on various downstream tasks where MI-based regularization is needed, the proposed method outperforms previous work on entropy / MI estimation.
main_review: ## Original Review

The methodological novelty of this work is the learnable weighting coefficients and centroids, i.e., modifying the Parzen estimator $\frac{1}{M}\sum_i k_{A_i}(\cdot - x_i)$ as $\sum_i u_i k_{A_i}(\cdot - a_i)$, where the parameters $(u_i, a_i)$ are chosen to minimize the KL divergence w.r.t. the data distribution, and $A_i$ is the learnable bandwidth.  It is argued that, in ML tasks where the data $x_i$ are the learned representations and change during training, the modification allows the estimator to be both flexible and efficient.

The argument will be reasonable in problems where learnable bandwidth to be necessary, yet more flexible estimation methods (e.g. with neural energy based models) are too expensive.  The strength of the method mostly lies in its practicality (in representation learning tasks): it does not provide proper lower or upper bounds for mutual information, which is the central quantity of interest in the downstream tasks.  The novelty is limited, but this is fine if there is consistent improvement in empirical performance.

I don't have any major issues with this work. There are nonetheless a few questions that need clarification:

1. Consistency of the results with previous work:
    * Table 1 doesn't appear consistent with any tables in Mahabadi et al (2021), e.g. in the MRPC experiments the accuracy is consistently higher than F1 in Mahabadi et al (2021), but not here. 
    * Table 2 doesn't appear consistent with Table 2 in Cheng et al (2020a), the baseline performance here seems consistently worse, and the difference is significant with the only exception of M->U/U->M.

In both cases, this paper claims to follow the setup in the corresponding previous work closely, so an explanation on the difference will be appreciated.

2. Usefulness of MI regularization in downstream tasks: while I can imagine MI-like/-inspired regularization can be useful, I'm less certain if accurate estimation of MI always translate to improvements.  For example, from a quick scan it appears Mahabadi et al (2021) used a very heuristic estimation of MI, by fitting multivariate Gaussians on the joint and conditional distributions, yet demonstrated a similar level of improvement in GLUE as here.  While the difference in experiment setup prevents any definitive conclusion, it would be convincing if the authors could implement the heuristic in Mahabadi et al (2021) in the GLUE experiment here and report the performance.

3. For entropy/MI regularization for downstream tasks, the gradient (score) estimators in the variational inference literature may serve the same purpose.  See, for example, Section 9.6 in Mohamed et al (2020).  While evaluation may be difficult to fit in the rebuttal timeline, it would greatly strengthen the paper if the authors could eventually compare with some works in this line, e.g., Song et al (2019) and Zhou et al (2020), as they usually claim applicability in problems with a similar complexity.

References:

* Mohamed et al (2020), Monte Carlo Gradient Estimation in Machine Learning, in JMLR.
* Song et al (2019), Sliced Score Matching: A Scalable Approach to Density and Score Estimation, in UAI.
* Zhou et al (2020), Nonparametric Score Estimators, in ICML.

## Post-rebuttal Update

The authors' response addressed my questions about the experiments and the updated proof appears correct.  The contribution of this work is largely empirical -- the listed requirements and the new estimator do not appear very novel to me, although it is understandable that the characteristics of the downstream tasks may prevent the development of more flexible methods.  The experiments clearly demonstrate improvements over past MI-based methods.  However, as I'm not familiar with the evaluation tasks, I cannot evaluate their significance in the broader context; that task will have to be left to the other reviewers.  Therefore, I'm changing my score to 6, in light of the resolved questions about the theory and experiments.
summary_of_the_review: + Simple method with consistent improvement in empirical performance.

- Uncertainties around the experiment results.

I will change my score if the two questions about the experiments are clarified.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies differentiable proxy estimators for density function, which is in turn used to compute various information metrics such as (conditional) entropy and mutual information. The main contribution of this paper is that it generalizes previous kernel-based density estimators by parameterizing the anchors and mixture probability of kernels. The advantage of the resulting method is that it 1) has increased capacity 2) it can adapt to input data distribution shift.
The author provides convergence of the resulting entropy, as well as providing extensive empirical studies on synthetic and real data, such as BERT+text classification and Unsupervised Domain Adaptation.

main_review: [Strength]\
Empirical evidence demonstrates the effectiveness of the proposed method in terms of its estimation error and adaptation to underlying data distribution shift. The results seem fairly consistent, at least for the tasks and baselines the authors included.

[Weakness]\
The technical significance of the proposed method seems incremental: The Schraudolph estimator proposed before already includes the covariances as learnable parameters, and there does not seem to be many technical challenges of making the anchor and mixture probability learnable as well. But I'll take the author's defense on this into consideration.

[Other comments]\
$\theta = (A, a, u)$ can be viewed as a parameter for the loss function How is $\theta$ updated, say for the VIBERT example? Is it updated together with the model parameters or in a bilevel fashion (alternating)?. If it is the former, how would it prevent mode collapse? Usually, in the AutoML literature where one wants to learn a surrogate loss function, people would alternatively update loss function parameters and the model parameters to prevent mode collapse or overfitting.

---- post rebuttal update
The author's response and the revised draft addressed my questions. The background information provided in the statement of novelty improves my view on the matter, though I hold to the opinion that extending the Schraudolph estimator does not seem to be particularly challenging. That been said, the proposed method is well motivated and its effectiveness is sufficiently backed by empirical evaluations. Overall it is a paper of quality. I raise the tech contribution score to 3 and confidence to 3 and am inclined towards acceptance.
summary_of_the_review: The technical aspect of this paper is not very significant in my opinion. However, empirical evidence seems sufficient and the improvements look consistent. My rating for this paper lies on the borderline for the moment. It is likely that I'll make changes after the rebuttal.
review_title: null

Meta-review: The focus of the submission is the estimation of the Shannon differential entropy (DE). The authors propose a differentiable DE estimator referred to as KNIFE (Kernelized Neural diFFerential Estimator): it is a plug-in method (5) using KDE (kernel density estimation; (4)). KNIFE has parameters including the locations (a), weights (w) and covariances (A) in KDE, which are tuned according to the upper bound heuristic in (6). The approach is illustrated on toy examples and in the context of training neural networks.

Estimating information theoretical quantities is a current topic of machine learning. Unfortunately, as assessed by the reviewers
1) the submission lacks context and comparison to available entropy estimators,
2) the estimator closely follows Schraudolph (2004); the technical novelty is quite limited.

More work and major revision are required.
Decision: Reject