# An invariant estimation of entropy and mutual information

Alexandre Giuliani, Felix Truong

<figure><img src="image_1.png" /></figure>

## To cite this version:

Alexandre Giuliani, Felix Truong. An invariant estimation of entropy and mutual information. 2024. [hal-04819020v2](https://hal.inrae.fr/hal-04819020v2)

HAL Id: hal-04819020

https://hal.inrae.fr/hal-04819020v2

Preprint submitted on 25 Feb 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# AN INVARIANT ESTIMATION OF ENTROPY AND MUTUAL INFORMATION

Félix Truong

Synchrotron SOLEIL
INRAE

felix.servant@synchrotron-soleil.fr

Alexandre Giuliani

Synchrotron SOLEIL
INRAE

alexandre.giuliani@synchrotron-soleil.fr

## ABSTRACT

This paper addresses the problem of estimating the limiting density of discrete points (LDDP), which serves as a continuous analogue to discrete entropy. We define an invariant measure and propose a method to compute it. This allows the estimation of differential entropy which is invariant to variable transformations. The method uses the k-nearest neighbor (kNN) approach for entropy estimation, addressing the limitations of traditional histogram-based methods. To illustrate the practical application of our approach, we developed the Julia package EntropyInvariant.jl, which implements the invariant measure-based entropy estimation method. The performance of this approach is compared to standard methods, highlighting its invariance properties, especially when dealing with small sample sizes or estimating the mutual information (MI).

**Keywords** Entropy · Differential entropy · limiting density of discrete points · nearest neighbor · invariant measure · Julia

## 1 Introduction

From soft to hard sciences, the identification of relationships between variables is a central goal in analyzing experimental data. Among the various statistical measures available, Mutual Information (MI) has attractive characteristics [1]. Due to its roots in information theory, MI quantifies the amount of information shared between variables, reflecting the loss of uncertainty on one variable given knowledge of another. Unlike metrics such as Pearson correlation, MI is sensitive not only to the covariance, but also to the whole dependence feature of the variables [2]. Therefore, MI is able to report on the non-linear dependency between variables [3, 4]. In addition, MI is robust to small sample sizes, a feature that can mitigate the noise sensitivity observed in other metrics [4]. This can sometimes be misleading for metrics such as Pearson correlation or regressions, which become more robust with larger sample sizes. The concept of MI originates from Shannon's entropy framework [5] first defined for discrete variables and later extended to continuous variables as differential entropy. By definition, the differential entropy is:

$$h(X) = - \int_X \mu_X(x) \log(\mu_X(x)) \cdot dx \qquad (1)$$

with $\mu_X$, the probability density function of the continuous random variable $X$. From the chain rules, we can define the MI of continuous variables as:

$$MI(X, Y) = h(X) + h(Y) - h(X, Y) \qquad (2)$$

with $h(x, y)$ the joint differential entropy as:

$$h(X, Y) = - \int_X \int_Y \mu_{X,Y}(x, y) \log(\mu_{X,Y}(x, y)) \cdot dy \cdot dx \qquad (3)$$

and with $\mu_{X,Y}$ the joint probability density function of $X$ and $Y$. MI is zero when $X$ and $Y$ are independent and increases with the dependence of the variable. This metric is always non-negative and has no upper bond, expressed as
<figure><img src="image_3.png" /><figcaption>Figure 1: Entropy estimation using the histogram method for: (a, b) a normal distribution $N(\mu, \sigma)$ with $\mu = 0$ and $\sigma = [1, 0.5, 3]$; (c, d) a uniform distribution $U(a, b)$ with $a = 0$ and $b = [1, 0.5, 3]$; (e, f) an exponential distribution $E(\lambda)$ with $\lambda = [1, 0.5, 3]$. The average entropy as a function of the sample size is shown in (a, c, e), while the entropy as a function of the number of bins is shown in (b, d, f).</figcaption></figure>

$MI(X, Y) \ge 0$. A significant challenge in many practical applications lies in estimating entropy from finite samples when the underlying probability density function (pdf) is unknown. Techniques for entropy estimation can be broadly classified into two main categories: parametric and nonparametric methods [6]. On one hand, parametric methods assume that the form of the pdf is known, reducing the problem to estimating the parameters of the pdf from the data [1]. On the other hand, nonparametric methods do not make assumptions about the form of the pdf, making them more versatile and widely applicable. Among these, popular approaches include histograms [7] and kernel density estimators (KDEs) [?]. Another nonparametric approach is based on the k-nearest neighbors (kNN) method, which is computationally efficient but may not be accurate [3, 8].

The main drawbacks of the histogram method is represented in Fig.1, where the entropy for normal, uniform and exponential distributions are simulated 100 times. The effects of the distribution parameters, the number of bins and the size of the sample are investigated. For each distribution centered at 0, the parameters $\sigma$, $\lambda$ and $b$ were set equal to 0.5, 1, and 3, respectively. The entropy using the histograms methods is calculated as follows:

$$h(X) \approx h_{\text{binned}}(H) = \sum_{i=1}^{\text{bins}} f_i \cdot \log(f_i) \qquad (4)$$

with $f_i = Card(B_i)/Card(X)$, $\bigcup_{i=1}^{\text{bins}} B_i = X$, $B_i$ the $i^{th}$ interval of length $(x_N - x_1)/\text{bins}$, $X = \{x_1; x_2...; x_N\}$, and $x_i \le x_{i+1}$.

The results Fig.1 (a,c,e) show that normal and exponential distributions requires fairly large sample size for the entropy to converge, whereas the entropy of uniform distributions reaches a steady value with limited sample sizes. Notably, the entropy for a given distribution is not affected by the magnitude of the parameter, as evidenced by the superimposed curves in Fig.1. Entropy estimation using the histogram method is conceptually similar to calculating a Riemann integral [9], where the probability distribution is discretized into intervals, analogous to summing over segments in the Riemann integral. However, unlike a true Riemann integral, which becomes more accurate as segment sizes decrease, the histogram method introduces bias as the number of bin increases. Each bin provides a discrete approximation of the underlying continuous distribution.

This sensitivity to the number of intervals is a notable limitation of the histogram method. Regardless of the distribution, increasing the number of bins leads to a persistent rise in entropy, introducing bias. This bias is evident in Fig.1 (b,d,f) where the average entropy for various sample sizes is increases as a function of the number of bins.

Usually, the number of intervals for histogram-based entropy estimation is chosen empirically, without any guarantee on the correctness of that choice. This can lead to significant bias in the estimation. To address these limitations, Kraskov and coworkers [3] proposed an entropy estimator based on kNN method. This estimator has been shown to be robust even with small sample sizes, [9], which contributes to its widespread popularity [9].

The kNN estimator relies on a single parameter, $k$, which can often be optimized by analyzing the convergence of the differential entropy as $k$ increases. For consistency, the same parameters simulations used for the histogram method are applied, except that the number of bins is replaced by the number of neighbors for the kNN method. As shown in Fig.2, the entropy estimation by the kNN method converges effectively. Additionally, the average differential entropy aligns with the theoretical values. By definition, these theoretical entropy can be computed as follows:

$$\bullet \text{ For } X \sim N(\mu, \sigma), H(X) = \ln(\sigma\sqrt{2\pi e}),$$
<figure><img src="image_4.png" /><figcaption>Figure 2: Entropy estimation using the kNN method for: (a,b) a normal distribution $N(\mu, \sigma)$ with $\sigma = [1, 0.5, 3]$; (c,d) an uniform distribution $U(a, b)$ with $a = 1$ and $b = [1, 0.5, 3]$; (e,f) an exponential distribution $E(\lambda)$ with $\lambda = [1, 0.5, 3]$, and . The average entropy as a function of the sample size is shown in (a,c,e), while the entropy as a function of the number of bins is shown in (b,d,f).</figcaption></figure>

* For $X \sim \mathcal{E}(\lambda)$, $H(X) = 1 - \ln(\lambda)$,

* For $X \sim U(a, b)$, $H(X) = \ln(b-a)$.

It is worth nothing that these theoretical values can be negative, as observed in Fig.2 (c,d,e,f). This occurs because differential entropy is not scale invariant [10], a property that arises from errors during the transition from summation to integration in the limit.

The sensitivity of kNN methods to variable transformation and scaling is well known. Thus, Jaynes [11] discusses and proposes a corrected definition of entropy for continuous variables, called the limiting density of discrete points:

$$H_c(X) = - \int_X \mu_X(x) \cdot \log\left(\frac{\mu_X(x)}{m(x)}\right) \cdot dx \quad (5)$$

where $m(x)$ is the so-called invariant measure function. Unfortunately, a proper definition of this measure, as well as a method for computing it, was not provided. Recently, Nagel and coworkers [12] reported a new approach to normalize MI that is invariant under variable transformations. Following Jaynes's suggestion [11], they propose an invariant measure that reflects complete ignorance of the probability density. Two estimators for this measure has been proposed: the first is a constant estimator, which is sensitive to outliers and assumes that the complete ignorance of a probability distribution is constant; the second method involves taking the inverse volume enclosed by the data points. Both approaches succeed in normalize the MI by subtracting the scaling factor. However, this normalization introduces an inconsistency in the entropy calculations, as the marginal entropy resulting from normalization vary depending on each new pair of random variables.

In the following, we propose a new definition for the invariant measure and means to estimate it. Our approach aims to estimate the LDDP and generalize it to higher dimensions. Finally, we compare this new invariant method with the established standard and highlight its advantages.

## 2 Invariant Measure

In our approach, we use the LDDP instead of differential entropy. According to Jaynes [11], LDDP is closely related to differential entropy (1), and it is explained that if $m(x)$ is constant over some interval of size $r$, and $p(x)$ is essentially zero outside this interval, then:

$$H_c(X) = h(X) - \log(r). \quad (6)$$

This invariance comes from Shannon's from the sum-integral operation of the discrete entropy to the continuous case. From (1) we can obtain:

$$h(aX) = h(x) + \log(|a|). \quad (7)$$

Following the description of Jaynes's, we propose three necessary and sufficient conditions to define the invariant measure $m(x)$.
**Proposition 1:** Let $m(x)$ be an invariant measure with the following properties:

$$[\text{label=}]m(X) = r_X > 0, m(aX) = am(X) = ar_X, m(X+b) = m(X) = r_X,$$

Then we define the invariant entropy $h_c(X) = h(X/(m(X)))$, and we aim to establish the equivalence $h_c(f(X)) = H_c(X)$ with $f$ a function representing a change of variable.

**Proof:** We pose $Y = a \cdot X + b$ as a change of variable, we have $m(Y) = m(a \cdot X + b) = a \cdot m(X) = a \cdot r_X$.

$$
\begin{align*}
h_c(Y) &= h\left(\frac{Y}{m(Y)}\right) \\
&= h\left(\frac{a \cdot X + b}{m(a \cdot X + b)}\right) \\
&= h\left(\frac{a \cdot X + b}{a \cdot r_X}\right), \text{ according to (1.ii)} \\
&= h(a \cdot X + b) + \log\left(|\frac{1}{a \cdot r_X}|\right), \text{ according to (7)} \\
&= h(X) + \log(|a|) - \log(|a|) - \log(r_X), \\
&\quad \text{according to (7) and (1.1)} \\
&= h(X) - \log(r_X) \\
h_c(Y) &= H_c(X), \text{ according to (6)}
\end{align*}
$$

Thus, the invariant entropy is equivalent to the LDDP and is invariant by change of variable.

We have considered two choices to define an invariant measure that respects the three properties of Proposition 1. It is possible to create a vector of the closest distances between each point from a data set. With this vector of distances, one can retain either the median or the mean of these distances (cf. Example 1).

There are several arguments in favor of using the median as an invariant measure. Referring to Edwin Thompson's definition of the "invariant measure" $m(x)$, which describes the distribution with 'complete ignorance' of $x$, we note that if we take a set of data and add a random point, the value of the invariant measure defined by the median remains unchanged. In contrast, the mean value is more likely to be affected if the randomly added point is disproportionate to the data set.

**Example 1:** Let $X = \{2, 3, 5, 7, 11, 17, 19, 23, 29\}$, and the nearest neighbor vector $NN = \{1, 1, 2, 2, 2, 2, 4, 6\}$. The candidates for the invariant measure are $Median(NN) = 2$ and $Mean(NN) = 2.4$. Now suppose, we add a random point $x = 42$, drawn randomly from the real numbers between $-\infty$ and $+\infty$. The randomness of this new point implies it is unlikely to align with the distribution of $X$. ing of a real between $-\infty$ and $+\infty$. The new set becomes: $X = \{2, 3, 5, 7, 11, 17, 19, 23, 29, 42\}$ The corresponding neighbor distances are: $NN = \{1, 1, 2, 2, 2, 2, 2, 4, 6, 13\}$. Thus, with no prior knowledge of $X$, we have $Median(NN) = 2$ and $Mean(NN) = 3.3636$. This example demonstrates that the median remains unchanged, while the mean is sensistive to the addition of an outlier. Hence, we conclude that the median is the more robust choice as an invariant measure. Another argument in favor of the median is that using the mean as an invariant measure can lead to negative entropy, which is undesirable. For instance, the entropy of an exponential distribution $\mathcal{E}(\lambda)$ and a normal distribution $\mathcal{N}(\mu, \sigma)$ calculated using the mean are negative:

<table><caption>Table 1: Comparison of entropy values using the mean and the median for exponential and normal distributions.</caption><thead><tr><th>Entropy (Mean)</th><th>Entropy (Median)</th></tr></thead><tbody><tr><td>$h_{c_{\text{mean}}} (\mathcal{E}(\lambda)) = -0.06$</td><td>$h_{c_{\text{median}}} (\mathcal{E}(\lambda)) = 1.22$</td></tr><tr><td>$h_{c_{\text{mean}}} (\mathcal{N}(\mu, \sigma)) = -0.80$</td><td>$h_{c_{\text{median}}} (\mathcal{N}(\mu, \sigma)) = 1.14$</td></tr></tbody></table>

Thus, the median avoids the issue of negative entropy and preserves the desired properties of the invariant measure.

To test this measure, we use the same simulation as those in Fig.2 and estimate the LDDP instead of the kNN-based entropy according to the invariant measure defined previously. The results Fig.3, exhibit similar properties to the kNN method. Specifically, the convergence of the estimation Fig.3 (a,c,e) is reached with a small sample size around 1500
<figure><img src="image_5.png" /><figcaption>Figure 3: LDDP estimation using the invariant kNN method for: (a,b) a normal distribution $N(\mu, \sigma)$ with $\sigma = [1, 0.5, 3]$; (c,d) an uniform distribution $U(a, b)$ with $a = 1$ and $b = [1, 0.5, 3]$; (e,f) an exponential distribution $\mathcal{E}(\lambda)$ with $\lambda = [1, 0.5, 3]$, and $\cdot$ The average entropy as a function of the sample size is shown in (a,c,e), while the entropy as a function of the number of bins is shown in (b,d,f).</figcaption></figure>

<table><caption>Table 2: List of invariant distribution with their respective invariant entropy.</caption><thead><tr><td>Distribution</td><td>Invariant entropy</td></tr></thead><tbody><tr><td>Arcsine(a, b)</td><td>1.008 ± 0.006</td></tr><tr><td>Uniform(a, b)</td><td>1.060 ± 0.005</td></tr><tr><td>Semicircle(r)</td><td>1.073 ± 0.005</td></tr><tr><td>Symmetric triangular(μ, σ)</td><td>1.106 ± 0.005</td></tr><tr><td>Normal(μ, σ)</td><td>1.150 ± 0.005</td></tr><tr><td>Normal canonical (η, λ)</td><td>1.150 ± 0.005</td></tr><tr><td>Cosine(μ, σ)</td><td>1.114 ± 0.005</td></tr><tr><td>Rayleigh(σ)</td><td>1.135 ± 0.005</td></tr><tr><td>Chi(ν)</td><td>1.149 ± 0.005</td></tr><tr><td>Noncentral chi-squared(ν, λ)</td><td>1.152 ± 0.006</td></tr><tr><td>Kolmogorov()</td><td>1.154 ± 0.005</td></tr><tr><td>Gumbel(μ, σ)</td><td>1.174 ± 0.005</td></tr><tr><td>Logistic(ν)</td><td>1.184 ± 0.005</td></tr><tr><td>Exponential(θ)</td><td>1.227 ± 0.005</td></tr><tr><td>Laplace(μ, σ)</td><td>1.227 ± 0.005</td></tr><tr><td>Cauchy(μ, σ)</td><td>1.517 ± 0.006</td></tr><tr><td>Levy(μ, σ)</td><td>1.973 ± 0.008</td></tr></tbody></table>

points. And Fig.3 (b,d,f), the estimation stabilizes with respect to the number of neighbors, $k$. It is important to note that the standard deviation of the kNN method with and without the invariant measure is displayed on different scales on the y-axis. However, with the invariant measure, we now obtain a consistent and unique entropy value for each distribution, regardless of the distribution parameters. This reinforces the idea that the invariant measure provides a solution to the scaling the problem of differential entropy.

From this measure we can identify famous distributions, where the invariant entropy remains unchanged despite variations in parameters. This invariant entropy can thus be used to classify the shape of different distribution. It is worth nothing that the uniform distribution does not have the smallest entropy, as one might expect, but rather the arcsine distribution. No concrete explanation has yet been found for this behavior.

Also, some additional distributions have an invariant entropy function of the shape parameters like Erlang, Frechet, Gamma, Inverse Gamma, Pareto, Skew Normal, and Weibull distributions.
# 3 Two-dimensional

## 3.1 Generalization

Edwin Thompson defined the LDDP between two variables as

$$H_c(X, Y) = - \int_{X,Y} p(x, y) \log\left(\frac{p(x, y)}{m(x, y)}\right) \cdot dx \cdot dy. \qquad (9)$$

However, Thompson did not express a closed-form result for this quantity. Therefore, following the equivalence between the invariant entropy and the LDDP in one dimension, we propose the following:

$$H_c(X, Y) = h_c(X, Y) = h\left(\frac{X}{m(X)}, \frac{Y}{m(Y)}\right). \qquad (10)$$

To demonstrate the validity of this proposal, we return to the original concept of the kNN estimator of entropy, as introduced by Vasicek [13], which relies on distances between points.

The challenge in two dimensions is that there is no simple formula for expressing the changes of variables. In one dimension, we have the equation $h(k \cdot X) = h(X) + \log(|k|)$, but in two dimensions, for the same scaling factor $k$ we get:

$$h(k \cdot X, k \cdot Y) = h(X, Y) + \log(k^2). \qquad (11)$$

However, it is not straightforward to express $h(k_1 \cdot X, k_2 \cdot Y)$, where $k_1$ and $k_2$ are different scaling factors for each variables. This issue becomes clearer when we consider the calculation of the Euclidean distance between two points, as done in the kNN estimator.

Let $A = (x_i, x_{i+1})$, $B = (y_i, y_{i+1})$, and $d(A, B)$ be the Euclidean distance between A and B with:

$$d(A, B) = \sqrt{(x_i - x_{i+1})^2 + (y_i - y_{i+1})^2} = d_{AB}. \qquad (12)$$

Multiplying both systems X and Y by the same constant, we have $kA = (k \cdot x_i, k \cdot x_{i+1})$ et $k \cdot B = (k \cdot y_i, k \cdot y_{i+1})$. The distance between these new points is:

$$\begin{aligned}
d(k \cdot A, k \cdot B) &= \sqrt{(k \cdot x_i - k \cdot x_{i+1})^2 + (k \cdot y_i - k \cdot y_{i+1})^2} \\
&= \sqrt{k^2 \cdot (x_i - x_{i+1})^2 + k^2 \cdot (y_i - y_{i+1})^2} \\
&= \sqrt{k^2 \cdot [(x_i - x_{i+1})^2 + (y_i - y_{i+1})^2]} \\
&= k \cdot d_{AB}
\end{aligned} \qquad (13)$$

Thus, we have proportionality for an identical change of variable. However, this is no longer possible when we scale the system X and Y by different factors $k_1$ and $k_2$, respectively. The distance between these points is:

$$\begin{aligned}
d(k_1 \cdot A, k_2 \cdot B) &= \sqrt{(k_1 \cdot x_i - k_1 \cdot x_{i+1})^2 + (k_2 \cdot y_i - k_2 \cdot y_{i+1})^2} \\
&= \sqrt{k_1^2 \cdot (x_i - x_{i+1})^2 + k_2^2 \cdot (y_i - y_{i+1})^2}
\end{aligned} \qquad (14)$$

Introducing the notion of invariant measurement, we can express the data in a reference frame emerging from the observations. Let $X' = X/r_X$ and $Y' = Y/r_Y$, where $r_X$ and $r_Y$ are invariant measure for each system. Thus, we have: $k_1 \cdot X' = \frac{k_1 \cdot X}{r_{k_1 \cdot X}} = \frac{k_1 \cdot X}{k_1 \cdot r_X} = \frac{X}{r_X}$ and $k_2 \cdot Y' = \frac{k_2 \cdot Y}{r_{k_2 \cdot Y}} = \frac{k_2 \cdot Y}{k_2 \cdot r_Y} = \frac{Y}{r_Y}$.

Now $A' = (\frac{x_i}{r_X}, \frac{x_{i+1}}{r_X})$ and $B' = (\frac{y_i}{r_Y}, \frac{y_{i+1}}{r_Y})$ are expressed in independent reference frames. The distance between these newly scaled points is:

$$\begin{aligned}
d(k_1 \cdot A', k_2 \cdot B') &= \sqrt{(k_1 \frac{x_i}{r_{k_1 \cdot X}} - k_1 \frac{x_{i+1}}{r_{k_1 \cdot X}})^2 + (k_2 \frac{y_i}{r_{k_2 \cdot Y}} - k_2 \frac{y_{i+1}}{r_{k_2 \cdot Y}})^2} \\
&= \sqrt{(k_1 \frac{x_i}{k_1 \cdot r_X} - k_1 \frac{x_{i+1}}{k_1 \cdot r_X})^2 + (k_2 \frac{y_i}{k_2 \cdot r_Y} - k_2 \frac{y_{i+1}}{k_2 \cdot r_Y})^2} \\
&= \sqrt{\left(\frac{x_i}{r_X} - \frac{x_{i+1}}{r_X}\right)^2 + \left(\frac{y_i}{r_Y} - \frac{y_{i+1}}{r_Y}\right)^2} \\
&= d(A', B')
\end{aligned} \qquad (15)$$
<figure><img src="image_6.png" /><figcaption>Figure 4: Estimation of the MI between three independent and random variables as a function of the number of points (a,c,e) and the number of bins on right (b,d,f) for the histogram method, and the number of neighbors for the kNN method. The histogram method results are shown in (a,b), the kNN method in (c,d) and the invariant estimation in (e,f). The variables follow $X \sim \mathcal{N}(0, 0.1)$, $Y \sim \mathcal{N}(0, 1)$ and $X \sim \mathcal{N}(0, 10)$.</figcaption></figure>

Thus each system can have its own change of variable, and we have $h(X/(m(X))$, $Y/(m(Y))$ invariant. We can generalize the invariant joint entropy of $n$ random variables as $h_c(X_1, X_2, ..., X_n)$ which is equivalent to $H(X_1, X_2, ..., X_n)$. The importance of the generalization lies in the use of the famous chain rules, allowing to build custom metrics, from joint entropy to conditional entropy. A commonly used metric is the MI, defined as:

$$MI(X, Y) = h(X) + h(Y) - h(X, Y), \qquad (16)$$

and the goal is to estimate the MI with the invariant measure.

## 3.2 Mutual information

The MI is a powerful metric that provides deeper insights compared to simple correlation. To ensure that this generalization in two dimensions remains valid, we perform a simulation of three independent variables. Each variable $X$, $Y$, and $Z$ follows a normal distribution with respective standard deviations $\sigma = 0.1$, $\sigma = 1$, and $\sigma = 10$. The differing standard deviation are expected to highlight the advantages of the invariant entropy estimation. The theoretical value of two independent variable should be 0.

In Fig.4, we estimate the average MI using three methods: the histogram method, the kNN method and the invariant method. Panels (a,c,e) show the MI as a function of the number of points, while panels (b,d,f) expose its dependence on the number of bins (for the histogram method) and the number of neighbors (for the kNN method). For the histogram method a large number of points is required to converge to zero, albeit with a slight bias. Additionally, in panel (c), the convergence of $I(X; Z)$ is slower compared to $I(X; Y)$ and $I(Y; Z)$. This behavior can be attributed to the large magnitude difference between variables $X$ and $Z$, which leads to slowest convergence. In contrast, the invariant estimation method in panel (d) reaches the theoretical value much faster.

Examining the parameter of each method, the histogram estimation shown a clear bias as illustrated Fig.1, leading to divergence in the MI estimation. In panel (d), $I(X; Z)$ is particularly problematic, exhibiting negative values and divergence. Additionally, the estimation error is higher for the histogram method and increase more rapidly compared to the invariant estimation. These simulations demonstrate the stability of the invariant approach both in terms of number of points and the number of neighbors. Furthermore, the invariant estimation consistently achieves a smaller error than the other methods.

In this observation, we aim to study how MI behave as a function of noise for both the kNN-based MI and the invariant MI. For reference, the correlation is also computed. In this example, we simulate four relationships as shown Fig.5. The panel (a) illustrates a strong non-linear dependence between the variables $x$ and $y$. The correlation is 0.02, highlighting its inability to detect the relationship. In contrast, the kNN-based MI is 0.37, and the invariant MI 3.89, showcasing the strength of information theory in quantifying dependencies. However, the kNN-based MI is significantly lower than the invariant MI. This trend persists in panel (b), where noise is added to the $y$ variable. Surprisingly, the kNN-based MI increases but remains four times lower than the invariant MI. In panel (c), as the noise level increases further, the MI values estimated by both methods become comparable. Finally, panel (d) demonstrates a similar observation as in Fig.4(d), with higher standard deviations from zero (the theoretical value for independent variables) and even negative values in some cases. This highlights the limitations of the kNN-based MI in noisy conditions. Overall, it is evident that the invariant MI provides a more robust quantification of relationships, especially as noise increases. This quantification remains more consistent and reflective of the true underlying relationship compared to the kNN-based MI.
<figure><img src="image_7.png" /><figcaption>Figure 5: Comparison of the values of correlation (Cor), kNN-based MI (MI(kNN)) and invariant MI (MI(Inv)) across four noise levels.</figcaption></figure>

# 4 Conclusion

In this work we present an invariant entropy with a method to estimate it, equivalent to the LDDP. This equivalence enables the calculation of the entropy of continuous variables in a way that remains invariant under variable scaling. Compared to other popular techniques, our method demonstrates smaller errors. Furthermore, its enhanced sensitivity to noise facilitates more precise quantification of dependent variables. Our approach introduces no significant computational complexity and remains efficient in terms of runtime, making it valuable for advancing entropy-based analysis. Beyond its technical contributions, we hope that this work will deepen the understanding of the concept of complete ignorance in the context of distributions. We also explore the role of invariant distributions, as presented in Table 1, and their potential to enhance our understanding of the maximum entropy principle, a framework critical to numerous disciplines. By offering a perspective that harmonizes theoretical foundations with practical utility, invariant entropy opens new avenues for research and application across diverse scientific fields.

# A Computational details

The calculations and examples of this manuscript have been made using the Julia computer language and the EntropyInvariant.jl package (https://github.com/Entropy-Invariant/EntropyInvariant.jl). The package is registered through Julia package manager and has to be installed as follows:

```
julia> ] add EntropyInvariant
```

The histogram, kNN and invariant method to estimate the entropy are implemented. The EntropyInvariant Julia package computes a variety of entropy-based metrics through the following entropy main public functions:
```julia
entropy mat::Matrix{<:Real};
    method::String = "inv",
    nbins::Int = 10,
    k::Int = 3,
    base::Real = e,
    verbose::Bool = false,
    degenerate::Bool = false,
    dim::Int = 1
):::Real
```

The choice of the method is made using the "method" variable. Setting the variable to "kNN" selects the kNN method, "histogram" calls the histogram method and "inv" uses the invariant estimation.

Example:

```julia
julia> data = rand(100, 2) # 100 points in 2D

#Using the kNN method
julia> value = entropy(data, method="knn", k=5)

# Using the histogram method
julia> value = entropy(data, method="histogram", nbins=10)

# using the invariant method
julia> value = entropy(data, method="inv", k=3)
```

The entropy function is used to estimate entropy in Figures 1, 2, and 3. For reproducibility, we fixed the seed. The following code reproduces the results shown in Figure 3. Similarly, by changing the value of the method argument and replacing the k argument with bins, we can estimate entropy using the histogram or k-NN estimators, as shown in Figures 1 and 2.

```julia
julia> seed = 42
julia> Random.seed!(seed)
julia> N = 100:500:10100 # sample size
julia> KNN = 3:3:30 # list different k
julia> NB = 100; # number of simulation
julia> inv_knn_nor = zeros(NB, length(KNN), length(N))
julia> inv_knn_uni = zeros(NB, length(KNN), length(N))
julia> inv_knn_exp = zeros(NB, length(KNN), length(N))
julia> for nb in 1:NB
    for knn in eachindex(KNN)
        for n in eachindex(N)
            nor_ = rand(Normal(0,1), N[n])
            inv_knn_nor[nb, knn, n] = EntropyInvariant.entropy(nor_, k=KNN[knn], method="inv")
            uni_ = rand(Uniform(0,1), N[n])
            inv_knn_uni[nb, knn, n] = EntropyInvariant.entropy(uni_, k=KNN[knn], method="inv")
            exp_ = rand(Exponential(1), N[n])
            inv_knn_exp[nb, knn, n] = EntropyInvariant.entropy(exp_, k=KNN[knn], method="inv")
        end
    end
end
```

The invariant method is set as the default due to all the benefits explained. In this simple example, a random vector is scaled using a trivial linear function to showcase the robustness of the invariant measure.

```julia
julia> n = 1000
julia> p1 = rand(n)
julia> println("Entropy invariant:")
julia> println(entropy(p1))
julia> println(entropy(1e5*p1.-123.465))
julia> println(entropy(1e-5*p1.+654.321))
```
Entropy invariant:
1.087225165954103
1.087225165954023
1.087224434765357

In information theory, different sets of quantities can be estimated using chain rules. The applications of such entropy-based metrics are diverse, and the interpretation of some metrics leads to interesting discussions. Mutual information is one such metric. As shown in Figure 4, the following code produces the data for graphics (a,d).

julia> mi_histogram2_nor1 = zeros(NB, length(BINS), length(N))
julia> mi_histogram2_nor3 = zeros(NB, length(BINS), length(N))
julia> for nb in 1:NB
  for bins in eachindex(BINS)
    for n in eachindex(N)
      nor1_ = rand(Normal(0, 0.1), N[n])
      nor2_ = rand(Normal(0, 1), N[n])
      nor3_ = rand(Normal(0, 10), N[n])
      mi_histogram2_nor1[nb, bins, n] = EntropyInvariant.mutual_information(nor1_, nor2_,
        nbins=BINS[bins], method="histogram")
      mi_histogram2_nor2[nb, bins, n] = EntropyInvariant.mutual_information(nor1_, nor3_,
        nbins=BINS[bins], method="histogram")
      mi_histogram2_nor3[nb, bins, n] = EntropyInvariant.mutual_information(nor2_, nor3_,
        nbins=BINS[bins], method="histogram")
    end
  end
end

[Acknowledgments] Special thanks go to Laurent Nahon for his unwavering support throughout the project.
FT acknowledges a PhD grant from Synchrotron SOLEIL and INRAE.

## References

[1] D. R. Brillinger, "Some data analyses using mutual information," *Brazilian Journal of Probability and Statistics*, pp. 163-182, 2004.

[2] R. E. Steuer, J. Kurths, C. O. Daub, J. Weise, and J. Selbig, "The mutual information: Detecting and evaluating dependencies between variables," *Bioinformatics*, vol. 18 Suppl 2, pp. S231-40, 2002.

[3] A. Kraskov, H. Stögbauer, and P. Grassberger, "Estimating mutual information," *Physical Review E*, vol. 69, p. 066138, June 2004.

[4] B. C. Ross, "Mutual Information between Discrete and Continuous Data Sets," *PLOS ONE*, vol. 9, p. e87357, Feb. 2014.

[5] C. E. Shannon, "A Mathematical Theory of Communication," *Bell System Technical Journal*, vol. 27, no. 3, pp. 379-423, 1948.

[6] D. Lombardi and S. Pant, "Nonparametric k-nearest-neighbor entropy estimator," *Physical Review E*, vol. 93, no. 1, p. 013310, 2016.

[7] R. Moddemeijer, "On estimation of entropy and mutual information of continuous distributions," *Signal Processing*, Mar. 1989.

[8] S. Khan, S. Bandyopadhyay, A. R. Ganguly, S. Saigal, D. J. Erickson III, V. Protopopescu, and G. Ostrouchov, "Relative performance of mutual information estimation methods for quantifying the dependence among short and noisy data," *Physical Review E-Statistical, Nonlinear, and Soft Matter Physics*, vol. 76, no. 2, p. 026209, 2007.

[9] H. Alizadeh Noughabi, "Entropy Estimation Using Numerical Methods," *Annals of Data Science*, vol. 2, pp. 231-241, June 2015.

[10] T. M. Cover and J. A. Thomas, *Elements of Information Theory*. John Wiley & Sons, Nov. 2012.

[11] E. T. Jaynes, "Prior probabilities," *IEEE Transactions on Systems Science and Cybernetics*, vol. 4, no. 3, pp. 227-241, 1968.

[12] D. Nagel, G. Diez, and G. Stock, "Accurate estimation of the normalized mutual information of multidimensional data," *The Journal of Chemical Physics*, vol. 161, p. 054108, 08 2024.
[13] O. Vasicek, "A Test for Normality Based on Sample Entropy," *Journal of the Royal Statistical Society: Series B (Methodological)*, vol. 38, pp. 54–59, Sept. 1976.