Repository: scipy/scipy

Issue:
ENH: `special.rel_entr`: avoid premature overflow
### Describe your issue.

Looks like there's opportunity for improvement in `scipy.special.rel_entr`. Presumably, it takes the ratio `x/y` before taking the logarithm because it's faster than taking the difference of two logs, but this can overflow when the correct result is finite. Perhaps a compromise is to check for the overflow and, in that case, take the time to compute the two logs separately.

This came up gh-20673 as a Hypothesis counterexample rather than a real use case, which is why I consider it an enhancement request.

### Reproducing Code Example

```python
import numpy as np
from scipy import special
x = np.asarray([2., 3., 4.])
y = np.asarray(np.finfo(np.float64).tiny)
special.rel_entr(x, y)  # array([1418.17913143, 2128.48509246,           inf])
x * (np.log(x) - np.log(y))  # array([1418.17913143, 2128.48509246, 2839.13085157])
```


### Error message

```shell
-
```


### SciPy/NumPy/Python version and system information

```shell
main
```



Discussion:
It is clearly desirable to avoid premature overflow.

If x and y are greater than one, taking the log and then subtracting might reduce the accuracy, as the domain of logarithm is “larger” than its image. On the other hand, accuracy is not compromised if the arguments are less than one, which may be typical if rel_entr is applied to two probability values.
@fancidev

> If x and y are greater than one, taking the log and then subtracting might reduce the accuracy, as the domain of logarithm is “larger” than its image. On the other hand, accuracy is not compromised if the arguments are less than one, which may be typical if rel_entr is applied to two probability values.

I checked this idea experimentally, and in general, the division approach has better accuracy than the logarithm approach. With three exceptions, the `x * log(x / y)` approach has equal or better accuracy than the `x * (log(x) - log(y))` approach. 

The three exceptions:

1. Overflow. If x / y == inf, then log subtraction is more accurate. (This is the lower-right white area of the graph below - here error is infinite for the ratio approach.)
2. Underflow. If x / y == 0, then log subtraction is more accurate. (This is the upper-left white area of the graph below - here error is infinite for the ratio approach.)
3. Subnormal numbers. If x / y is a [subnormal number](https://en.wikipedia.org/wiki/Subnormal_number), then log subtraction is more accurate. (This is the diagonal band of yellow squares above the main diagonal. This has finite error, but is still quite inaccurate.)

Other than these three exceptions, the division approach is more accurate.

Here's a comparison of the error of the two methods:

![ratio](https://github.com/scipy/scipy/assets/666402/efcd2fa5-d259-463a-bd4a-416f21407cef)

![logdiff](https://github.com/scipy/scipy/assets/666402/6a2af61d-ea5a-454a-9c32-2bb2fd2e9877)

(Note: Errors have been clipped at 95% to make the graphs readable. Some cells have thousands of ULP of error. See full notebook for details.)



With that in mind, here is the logic I would propose:

```
# assume all of the corner cases have already been handled
ratio = x / y
if ratio == inf or ratio < np.finfo(np.float64).tiny:
    return x * (log(x) - log(y))
else:
    return x * log(ratio)
```

The `ratio == inf` check handles case 1. The `ratio < np.finfo(np.float64).tiny` check handles cases 2 and 3.

If you look only at the area where x < 1 and y < 1, which is the bottom-left quadrant of the graph, then the `x * log(x / y)` approach still has equal or better accuracy than the `x * (log(x) - log(y))` approach.

[Notebook source](https://gist.github.com/nickodell/fd7af4afdf03aaece462c5c638d23d66)
Very interesting analysis @nickodell ! I suspect the "clipped" values with large error could be related to `x` and `y` that are close.

To illustrate this, I tried a contrived example:

```python
import numpy as np
from mpmath import mp
mp.dps = 1000
x = 1234
y = np.nextafter(x, 100)
print(f'{np.log(y/x)=}')
print(f'{np.log(y) - np.log(x)=}')
print(f'{float(mp.log(y)-mp.log(x))=}')
print(f'{float(mp.log(mp.mpf(y)/x))=}')
print(f'{np.log1p((y-x)/x)=}')
```

Output:

```
np.log(y/x)=-2.2204460492503136e-16
np.log(y) - np.log(x)=-8.881784197001252e-16
float(mp.log(y)-mp.log(x))=-1.842574355293615e-16
float(mp.log(mp.mpf(y)/x))=-1.842574355293615e-16
np.log1p((y-x)/x)=-1.842574355293615e-16
```

So for those two numbers that are really close, `log(y/x)` is off by 20%, `log(y)-log(x)` is off by 380%, and `log1p` gives the correct answer.

The contrived example shows that if we insist on the accuracy from a numerical perspective, there's a fair bit of work to do.
That's a really smart idea. Here's another graph of function error for the log1p method. (I used `np.log1p((x - y) / y)` rather than `np.log1p((y-x)/x)`, though.)

![Untitled](https://github.com/scipy/scipy/assets/666402/1eb7a38a-e74a-4650-951a-a07ef436264b)

So I could piece together a more accurate `rel_entr()` from three sources: log1p for the diagonal, where x and y are very close, ratio for everywhere else, and log difference for places where there is under/overflow.

Resolving patch:
diff --git a/scipy/stats/_entropy.py b/scipy/stats/_entropy.py
--- a/scipy/stats/_entropy.py
+++ b/scipy/stats/_entropy.py
@@ -124,7 +124,7 @@ def entropy(pk: np.typing.ArrayLike,
     >>> D = entropy(pk, qk, base=base)
     >>> D
     0.7369655941662062
-    >>> D == np.sum(pk * np.log(pk/qk)) / np.log(base)
+    >>> np.isclose(D, np.sum(pk * np.log(pk/qk)) / np.log(base), rtol=4e-16, atol=0)
     True
 
     The cross entropy can be calculated as the sum of the entropy and


Test patch:
diff --git a/scipy/special/tests/test_basic.py b/scipy/special/tests/test_basic.py
--- a/scipy/special/tests/test_basic.py
+++ b/scipy/special/tests/test_basic.py
@@ -4152,6 +4152,46 @@ def xfunc(x, y):
     assert_func_equal(special.rel_entr, w, z, rtol=1e-13, atol=1e-13)
 
 
+def test_rel_entr_gh_20710_near_zero():
+    # Check accuracy of inputs which are very close
+    inputs = np.array([
+        # x, y
+        (0.9456657713430001, 0.9456657713430094),
+        (0.48066098564791515, 0.48066098564794774),
+        (0.786048657854401, 0.7860486578542367),
+    ])
+    # Known values produced using `x * mpmath.log(x / y)` with dps=30
+    expected = [
+        -9.325873406851269e-15,
+        -3.258504577274724e-14,
+        1.6431300764454033e-13,
+    ]
+    x = inputs[:, 0]
+    y = inputs[:, 1]
+    assert_allclose(special.rel_entr(x, y), expected, rtol=1e-13, atol=0)
+
+
+def test_rel_entr_gh_20710_overflow():
+    inputs = np.array([
+        # x, y
+        # Overflow
+        (4, 2.22e-308),
+        # Underflow
+        (1e-200, 1e+200),
+        # Subnormal
+        (2.22e-308, 1e15),
+    ])
+    # Known values produced using `x * mpmath.log(x / y)` with dps=30
+    expected = [
+        2839.139983229607,
+        -9.210340371976183e-198,
+        -1.6493212008074475e-305,
+    ]
+    x = inputs[:, 0]
+    y = inputs[:, 1]
+    assert_allclose(special.rel_entr(x, y), expected, rtol=1e-13, atol=0)
+
+
 def test_huber():
     assert_equal(special.huber(-1, 1.5), np.inf)
     assert_allclose(special.huber(2, 1.5), 0.5 * np.square(1.5))