Repository: scipy/scipy (Python)

Issue:
ENH: Add softplus implementation
### Is your feature request related to a problem? Please describe.

It might be nice to have a numerically stable [softplus](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)#Softplus) implementation, ie `np.log1p(np.exp(x))`

### Describe the solution you'd like.

Something based on one of these stackoverflow answers:

- https://cs.stackexchange.com/questions/110798/numerically-stable-log1pexp-calculation
- https://stackoverflow.com/questions/44230635/avoid-overflow-with-softplus-function-in-python


### Describe alternatives you've considered.

I could use `logsumexp` with zeros in the first argument
```
>>> from scipy.special import logsumexp
>>> import numpy as np
>>> logsumexp(np.vstack((np.zeros(10), 10**np.arange(10))), axis=0)
```
but it's a bit cumbersome. Not sure if it's enough of a pain to warrant this enhancement proposal though.

### Additional context (e.g. screenshots, GIFs)

_No response_


Discussion:
Thanks @tommyod. Isn't it `np.log1p(np.exp(x))`?

I am not sure about adding this because then we need to add all of the various activation functions? If there are other use than ML/AI though, why not.
Yes it is. Corrected it above!

I'm not sure if it's a good fit for scipy either. It came up in a different context to me. It's tied to some other functions implemented:
- Its derivative is [scipy.special.expit](https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.expit.html#scipy.special.expit)
- As mentioned above it's equal to `logsumexp([0, x])` if `x` is a scalar.

I'm sure it has other uses, but when I searched github for usage I mostly found ML/AI. That might be due to the popularity of implementing NNs from scratch on github. I think it's a quite natural penalty function in optimization if one desires a smooth penalty in some optimization problems.

Slightly related: https://github.com/scipy/scipy/issues/12490
I am going to say +1 then. You might want to get more opinions by asking the mailing list (and this would be required before inclusion if we wanted to move forward.)
If we are implementing this, can I work on it?
@aadya940 thank you for the proposal. If I recall correctly, this still needs to go through the mailing list. Correct me if I am wrong @tommyod. Either way, @tommyod would have the primmer as he opened the issue. We do not want folks to compete 😃 We have more than enough issues/PRs to work on so we don't have to do that.
I wanted to air the suggestion. I would use a softplus implementation in scipy, and I personally think it's within the scope of the special functions, but unfortunately I don't have capacity to do the implementation right now.

Therefore I won't follow this up through the mailing list, so if @aadya940 wants to do that, then go ahead :) Thanks!
Thank you @tommyod. @aadya940 would you mind sending an email to the mailing list motivating this change? This is the first step for inclusion in SciPy.
> 

Done
Thank you @aadya940, I've replied. Posting here as well: StackOverflow is not compatible with our license and cannot be used/read. Good thing is that the maths is simple enough so we can easily implement it ourselves from scratch.
Hi @tupui , It would be good if you can help me with the implementation considering I don't know much about the scipy architecture. From what I understand is I can define the softplus function in scipy.special.utils._basic and then add it to "___ all __ ", later write a parametrized test checking the function in scipy.special.tests. What am I missing?
Before doing anything, we need to wait a bit for the mailing list. No reply for a week or so would mean OK.

To start with, you can have a look at our contributing documentation: https://scipy.github.io/devdocs/dev/contributor/contributor_toc.html

We also have monthly newcomers meeting and a Slack server. You will find all info here: https://scipy.org/community/
I would also be glad to have it in scipy, in particular because it‘s non-trivial to implement it numerically stable for large input values.

It might be if interest that this was implemented internally in scikit-learn in https://github.com/scikit-learn/scikit-learn/pull/21814. It follows closely, with some deviations, Eq. (10) of https://cran.r-project.org/web/packages/Rmpfr/vignettes/log1mexp-note.pdf.

Hum this resource links to a R package which is GPL-2… I would prefer if we would stay away from the grey area of "n-lines of code should be fine” (EDIT: got mixed up with the mailing list and this refers more to the comment around SO). This is linking to an eq so this is ok.
Scikit-learn is BSD and I implemented it based on the pdf, not the R code!
Yes sorry for the confusion, I edited my reply above. OC referring to maths is no problem and same goes with sharing code between Scikit-learn and SciPy 😃 
FYI: A simple option that works now is to use `numpy.logaddexp` with 0 as the first argument.  For example,

```
In [507]: x = 2.5

In [508]: np.log1p(np.exp(x))  # A naive implementation
Out[508]: 2.5788897342925496

In [509]: np.logaddexp(0, x)
Out[509]: 2.5788897342925496

In [510]: x = 20

In [511]: np.log1p(np.exp(x))
Out[511]: 20.000000002061153

In [512]: np.logaddexp(0, x)
Out[512]: 20.000000002061153

In [513]: x = 1000  # With x this large, `exp(x)` in the naive implementation overflows

In [514]: np.log1p(np.exp(x))
<ipython-input-514-1647fa2e1120>:1: RuntimeWarning: overflow encountered in exp
  np.log1p(np.exp(x))
Out[514]: inf

In [515]: np.logaddexp(0, x)
Out[515]: 1000.0

``` 

`np.logaddexp` is a ufunc, so it is vectorized:

```
In [516]: a = [-500, -100, -10, 0, 1e-6, 2, 10, 25]

In [517]: np.logaddexp(0, a)
Out[517]: 
array([7.12457641e-218, 3.72007598e-044, 4.53988992e-005, 6.93147181e-001,
       6.93147681e-001, 2.12692801e+000, 1.00000454e+001, 2.50000000e+001])
```

-----

> I could use `logsumexp` with zeros in the first argument

In theory, that also works, since `scipy.special.logsumexp` is a generalization of `np.logaddexp`. `np.logaddexp` is simpler for this use, and as a ufunc, probably faster.  Also, while exploring this, I found that `logsumexp` has a bug where it appears to lose precision unnecessarily:

```
In [523]: from scipy.special import logsumexp

In [524]: logsumexp([0, -40.0])
Out[524]: 0.0
```
The correct result of `logsumexp([0, -40.0])` is `4.248354255291589e-18`:
```
In [525]: np.logaddexp(0, -40)
Out[525]: 4.248354255291589e-18
```

I created an issue for the `logsumexp` bug: https://github.com/scipy/scipy/issues/18295

@tupui Can I start working on this feature?
@aadya940 seems like there is an agreement that this would be wanted. Feel free to start working on it then. I suppose we can just take what scikit-learn is doing with giving proper credit.
The scikit-learn implementation of `log1pexp` is a `cdef` Cython function:
https://github.com/scikit-learn/scikit-learn/blob/6ca4970ca28840b61229b8016b60382e3a930915/sklearn/_loss/_loss.pyx.tp#L250-L266
It is not exposed as a function callable from Python.

If we want to add this to SciPy, a simple implementation based on my [comment above](https://github.com/scipy/scipy/issues/17905#issuecomment-1507247170) is a one-line wrapper of `np.logaddexp`.  As a wrapper, it would not be a ufunc, but with `**kwargs` passed through to `np.logaddexp`, we can preserve much of the ufunc goodness:

```
def log1pexp(x, **kwargs):
    return np.logaddexp(0.0, x, **kwargs)
```

E.g.
```
In [31]: def log1pexp(x, **kwargs):
    ...:     return np.logaddexp(0.0, x, **kwargs)
    ...: 

In [32]: log1pexp(2.5)
Out[32]: 2.5788897342925496

In [33]: log1pexp(1000)
Out[33]: 1000.0

In [34]: a = [-500, -100, -10, 0, 1e-6, 2, 10, 25]

In [35]: log1pexp(a)
Out[35]: 
array([7.12457641e-218, 3.72007598e-044, 4.53988992e-005, 6.93147181e-001,
       6.93147681e-001, 2.12692801e+000, 1.00000454e+001, 2.50000000e+001])

In [36]: out = np.empty_like(a)

In [37]: log1pexp(a, out=out)
Out[37]: 
array([7.12457641e-218, 3.72007598e-044, 4.53988992e-005, 6.93147181e-001,
       6.93147681e-001, 2.12692801e+000, 1.00000454e+001, 2.50000000e+001])

In [38]: out
Out[38]: 
array([7.12457641e-218, 3.72007598e-044, 4.53988992e-005, 6.93147181e-001,
       6.93147681e-001, 2.12692801e+000, 1.00000454e+001, 2.50000000e+001])
```

-----

We could implement it as a ufunc (as @rkern suggested in the mailing list thread), but that would not be a good first issue for someone new to SciPy development.  It might be a good issue for an existing contributor who wants to dive deeper into how ufuncs are implemented in `scipy.special`.

Also, the inverse function is really useful when using softplus to map between the real line and the positive real line, and back:
```python
def inverse_softplus(y):
    xp = get_namespace(y)
    return xp.where(y > 80.0,  # noqa: PLR2004
                    y,
                    xp.log(xp.expm1(y)))
```



Patch:
diff --git a/scipy/special/__init__.py b/scipy/special/__init__.py
index 07bcb0439888..07a802246a69 100644
--- a/scipy/special/__init__.py
+++ b/scipy/special/__init__.py
@@ -750,6 +750,7 @@
    spence      -- Spence's function, also known as the dilogarithm.
    zeta        -- Riemann zeta function.
    zetac       -- Riemann zeta function minus 1.
+   softplus    -- Softplus function.
 
 Convenience functions
 ---------------------
diff --git a/scipy/special/_basic.py b/scipy/special/_basic.py
index d10d9e31da33..df7f03118e39 100644
--- a/scipy/special/_basic.py
+++ b/scipy/special/_basic.py
@@ -75,6 +75,7 @@
     'riccati_jn',
     'riccati_yn',
     'sinc',
+    'softplus',
     'stirling2',
     'y0_zeros',
     'y1_zeros',
@@ -3413,3 +3414,36 @@ def zeta(x, q=None, out=None):
         return _ufuncs._riemann_zeta(x, out)
     else:
         return _ufuncs._zeta(x, q, out)
+
+      
+def softplus(x, **kwargs):
+    r"""
+    Compute the softplus function element-wise.
+
+    The softplus function is defined as: ``softplus(x) = log(1 + exp(x))``.
+    It is a smooth approximation of the rectifier function (ReLU).
+
+    Parameters
+    ----------
+    x : array_like
+        Input value.
+    **kwargs
+        For other keyword-only arguments, see the
+        `ufunc docs <https://numpy.org/doc/stable/reference/ufuncs.html>`_.
+
+    Returns
+    -------
+    softplus : ndarray
+        Logarithm of ``exp(0) + exp(x)``.
+
+    Examples
+    --------
+    >>> from scipy import special
+    
+    >>> special.softplus(0)
+    0.6931471805599453
+
+    >>> special.softplus([-1, 0, 1])
+    array([0.31326169, 0.69314718, 1.31326169])
+    """
+    return np.logaddexp(0, x, **kwargs)


Test patch:
diff --git a/scipy/special/tests/test_basic.py b/scipy/special/tests/test_basic.py
index efb34b2c5238..d0525fdf90af 100644
--- a/scipy/special/tests/test_basic.py
+++ b/scipy/special/tests/test_basic.py
@@ -39,6 +39,7 @@
 import scipy.special._ufuncs as cephes
 from scipy.special import ellipe, ellipk, ellipkm1
 from scipy.special import elliprc, elliprd, elliprf, elliprg, elliprj
+from scipy.special import softplus
 from scipy.special import mathieu_odd_coef, mathieu_even_coef, stirling2
 from scipy._lib._util import np_long, np_ulong
 
@@ -3680,6 +3681,47 @@ def test_riccati_yn(self):
         assert_array_almost_equal(C, special.riccati_yn(n, x), 8)
 
 
+class TestSoftplus:
+    def test_softplus(self):
+        # Test cases for the softplus function. Selected based on Eq.(10) of:
+        # Mächler, M. (2012). log1mexp-note.pdf. Rmpfr: R MPFR - Multiple Precision
+        # Floating-Point Reliable. Retrieved from:
+        # https://cran.r-project.org/web/packages/Rmpfr/vignettes/log1mexp-note.pdf
+        # Reference values computed with `mpmath`
+        import numpy as np
+        rng = np.random.default_rng(3298432985245)
+        n = 3
+        a1 = rng.uniform(-100, -37, size=n)
+        a2 = rng.uniform(-37, 18, size=n)
+        a3 = rng.uniform(18, 33.3, size=n)
+        a4 = rng.uniform(33.33, 100, size=n)
+        a = np.stack([a1, a2, a3, a4])
+
+        # from mpmath import mp
+        # mp.dps = 100
+        # @np.vectorize
+        # def softplus(x):
+        #     return float(mp.log(mp.one + mp.exp(x)))
+        # softplus(a).tolist()
+        ref = [[1.692721323272333e-42, 7.42673911145206e-41, 8.504608846033205e-35],
+               [1.8425343736349797, 9.488245799395577e-15, 7.225195764021444e-08],
+               [31.253760266045106, 27.758244090327832, 29.995959179643634],
+               [73.26040086468937, 76.24944728617226, 37.83955519155184]]
+
+        res = softplus(a)
+        assert_allclose(res, ref, rtol=2e-15)
+
+    def test_softplus_with_kwargs(self):
+        x = np.arange(5) - 2
+        out = np.ones(5)
+        ref = out.copy()
+        where = x > 0
+
+        softplus(x, out=out, where=where)
+        ref[where] = softplus(x[where])
+        assert_allclose(out, ref)
+
+
 class TestRound:
     def test_round(self):
         rnd = list(map(int, (special.round(10.1),