Repository: scipy/scipy

Issue:
ENH: help `skewnorm.fit` find global minimum
### Describe your issue.

While the documentation for `scipy.stats.rv_continuous.fit` clearly states

> the returned answer is not guaranteed to be globally optimal; it may only be locally optimal, or the optimization may fail altogether

it is clearly beneficial when `fit` correctly returns the global optimum. In the case of MLE (maximum likelihood estimation) for `skewnorm`, there is usually (always?) a local optimum around `a=0` corresponding to a non-skewed normal distribution which makes it challenging to find the global optimum.

In most cases, this challenge can be overcome by choosing clever initial conditions for the fitting algorithm via MoM (method of moments). However, MoM is technically not possible when the skewness of the data exceeds the maximum theoretical skewness `s_max` of a skewnorm distribution (which is ~0.995, the [skewness of the half-normal distribution](https://en.wikipedia.org/wiki/Half-normal_distribution), which is the limiting case as the skew parameter `a` tends to $\pm \infty$).

In the current implementation of `skewnorm.fit`, when the skewness of the data exceeds `s_max`, the code aborts MoM initialization and calls `rv_continuous.fit` with default values (implicitly `a=0`): https://github.com/scipy/scipy/blob/f990b1d2471748c79bc4260baf8923db0a5248af/scipy/stats/_continuous_distns.py#L8838-L8839

This increases the likelihood that the optimizer will get stuck around this `a=0` non-global optimum.

On the [skewnorm Wikipedia page](https://en.wikipedia.org/wiki/Skew_normal_distribution) it's suggested to instead clip the skew to ±0.99 and proceed with MoM. In my experiments, this leads to a very significant improvement. PR incoming...

### Reproducing Code Example

The correct value for `a` is `-1.609587`. However, in the following example, by perturbing the data by a qualitatively insignificant amount results in the optimizer jumping around:

```python
import numpy as np
from numpy.testing import assert_allclose
from scipy import stats

a_correct, loc_correct, scale_correct = -1.60957, 2.2990, 2.46210

res = []
for n in range(8):
    rvs = np.array([-5, -1, n / 100_000] + 12 * [1] + [5])
    a8, loc8, scale8 = stats.skewnorm.fit(rvs)
    res.append((n, a8, loc8, scale8))
    print(f"n: {n} a: {a8:.6f} loc: {loc8:.6f} scale: {scale8:.6f}")

# n: 0 a: 0.000154 loc: 0.687272 scale: 1.861410
# n: 1 a: 0.000046 loc: 0.687432 scale: 1.861410
# n: 2 a: -1.609587 loc: 2.299081 scale: 2.462081
# n: 3 a: -1.609609 loc: 2.299075 scale: 2.462130
# n: 4 a: -1.609553 loc: 2.299062 scale: 2.462120
# n: 5 a: 0.000253 loc: 0.687127 scale: 1.861408
# n: 6 a: -0.402228 loc: 1.284761 scale: 1.861428
# n: 7 a: -0.966153 loc: 1.726831 scale: 1.904278

for n, a8, loc8, scale8 in res:
    assert_allclose(
        (a8, loc8, scale8), (a_correct, loc_correct, scale_correct), atol=1e-4
    )
```


### Error message

```shell
AssertionError                            Traceback (most recent call last)
Cell In[1], line 24
     14 # n: 0 a: 0.000154 loc: 0.687272 scale: 1.861410
     15 # n: 1 a: 0.000046 loc: 0.687432 scale: 1.861410
     16 # n: 2 a: -1.609587 loc: 2.299081 scale: 2.462081
   (...)
     20 # n: 6 a: -0.402228 loc: 1.284761 scale: 1.861428
     21 # n: 7 a: -0.966153 loc: 1.726831 scale: 1.904278
     23 for n, a8, loc8, scale8 in res:
---> 24     assert_allclose(
     25         (a8, loc8, scale8), (a_correct, loc_correct, scale_correct), atol=1e-4
     26     )

    [... skipping hidden 1 frame]

File ~/micromamba/envs/scipy-dev/lib/python3.11/contextlib.py:81, in ContextDecorator.__call__.<locals>.inner(*args, **kwds)
     78 @wraps(func)
     79 def inner(*args, **kwds):
     80     with self._recreate_cm():
---> 81         return func(*args, **kwds)

File ~/micromamba/envs/scipy-dev/lib/python3.11/site-packages/numpy/testing/_private/utils.py:797, in assert_array_compare(comparison, x, y, err_msg, verbose, header, precision, equal_nan, equal_inf, strict)
    793         err_msg += '\n' + '\n'.join(remarks)
    794         msg = build_err_msg([ox, oy], err_msg,
    795                             verbose=verbose, header=header,
    796                             names=('x', 'y'), precision=precision)
--> 797         raise AssertionError(msg)
    798 except ValueError:
    799     import traceback

AssertionError: 
Not equal to tolerance rtol=1e-07, atol=0.0001

Mismatched elements: 3 / 3 (100%)
Max absolute difference: 1.61172781
Max relative difference: 1.00009542
 x: array([1.535906e-04, 6.872722e-01, 1.861410e+00])
 y: array([-1.60957,  2.299  ,  2.4621 ])
```


### SciPy/NumPy/Python version and system information

```shell
1.12.0.dev0+1797.e2d3aee 1.26.0 sys.version_info(major=3, minor=11, micro=5, releaselevel='final', serial=0)
Build Dependencies:
  blas:
    detection method: pkgconfig
    found: true
    include directory: ~/micromamba/envs/scipy-dev/include
    lib directory: ~/micromamba/envs/scipy-dev/lib
    name: openblas
    openblas configuration: USE_64BITINT=0 DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS=
      NO_LAPACK=0 NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP=0 PRESCOTT MAX_THREADS=128
    pc file directory: ~/micromamba/envs/scipy-dev/lib/pkgconfig
    version: 0.3.24
  lapack:
    detection method: pkgconfig
    found: true
    include directory: ~/micromamba/envs/scipy-dev/include
    lib directory: ~/micromamba/envs/scipy-dev/lib
    name: openblas
    openblas configuration: USE_64BITINT=0 DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS=
      NO_LAPACK=0 NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP=0 PRESCOTT MAX_THREADS=128
    pc file directory: ~/micromamba/envs/scipy-dev/lib/pkgconfig
    version: 0.3.24
  pybind11:
    detection method: pkgconfig
    include directory: ~/micromamba/envs/scipy-dev/include
    name: pybind11
    version: 2.11.1
Compilers:
  c:
    args: -march=nocona, -mtune=haswell, -ftree-vectorize, -fPIC, -fstack-protector-strong,
      -fno-plt, -O2, -ffunction-sections, -pipe, -isystem, ~/micromamba/envs/scipy-dev/include,
      -DNDEBUG, -D_FORTIFY_SOURCE=2, -O2, -isystem, ~/micromamba/envs/scipy-dev/include
    commands: ~/micromamba/envs/scipy-dev/bin/x86_64-conda-linux-gnu-cc
    linker: ld.bfd
    linker args: -Wl,-O2, -Wl,--sort-common, -Wl,--as-needed, -Wl,-z,relro, -Wl,-z,now,
      -Wl,--disable-new-dtags, -Wl,--gc-sections, -Wl,--allow-shlib-undefined, -Wl,-rpath,~/micromamba/envs/scipy-dev/lib,
      -Wl,-rpath-link,~/micromamba/envs/scipy-dev/lib, -L~/micromamba/envs/scipy-dev/lib,
      -march=nocona, -mtune=haswell, -ftree-vectorize, -fPIC, -fstack-protector-strong,
      -fno-plt, -O2, -ffunction-sections, -pipe, -isystem, ~/micromamba/envs/scipy-dev/include,
      -DNDEBUG, -D_FORTIFY_SOURCE=2, -O2, -isystem, ~/micromamba/envs/scipy-dev/include
    name: gcc
    version: 12.3.0
  c++:
    args: -fvisibility-inlines-hidden, -fmessage-length=0, -march=nocona, -mtune=haswell,
      -ftree-vectorize, -fPIC, -fstack-protector-strong, -fno-plt, -O2, -ffunction-sections,
      -pipe, -isystem, ~/micromamba/envs/scipy-dev/include, -DNDEBUG, -D_FORTIFY_SOURCE=2,
      -O2, -isystem, ~/micromamba/envs/scipy-dev/include
    commands: ~/micromamba/envs/scipy-dev/bin/x86_64-conda-linux-gnu-c++
    linker: ld.bfd
    linker args: -Wl,-O2, -Wl,--sort-common, -Wl,--as-needed, -Wl,-z,relro, -Wl,-z,now,
      -Wl,--disable-new-dtags, -Wl,--gc-sections, -Wl,--allow-shlib-undefined, -Wl,-rpath,~/micromamba/envs/scipy-dev/lib,
      -Wl,-rpath-link,~/micromamba/envs/scipy-dev/lib, -L~/micromamba/envs/scipy-dev/lib,
      -fvisibility-inlines-hidden, -fmessage-length=0, -march=nocona, -mtune=haswell,
      -ftree-vectorize, -fPIC, -fstack-protector-strong, -fno-plt, -O2, -ffunction-sections,
      -pipe, -isystem, ~/micromamba/envs/scipy-dev/include, -DNDEBUG, -D_FORTIFY_SOURCE=2,
      -O2, -isystem, ~/micromamba/envs/scipy-dev/include
    name: gcc
    version: 12.3.0
  cython:
    commands: cython
    linker: cython
    name: cython
    version: 3.0.2
  fortran:
    args: -march=nocona, -mtune=haswell, -ftree-vectorize, -fPIC, -fstack-protector-strong,
      -fno-plt, -O2, -ffunction-sections, -pipe, -isystem, ~/micromamba/envs/scipy-dev/include
    commands: ~/micromamba/envs/scipy-dev/bin/x86_64-conda-linux-gnu-gfortran
    linker: ld.bfd
    linker args: -Wl,-O2, -Wl,--sort-common, -Wl,--as-needed, -Wl,-z,relro, -Wl,-z,now,
      -Wl,--disable-new-dtags, -Wl,--gc-sections, -Wl,--allow-shlib-undefined, -Wl,-rpath,~/micromamba/envs/scipy-dev/lib,
      -Wl,-rpath-link,~/micromamba/envs/scipy-dev/lib, -L~/micromamba/envs/scipy-dev/lib,
      -march=nocona, -mtune=haswell, -ftree-vectorize, -fPIC, -fstack-protector-strong,
      -fno-plt, -O2, -ffunction-sections, -pipe, -isystem, ~/micromamba/envs/scipy-dev/include
    name: gcc
    version: 12.3.0
  pythran:
    include directory: ../../../micromamba/envs/scipy-dev/lib/python3.11/site-packages/pythran
    version: 0.14.0
Machine Information:
  build:
    cpu: x86_64
    endian: little
    family: x86_64
    system: linux
  cross-compiled: false
  host:
    cpu: x86_64
    endian: little
    family: x86_64
    system: linux
Python Information:
  path: ~/micromamba/envs/scipy-dev/bin/python3.11
  version: '3.11'
```



Discussion:

Resolving patch:
diff --git a/scipy/stats/_continuous_distns.py b/scipy/stats/_continuous_distns.py
--- a/scipy/stats/_continuous_distns.py
+++ b/scipy/stats/_continuous_distns.py
@@ -9221,6 +9221,8 @@ def _munp(self, order, a):
         shape parameter ``a`` will be infinite.
         \n\n""")
     def fit(self, data, *args, **kwds):
+        if kwds.pop("superfit", False):
+            return super().fit(data, *args, **kwds)
         if isinstance(data, CensoredData):
             if data.num_censored() == 0:
                 data = data._uncensor()
@@ -9244,13 +9246,6 @@ def d_skew(skew):  # delta in terms of skewness
                 np.pi/2 * s_23 / (s_23 + ((4 - np.pi)/2)**(2/3))
             )
 
-        # If skewness of data is greater than max possible population skewness,
-        # MoM won't provide a good guess. Get out early.
-        s = stats.skew(data)
-        s_max = skew_d(1)
-        if abs(s) >= s_max and method != "mm" and fa is None and not args:
-            return super().fit(data, *args, **kwds)
-
         # If method is method of moments, we don't need the user's guesses.
         # Otherwise, extract the guesses from args and kwds.
         if method == "mm":
@@ -9263,7 +9258,14 @@ def d_skew(skew):  # delta in terms of skewness
         if fa is None and a is None:  # not fixed and no guess: use MoM
             # Solve for a that matches sample distribution skewness to sample
             # skewness.
-            s = np.clip(s, -s_max, s_max)
+            s = stats.skew(data)
+            if method == 'mle':
+                # For MLE initial conditions, clip skewness to a large but
+                # reasonable value in case the data skewness is out-of-range.
+                s = np.clip(s, -0.99, 0.99)
+            else:
+                s_max = skew_d(1)
+                s = np.clip(s, -s_max, s_max)
             d = d_skew(s)
             with np.errstate(divide='ignore'):
                 a = np.sqrt(np.divide(d**2, (1-d**2)))*np.sign(s)


Test patch:
diff --git a/scipy/stats/tests/test_distributions.py b/scipy/stats/tests/test_distributions.py
--- a/scipy/stats/tests/test_distributions.py
+++ b/scipy/stats/tests/test_distributions.py
@@ -33,7 +33,7 @@
 from scipy.stats._distr_params import distcont, invdistcont
 from .test_discrete_basic import distdiscrete, invdistdiscrete
 from scipy.stats._continuous_distns import FitDataError, _argus_phi
-from scipy.optimize import root, fmin
+from scipy.optimize import root, fmin, differential_evolution
 from itertools import product
 
 # python -OO strips docstrings
@@ -3867,6 +3867,33 @@ def test_fit(self):
         a7m, loc7m, scale7m = stats.skewnorm.fit(-rvs, method='mm')
         assert_allclose([a7m, loc7m, scale7m], [-a7p, -loc7p, scale7p])
 
+    def test_fit_gh19332(self):
+        # When the skewness of the data was high, `skewnorm.fit` fell back on
+        # generic `fit` behavior with a bad guess of the skewness parameter.
+        # Test that this is improved; `skewnorm.fit` is now better at finding
+        # the global optimum when the sample is highly skewed. See gh-19332.
+        x = np.array([-5, -1, 1 / 100_000] + 12 * [1] + [5])
+
+        params = stats.skewnorm.fit(x)
+        res = stats.skewnorm.nnlf(params, x)
+
+        # Compare overridden fit against generic fit.
+        # res should be about 32.01, and generic fit is worse at 32.64.
+        # In case the generic fit improves, remove this assertion (see gh-19333).
+        params_super = stats.skewnorm.fit(x, superfit=True)
+        ref = stats.skewnorm.nnlf(params_super, x)
+        assert res < ref - 0.5
+
+        # Compare overridden fit against stats.fit
+        rng = np.random.default_rng(9842356982345693637)
+        bounds = {'a': (-5, 5), 'loc': (-10, 10), 'scale': (1e-16, 10)}
+        def optimizer(fun, bounds):
+            return differential_evolution(fun, bounds, seed=rng)
+
+        fit_result = stats.fit(stats.skewnorm, x, bounds, optimizer=optimizer)
+        np.testing.assert_allclose(params, fit_result.params, rtol=1e-4)
+
+
 class TestExpon:
     def test_zero(self):
         assert_equal(stats.expon.pdf(0), 1)