Repository: scipy/scipy
Pull request: MAINT: stats: attempt to consolidate warnings and errors
#### Reference issue
Closes gh-13462

#### What does this implement/fix?
This attempts to resolve gh-13462, which reported that there are too many warnings in the `scipy.stats` public API, and that we need a better way to organize them. 

Things got messy because I tried to put all the warnings and errors into the `_result_classes` file and ended up with circular imports that I found difficult to resolve. I think the better way to go for now is to move warnings and errors into their own file.

Let's see if I got the docs to build correctly.


Problem statement:
Too many warnings and results objects in public API for scipy.stats
When working on gh-13461, I noticed these new objects listed on the overview page for `scipy.stats`:

- `BinomTestResult`
- `F_onewayConstantInputWarning`
- `F_onewayBadInputSizesWarning`
- `PearsonRConstantInputWarning`
- `PearsonRNearConstantInputWarning`
- `SpearmanRConstantInputWarning`

That doesn't look good, the warnings are too detailed and this is way too much to add to the public API. I think what should happen here is:

- Remove `BinomTest` from the public API, and don't add any other `*Result` classes. Just don't add them to `__all__` dicts.
- Create one or a few useful warnings, e.g. just `StatisticalTestWarning` as the top-level (or only public) warning, and create warnings like `ConstantInputWarning` that inherit from it.
- Treat these things like other additions to the public API - I cannot remember any discussion about adding them.

Cc @WarrenWeckesser and @mdhaber.

Hints from issue discussion:
I've re-opened gh-13118 about what statistical tests should return, and I'll send a note to the mailing list this time. We may need something for statistical tests results in the public API, though, because it is beginning to look natural for the returned objects to have methods rather than just contain data.

I don't know anything about the rest, but I agree that the list could probably be trimmed to just a superclass warning. 
RE: too many warnings

The spearman warning `SpearmanRConstantInputWarning` I put in PR:
https://github.com/scipy/scipy/pull/11225  

The reasoning for separate warnings we're discussed in Warren's original PR:
https://github.com/scipy/scipy/pull/9562#issuecomment-481457819 
and further down in that pr with comments from several of us. 

Actually in that PR @chrisb83 had suggested a more generic constant input warning :)  

Spearman calculates correlations on ranks so doesn't have the floating point issues that pearson can sometimes encounter. That being said, because of 0 vs non-zero variances impacting lots of statistical calculations-and many of them being on fp numbers-I'd be in favor of two generic warnings: 

(1) Constant input warning 
(2) Near constant intput

The anova calculations will be subject to having these same sorts of issues too but those are fairly recent additions so have less exposure I'd guess.

Also, the Warnings for pearson have been out for 2 or 3 release cycles now, is it safe to consider these defects? (e.g. no deprecation cycle) 
Thanks for digging up that history @rlucas7. 

> That being said, because of 0 vs non-zero variances impacting lots of statistical calculations-and many of them being on fp numbers-I'd be in favor of two generic warnings:
> 
> (1) Constant input warning
> (2) Near constant intput

That makes sense. We can even inherit from those to make the emitted warnings more specific if desired (`SpearmanRConstantInputWarning` is fine), but it does not really make sense to me to put that many warnings in the public API.
> I've re-opened [gh-13118](https://github.com/scipy/scipy/issues/13118) about what statistical tests should return, and I'll send a note to the mailing list this time. We may need something for statistical tests results in the public API, though, because it is beginning to look natural for the returned objects to have methods rather than just contain data.

Okay let's move this part of the discussion to that issue then, and keep this one for the warnings.

Commit message: MAINT: stats: link additional custom warnings

[skip azp] [skip actions]

Patch to review:
diff --git a/scipy/stats/__init__.py b/scipy/stats/__init__.py
index feedc62099e8..dfbdc4067af2 100644
--- a/scipy/stats/__init__.py
+++ b/scipy/stats/__init__.py
@@ -455,15 +455,15 @@
 .. autosummary::
    :toctree: generated/
 
-   F_onewayConstantInputWarning
-   F_onewayBadInputSizesWarning
-   PearsonRConstantInputWarning
-   PearsonRNearConstantInputWarning
-   SpearmanRConstantInputWarning
-   BootstrapDegenerateDistributionWarning
+   DegenerateDataWarning
+   ConstantInputWarning
+   NearConstantInputWarning
+   FitError
 
 """
 
+from ._warnings_errors import (ConstantInputWarning, NearConstantInputWarning,
+                               DegenerateDataWarning, FitError)
 from ._stats_py import *
 from ._variation import variation
 from .distributions import *
@@ -476,8 +476,7 @@
 from ._multivariate import *
 from . import contingency
 from .contingency import chi2_contingency
-from ._resampling import (bootstrap, BootstrapDegenerateDistributionWarning,
-                          monte_carlo_test, permutation_test)
+from ._resampling import (bootstrap, monte_carlo_test, permutation_test)
 from ._entropy import *
 from ._hypotests import *
 from ._rvs_sampling import rvs_ratio_uniforms, NumericalInverseHermite  # noqa
@@ -485,6 +484,7 @@
 from ._mannwhitneyu import mannwhitneyu
 from ._fit import fit
 
+
 # Deprecated namespaces, to be removed in v2.0.0
 from . import (
     biasedurn, kde, morestats, mstats_basic, mstats_extras, mvn, statlib, stats
diff --git a/scipy/stats/_continuous_distns.py b/scipy/stats/_continuous_distns.py
index 28793457feca..f075e194a9f2 100644
--- a/scipy/stats/_continuous_distns.py
+++ b/scipy/stats/_continuous_distns.py
@@ -9,7 +9,6 @@
 import ctypes
 
 import numpy as np
-
 from scipy._lib.doccer import (extend_notes_in_docstring,
                                replace_notes_in_docstring,
                                inherit_docstring_from)
@@ -31,6 +30,7 @@
                          _SQRT_2_OVER_PI, _LOG_SQRT_2_OVER_PI)
 import scipy.stats._boost as _boost
 from scipy.optimize import root_scalar
+from scipy.stats._warnings_errors import FitError
 
 
 def _remove_optimizer_parameters(kwds):
@@ -558,6 +558,7 @@ def _entropy(self):
 
 
 class FitDataError(ValueError):
+    """Raised when input data is inconsistent with fixed parameters."""
     # This exception is raised by, for example, beta_gen.fit when both floc
     # and fscale are fixed and there are values in the data not in the open
     # interval (floc, floc+fscale).
@@ -570,7 +571,10 @@ def __init__(self, distr, lower, upper):
         )
 
 
-class FitSolverError(RuntimeError):
+class FitSolverError(FitError):
+    """
+    Raised when a solver fails to converge while fitting a distribution.
+    """
     # This exception is raised by, for example, beta_gen.fit when
     # optimize.fsolve returns with ier != 1.
     def __init__(self, mesg):
diff --git a/scipy/stats/_distn_infrastructure.py b/scipy/stats/_distn_infrastructure.py
index 57dc3fa43ac8..64e2d108d050 100644
--- a/scipy/stats/_distn_infrastructure.py
+++ b/scipy/stats/_distn_infrastructure.py
@@ -39,6 +39,7 @@
 
 import numpy as np
 from ._constants import _XMAX
+from scipy.stats._warnings_errors import FitError
 
 # These are the docstring parts used for substitution in specific
 # distribution docstrings
@@ -1684,10 +1685,6 @@ def _get_fixed_fit_value(kwds, names):
     return vals[0][1] if vals else None
 
 
-class FitError(RuntimeError):
-    """Represents an error condition when fitting a distribution to data"""
-    pass
-
 #  continuous random variables: implement maybe later
 #
 #  hf  --- Hazard Function (PDF / SF)
@@ -2560,7 +2557,7 @@ def fit(self, data, *args, **kwds):
         ------
         TypeError, ValueError
             If an input is invalid
-        FitError
+        `~scipy.stats.FitError`
             If fitting fails or the fit produced would be invalid
 
         Returns
diff --git a/scipy/stats/_resampling.py b/scipy/stats/_resampling.py
index 55a99d4edc4e..47bcb89b4bc3 100644
--- a/scipy/stats/_resampling.py
+++ b/scipy/stats/_resampling.py
@@ -8,11 +8,12 @@
 from dataclasses import make_dataclass
 from ._common import ConfidenceInterval
 from ._axis_nan_policy import _broadcast_concatenate, _broadcast_arrays
+from ._warnings_errors import DegenerateDataWarning
 
 __all__ = ['bootstrap', 'monte_carlo_test', 'permutation_test']
 
 
-class BootstrapDegenerateDistributionWarning(RuntimeWarning):
+class BootstrapDegenerateDistributionWarning(DegenerateDataWarning):
     """
     Warning generated by `bootstrap` when BCa method is used and
     the bootstrap distribution is degenerate.
@@ -314,6 +315,12 @@ def bootstrap(data, statistic, *, n_resamples=9999, batch=None,
             The bootstrap standard error, that is, the sample standard
             deviation of the bootstrap distribution
 
+    Warns
+    -----
+    `~scipy.stats.DegenerateDataWarning`
+        Generated when ``method='BCa'`` and the bootstrap distribution is
+        degenerate (e.g. all elements are identical).
+
     Notes
     -----
     Elements of the confidence interval may be NaN for ``method='BCa'`` if
diff --git a/scipy/stats/_result_classes.py b/scipy/stats/_result_classes.py
index 46c1eb76be3e..37907374a646 100644
--- a/scipy/stats/_result_classes.py
+++ b/scipy/stats/_result_classes.py
@@ -1,5 +1,6 @@
 # This module exists only to allow Sphinx to generate docs
-# for the result objects returned by some functions in stats.
+# for the result objects returned by some functions in stats
+# _without_ adding them to the main stats documentation page.
 
 """
 Result classes
diff --git a/scipy/stats/_stats_py.py b/scipy/stats/_stats_py.py
index db49bdcadee1..63985edff292 100644
--- a/scipy/stats/_stats_py.py
+++ b/scipy/stats/_stats_py.py
@@ -56,6 +56,7 @@
                                _broadcast_concatenate)
 from ._binomtest import _binary_search_for_binom_tst as _binary_search
 from scipy._lib._bunch import _make_tuple_bunch
+from scipy import stats
 
 
 # Functions/classes in other files should be added in `__init__.py`, not here
@@ -3400,7 +3401,7 @@ def trim_mean(a, proportiontocut, axis=0):
 F_onewayResult = namedtuple('F_onewayResult', ('statistic', 'pvalue'))
 
 
-class F_onewayConstantInputWarning(RuntimeWarning):
+class F_onewayConstantInputWarning(stats.ConstantInputWarning):
     """
     Warning generated by `f_oneway` when an input is constant, e.g.
     each of the samples provided is an array of identical values.
@@ -3413,12 +3414,11 @@ def __init__(self, msg=None):
         self.args = (msg,)
 
 
-class F_onewayBadInputSizesWarning(RuntimeWarning):
+class F_onewayBadInputSizesWarning(stats.DegenerateDataWarning):
     """
     Warning generated by `f_oneway` when an input has length 0,
     or if all the inputs have length 1.
     """
-    pass
 
 
 def _create_f_oneway_nan_result(shape, axis):
@@ -3469,12 +3469,12 @@ def f_oneway(*samples, axis=0):
 
     Warns
     -----
-    F_onewayConstantInputWarning
+    `~scipy.stats.ConstantInputWarning`
         Raised if all values within each of the input arrays are identical.
         In this case the F statistic is either infinite or isn't defined,
         so ``np.inf`` or ``np.nan`` is returned.
 
-    F_onewayBadInputSizesWarning
+    `~scipy.stats.DegenerateDataWarning`
         Raised if the length of any input array is 0, or if all the input
         arrays have length 1.  ``np.nan`` is returned for the F statistic
         and the p-value in these cases.
@@ -3704,7 +3704,7 @@ def alexandergovern(*samples, nan_policy='propagate'):
 
     Warns
     -----
-    AlexanderGovernConstantInputWarning
+    `~scipy.stats.ConstantInputWarning`
         Raised if an input is a constant array.  The statistic is not defined
         in this case, so ``np.nan`` is returned.
 
@@ -3836,7 +3836,7 @@ def _alexandergovern_input_validation(samples, nan_policy):
                                                                  "pvalue"))
 
 
-class AlexanderGovernConstantInputWarning(RuntimeWarning):
+class AlexanderGovernConstantInputWarning(stats.ConstantInputWarning):
     """Warning generated by `alexandergovern` when an input is constant."""
 
     def __init__(self, msg=None):
@@ -3845,7 +3845,7 @@ def __init__(self, msg=None):
         self.args = (msg,)
 
 
-class PearsonRConstantInputWarning(RuntimeWarning):
+class PearsonRConstantInputWarning(stats.ConstantInputWarning):
     """Warning generated by `pearsonr` when an input is constant."""
 
     def __init__(self, msg=None):
@@ -3855,7 +3855,7 @@ def __init__(self, msg=None):
         self.args = (msg,)
 
 
-class PearsonRNearConstantInputWarning(RuntimeWarning):
+class PearsonRNearConstantInputWarning(stats.NearConstantInputWarning):
     """Warning generated by `pearsonr` when an input is nearly constant."""
 
     def __init__(self, msg=None):
@@ -4025,11 +4025,11 @@ def pearsonr(x, y, *, alternative='two-sided'):
 
     Warns
     -----
-    PearsonRConstantInputWarning
+    `~scipy.stats.ConstantInputWarning`
         Raised if an input is a constant array.  The correlation coefficient
         is not defined in this case, so ``np.nan`` is returned.
 
-    PearsonRNearConstantInputWarning
+    `~scipy.stats.NearConstantInputWarning`
         Raised if an input is "nearly" constant.  The array ``x`` is considered
         nearly constant if ``norm(x - mean(x)) < 1e-13 * abs(mean(x))``.
         Numerical errors in the calculation ``x - mean(x)`` in this case might
@@ -4469,7 +4469,7 @@ def pmf(x):
     return oddsratio, pvalue
 
 
-class SpearmanRConstantInputWarning(RuntimeWarning):
+class SpearmanRConstantInputWarning(stats.ConstantInputWarning):
     """Warning generated by `spearmanr` when an input is constant."""
 
     def __init__(self, msg=None):
@@ -4544,6 +4544,12 @@ def spearmanr(a, b=None, axis=0, nan_policy='propagate',
         for alternative hypotheses. `pvalue` has the same
         shape as `correlation`.
 
+    Warns
+    -----
+    `~scipy.stats.ConstantInputWarning`
+        Raised if an input is a constant array.  The correlation coefficient
+        is not defined in this case, so ``np.nan`` is returned.
+
     References
     ----------
     .. [1] Zwillinger, D. and Kokoska, S. (2000). CRC Standard
diff --git a/scipy/stats/_warnings_errors.py b/scipy/stats/_warnings_errors.py
new file mode 100644
index 000000000000..77a2e650b06c
--- /dev/null
+++ b/scipy/stats/_warnings_errors.py
@@ -0,0 +1,38 @@
+# Warnings
+
+
+class DegenerateDataWarning(RuntimeWarning):
+    """Warns when data is degenerate and results may not be reliable."""
+    def __init__(self, msg=None):
+        if msg is None:
+            msg = ("Degenerate data encountered; results may not be reliable.")
+        self.args = (msg,)
+
+
+class ConstantInputWarning(DegenerateDataWarning):
+    """Warns when all values in data are exactly equal."""
+    def __init__(self, msg=None):
+        if msg is None:
+            msg = ("All values in data are exactly equal; "
+                   "results may not be reliable.")
+        self.args = (msg,)
+
+
+class NearConstantInputWarning(DegenerateDataWarning):
+    """Warns when all values in data are nearly equal."""
+    def __init__(self, msg=None):
+        if msg is None:
+            msg = ("All values in data are nearly equal; "
+                   "results may not be reliable.")
+        self.args = (msg,)
+
+
+# Errors
+
+
+class FitError(RuntimeError):
+    """Represents an error condition when fitting a distribution to data."""
+    def __init__(self, msg=None):
+        if msg is None:
+            msg = ("An error occured when fitting a distribution to data.")
+        self.args = (msg,)
diff --git a/scipy/stats/meson.build b/scipy/stats/meson.build
index ee1495bb91a3..9539a26cbd66 100644
--- a/scipy/stats/meson.build
+++ b/scipy/stats/meson.build
@@ -214,6 +214,7 @@ py3.install_sources([
     '_stats_py.py',
     '_tukeylambda_stats.py',
     '_variation.py',
+    '_warnings_errors.py',
     'biasedurn.py',
     'contingency.py',
     'distributions.py',
diff --git a/scipy/stats/tests/test_resampling.py b/scipy/stats/tests/test_resampling.py
index 9b733d774719..74170c8e8883 100644
--- a/scipy/stats/tests/test_resampling.py
+++ b/scipy/stats/tests/test_resampling.py
@@ -1,7 +1,6 @@
 import numpy as np
 import pytest
-from scipy.stats import (bootstrap, BootstrapDegenerateDistributionWarning,
-                         monte_carlo_test, permutation_test)
+from scipy.stats import bootstrap, monte_carlo_test, permutation_test
 from numpy.testing import assert_allclose, assert_equal, suppress_warnings
 from scipy import stats
 from scipy import special
@@ -361,7 +360,7 @@ def test_bootstrap_degenerate(method):
     data = 35 * [10000.]
     if method == "BCa":
         with np.errstate(invalid='ignore'):
-            with pytest.warns(BootstrapDegenerateDistributionWarning):
+            with pytest.warns(_resampling.BootstrapDegenerateDistributionWarning):  # noqa
                 res = bootstrap([data, ], np.mean, method=method)
                 assert_equal(res.confidence_interval, (np.nan, np.nan))
     else:
diff --git a/tools/refguide_check.py b/tools/refguide_check.py
index 26c6782d7cda..e23f7bdf529d 100755
--- a/tools/refguide_check.py
+++ b/tools/refguide_check.py
@@ -150,6 +150,11 @@
     r'scipy\.stats\.reciprocal',  # alias for lognormal
     r'scipy\.stats\.gilbrat',  # alias for gibrat
     r'scipy\.stats\.trapz',   # alias for trapezoid
+    r'scipy\.stats\.F_onewayBadInputSizesWarning',  # shouldn't
+    r'scipy\.stats\.F_onewayConstantInputWarning',  # have
+    r'scipy\.stats\.PearsonRConstantInputWarning',  # been
+    r'scipy\.stats\.PearsonRNearConstantInputWarning',  # in
+    r'scipy\.stats\.SpearmanRConstantInputWarning',  # __all__
 ]
 # deprecated windows in scipy.signal namespace
 for name in ('barthann', 'bartlett', 'blackmanharris', 'blackman', 'bohman',


Code review comments:
diff_hunk: @@ -3836,7 +3836,7 @@ def _alexandergovern_input_validation(samples, nan_policy):
                                                                  "pvalue"))
 
 
-class AlexanderGovernConstantInputWarning(RuntimeWarning):
+class AlexanderGovernConstantInputWarning(stats.ConstantInputWarning):
path: scipy/stats/_stats_py.py
text: @user1:
Maybe simplify it further and emit a ConstantInputWarning, here and elsewhere? 

@author:
I left it that way based on what @user2 wrote [here](https://github.com/scipy/scipy/issues/13462#issuecomment-770392420): leave the existing test-specific warnings, but only make a more general superclass warning public.


@user2:
I wasn't actually advocating for that, what @user3-br says it's better. I just meant that it's acceptable to keep this private if you'd really want function-specific warnings. But to me it doesn't have any value, just code bloat.

@author:
I don't disagree. I just went with what was written there.
I eliminated `AlexanderGovernConstantInputWarning` because it wasn't public.
I eliminated `BootstrapDegenerateDistributionWarning` because I'm comfortable doing so, as it is very recent and probably nobody uses it.
I left the rest alone because they have been public for a while and I don't fully understand the original rationale for making them separate in the first place, given that they considered a single warning for all of them at the time.
Let me know if you want those removed from the `stats` namespace without warning, though. _Update: commit 4b62fb68556315e0a743b3b8ddc49d7e2cfb1709 does this and can be merged into this branch if desired_.

@user2:
> I left the rest alone because they have been public for a while

Fair enough. Perhaps they're in use somewhere. It's not great design, but I agree with the current state of this PR - no need to rip it out without a deprecation. How about leaving it as is and opening an issue to deprecate the rest? Then it can be done later, perhaps as an intro task for a newer contributor.

@author:
OK. For reference, this would entail:
* `warnings.warn(msg, DeprecationWarning)`, where `msg` is something like:

> This function currently emits a subclass of `ConstantInputWarning`, `F_onewayConstantInputWarning`. As of SciPy 1.9.0, `F_onewayConstantInputWarning` is deprecated in favor of its base class and will be removed in SciPy 1.11.0. 

* Adding `assert_warns(DeprecationWarning)` in all the tests that would start to fail
* Adding a notice in the release notes
* Adding a `.. deprecated::` section to the documentation with a message similar to the one above

?

@author:
I'll take care of this to get the deprecation in 1.9.0.