Repository: statsmodels/statsmodels

Issue:
MAINT: RuntimeWarning: invalid value encountered in _nct_sf
**update**
resolution for 0.14 is to replace scipy.stats by scipy.special functions to avoid new boost versions.


#### Describe the bug

Recently I started to get a `RuntimeWarning` where I did not get one before:

``` python
import statsmodels.api as sm
analysis = sm.stats.TTestIndPower()
analysis.solve_power(nobs1=None, effect_size=0.978343, alpha=0.05, power=0.9)
/.../site-packages/scipy/stats/_continuous_distns.py:6832: RuntimeWarning: invalid value encountered in _nct_sf
  return np.clip(_boost._nct_sf(x, df, nc), 0, 1)
/.../site-packages/scipy/stats/_continuous_distns.py:6826: RuntimeWarning: invalid value encountered in _nct_cdf
  return np.clip(_boost._nct_cdf(x, df, nc), 0, 1)
22.959830247088764
``` 

#### Expected Output

Just the number 22.9...

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>


INSTALLED VERSIONS
------------------
Python: 3.10.6.final.0
OS: Linux 6.0.6-76060006-generic #202210290932~1669062050~22.04~d94609a SMP PREEMPT_DYNAMIC Mon N x86_64
byteorder: little
LC_ALL: en_US.UTF-8
LANG: C

statsmodels
===========

Installed: 0.13.5 (/.../site-packages/statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.24.1 (/.../site-packages/numpy)
scipy: 1.10.0 (/.../site-packages/scipy)
pandas: 1.5.3 (/.../site-packages/pandas)
    dateutil: 2.8.2 (/.../site-packages/dateutil)
patsy: 0.5.3 (/.../site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.6.3 (/.../site-packages/matplotlib)
    backend: QtAgg 
cvxopt: Not installed
joblib: 1.2.0 (/.../site-packages/joblib)

Developer Tools
================

IPython: 8.8.0 (/.../site-packages/IPython)
    jinja2: 3.1.2 (/.../site-packages/jinja2)
sphinx: Not installed
    pygments: 2.14.0 (/.../site-packages/pygments)
pytest: 7.2.1 (/.../site-packages/pytest)
virtualenv: 20.17.1 (/.../site-packages/virtualenv)


</details>



Discussion:
scipy is moving many distributions from using the scipy special code to using boost.
Some behavior is different and boost also still has some bugs in edge cases that are getting fixed over time.
Also, I guess that warnings that are just an implementation detail should be removed or silenced over time in scipy or boost.

For the specific case:
This might occur during rootfinding if we have not imposed enough constraints to restrict values to valid ranges.
You can reverse or roundtrip the results to check that the results are consistent

AFAIR, this should work:
`analysis.power(nobs1=22.959830247088764, effect_size=0.978343, alpha=0.05)`

The scipy version that I currently use is too old for me to check those boost cases.
e.g. What I usually do is use a unit test, convert warnings to raise an error and look at it with `pytest --pdb`. This is usually a quick way to find the parameters that raise a warning. Sometimes those parameters are edge cases that need special handling.
In this case, scipy already clips the sf, cdf values to be within the [0, 1] range, so I guess statsmodels will not need specific code to treat this.
That's what I get

```
import statsmodels.api as sm
analysis = sm.stats.TTestIndPower()
analysis.solve_power(nobs1=None, effect_size=0.978343, alpha=0.05, power=0.9)
22.959831488068428

analysis.power(nobs1=22.959831488068428, effect_size=0.978343, alpha=0.05)
0.9000000004714136

import scipy
scipy.__version__
'1.7.3'
```
Right, I know the results are correct, thank you.
The question is more along the lines "this is a regression - how do I deal with it?"
You explained that the regression is due to `scipy` moving to use `boost`, thank you.
Now, who should change the code to avoid this warning?
I would argue that your should change `statsmodels` rather than me (and countless others) changing user code.
Or maybe I misunderstood you?
Maybe it is `scipy` who should change?
Thank you for such a speedy reply!

scipy needs to change if it's a numerical problem in boost. (In this case the problem will go away in one or a few scipy versions.)

statsmodels needs to change if we evaluate at invalid parameter values.

Currently I do not know which it is.
I just saw https://github.com/scipy/scipy/issues/14901

I don't know if that is the same as the warning here.

Resolving patch:
diff --git a/statsmodels/stats/oneway.py b/statsmodels/stats/oneway.py
--- a/statsmodels/stats/oneway.py
+++ b/statsmodels/stats/oneway.py
@@ -11,6 +11,9 @@
 from scipy import stats
 from scipy.special import ncfdtrinc
 
+# functions that use scipy.special instead of boost based function in stats
+from statsmodels.stats.power import ncf_cdf, ncf_ppf
+
 from statsmodels.stats.robust_compare import TrimmedMean, scale_transform
 from statsmodels.tools.testing import Holder
 from statsmodels.stats.base import HolderTuple
@@ -774,7 +777,7 @@ def equivalence_oneway_generic(f_stat, n_groups, nobs, equiv_margin, df,
         type_effectsize = "Cohen's f_squared"
     else:
         raise ValueError('`margin_type` should be "f2" or "wellek"')
-    crit_f = stats.ncf.ppf(alpha, df[0], df[1], nc_null)
+    crit_f = ncf_ppf(alpha, df[0], df[1], nc_null)
 
     if margin_type == "wellek":
         # TODO: do we need a sqrt
@@ -784,8 +787,8 @@ def equivalence_oneway_generic(f_stat, n_groups, nobs, equiv_margin, df,
 
     reject = (es < crit_es)
 
-    pv = stats.ncf.cdf(f_stat, df[0], df[1], nc_null)
-    pwr = stats.ncf.cdf(crit_f, df[0], df[1], 1e-13)  # scipy, cannot be 0
+    pv = ncf_cdf(f_stat, df[0], df[1], nc_null)
+    pwr = ncf_cdf(crit_f, df[0], df[1], 1e-13)  # scipy, cannot be 0
     res = HolderTuple(statistic=f_stat,
                       pvalue=pv,
                       effectsize=es,  # match es type to margin_type
@@ -916,7 +919,7 @@ def _power_equivalence_oneway_emp(f_stat, n_groups, nobs, eps, df, alpha=0.05):
     nobs_mean = nobs.sum() / n_groups
     fn = f_stat  # post-hoc power, empirical power at estimate
     esn = fn * (n_groups - 1) / nobs_mean  # Wellek psi
-    pow_ = stats.ncf.cdf(res.crit_f, df[0], df[1], nobs_mean * esn)
+    pow_ = ncf_cdf(res.crit_f, df[0], df[1], nobs_mean * esn)
 
     return pow_
 
@@ -980,8 +983,8 @@ def power_equivalence_oneway(f2_alt, equiv_margin, nobs_t, n_groups=None,
     else:
         raise ValueError('`margin_type` should be "f2" or "wellek"')
 
-    crit_f_margin = stats.ncf.ppf(alpha, df[0], df[1], nobs_t * f2_null)
-    pwr_alt = stats.ncf.cdf(crit_f_margin, df[0], df[1], nobs_t * f2_alt)
+    crit_f_margin = ncf_ppf(alpha, df[0], df[1], nobs_t * f2_null)
+    pwr_alt = ncf_cdf(crit_f_margin, df[0], df[1], nobs_t * f2_alt)
     return pwr_alt
 
 
diff --git a/statsmodels/stats/power.py b/statsmodels/stats/power.py
--- a/statsmodels/stats/power.py
+++ b/statsmodels/stats/power.py
@@ -33,9 +33,30 @@
 import warnings
 
 import numpy as np
-from scipy import stats, optimize
+from scipy import stats, optimize, special
 from statsmodels.tools.rootfinding import brentq_expanding
 
+
+def nct_cdf(x, df, nc):
+    return special.nctdtr(df, nc, x)
+
+
+def nct_sf(x, df, nc):
+    return 1 - special.nctdtr(df, nc, x)
+
+
+def ncf_cdf(x, dfn, dfd, nc):
+    return special.ncfdtr(dfn, dfd, nc, x)
+
+
+def ncf_sf(x, dfn, dfd, nc):
+    return 1 - special.ncfdtr(dfn, dfd, nc, x)
+
+
+def ncf_ppf(q, dfn, dfd, nc):
+    return special.ncfdtri(dfn, dfd, nc, q)
+
+
 def ttest_power(effect_size, nobs, alpha, df=None, alternative='two-sided'):
     '''Calculate power of a ttest
     '''
@@ -60,16 +81,20 @@ def ttest_power(effect_size, nobs, alpha, df=None, alternative='two-sided'):
             # avoid endless loop, https://github.com/scipy/scipy/issues/2667
             pow_ = np.nan
         else:
-            pow_ = stats.nct._sf(crit_upp, df, d*np.sqrt(nobs))
+            # pow_ = stats.nct._sf(crit_upp, df, d*np.sqrt(nobs))
+            # use scipy.special
+            pow_ = nct_sf(crit_upp, df, d*np.sqrt(nobs))
     if alternative in ['two-sided', '2s', 'smaller']:
         crit_low = stats.t.ppf(alpha_, df)
         #print crit_low, df, d*np.sqrt(nobs)
         if np.any(np.isnan(crit_low)):
             pow_ = np.nan
         else:
-            pow_ += stats.nct._cdf(crit_low, df, d*np.sqrt(nobs))
+            # pow_ += stats.nct._cdf(crit_low, df, d*np.sqrt(nobs))
+            pow_ += nct_cdf(crit_low, df, d*np.sqrt(nobs))
     return pow_
 
+
 def normal_power(effect_size, nobs, alpha, alternative='two-sided', sigma=1.):
     """Calculate power of a normal distributed test statistic
 
@@ -227,11 +252,11 @@ def ftest_anova_power(effect_size, nobs, alpha, k_groups=2, df=None):
 
     should be general nobs observations, k_groups restrictions ???
     '''
-    df_num = nobs - k_groups
-    df_denom = k_groups - 1
-    crit = stats.f.isf(alpha, df_denom, df_num)
-    pow_ = stats.ncf.sf(crit, df_denom, df_num, effect_size**2 * nobs)
-    return pow_#, crit
+    df_num = k_groups - 1
+    df_denom = nobs - k_groups
+    crit = stats.f.isf(alpha, df_num, df_denom)
+    pow_ = ncf_sf(crit, df_num, df_denom, effect_size**2 * nobs)
+    return pow_
 
 
 def ftest_power(effect_size, df2, df1, alpha, ncc=1):
@@ -240,8 +265,7 @@ def ftest_power(effect_size, df2, df1, alpha, ncc=1):
     Parameters
     ----------
     effect_size : float
-        standardized effect size, mean divided by the standard deviation.
-        effect size has to be positive.
+        The effect size is here Cohen's ``f``, the square root of ``f2``.
     df2 : int or float
         Denominator degrees of freedom.
         This corresponds to the df_resid in Wald tests.
@@ -282,10 +306,73 @@ def ftest_power(effect_size, df2, df1, alpha, ncc=1):
     df_num, df_denom = df1, df2
     nc = effect_size**2 * (df_denom + df_num + ncc)
     crit = stats.f.isf(alpha, df_num, df_denom)
-    pow_ = stats.ncf.sf(crit, df_num, df_denom, nc)
+    # pow_ = stats.ncf.sf(crit, df_num, df_denom, nc)
+    # use scipy.special for ncf
+    pow_ = ncf_sf(crit, df_num, df_denom, nc)
     return pow_ #, crit, nc
 
 
+def ftest_power_f2(effect_size, df_num, df_denom, alpha, ncc=1):
+    '''Calculate the power of a F-test.
+
+    Based on Cohen's `f^2` effect size.
+
+    This assumes
+
+        df_num : numerator degrees of freedom, (number of constraints)
+        df_denom : denominator degrees of freedom (df_resid in regression)
+        nobs = df_denom + df_num + ncc
+        nc = effect_size * nobs  (noncentrality index)
+
+    Power is computed one-sided in the upper tail.
+
+    Parameters
+    ----------
+    effect_size : float
+        Cohen's f2 effect size or noncentrality divided by nobs.
+    df_num : int or float
+        Numerator degrees of freedom.
+        This corresponds to the number of constraints in Wald tests.
+    df_denom : int or float
+        Denominator degrees of freedom.
+        This corresponds to the df_resid in Wald tests.
+    alpha : float in interval (0,1)
+        significance level, e.g. 0.05, is the probability of a type I
+        error, that is wrong rejections if the Null Hypothesis is true.
+    ncc : int
+        degrees of freedom correction for non-centrality parameter.
+        see Notes
+
+    Returns
+    -------
+    power : float
+        Power of the test, e.g. 0.8, is one minus the probability of a
+        type II error. Power is the probability that the test correctly
+        rejects the Null Hypothesis if the Alternative Hypothesis is true.
+
+    Notes
+
+    The sample size is given implicitly by ``df_denom`` with fixed number of
+    constraints given by numerator degrees of freedom ``df_num``:
+
+        nobs = df_denom + df_num + ncc
+
+    Set ncc=0 to match t-test, or f-test in LikelihoodModelResults.
+    ncc=1 matches the non-centrality parameter in R::pwr::pwr.f2.test
+
+    ftest_power with ncc=0 should also be correct for f_test in regression
+    models, with df_num (df1) as number of constraints and d_denom (df2) as
+    df_resid.
+    '''
+
+    nc = effect_size * (df_denom + df_num + ncc)
+    crit = stats.f.isf(alpha, df_num, df_denom)
+    # pow_ = stats.ncf.sf(crit, df_num, df_denom, nc)
+    # use scipy.special for ncf
+    pow_ = ncf_sf(crit, df_num, df_denom, nc)
+    return pow_
+
+
 #class based implementation
 #--------------------------
 
@@ -881,10 +968,17 @@ def solve_power(self, effect_size=None, nobs1=None, alpha=None, power=None,
 class FTestPower(Power):
     """Statistical Power calculations for generic F-test of a constraint
 
+    This class is not recommended, use `FTestPowerF2` with corrected interface.
+
     This is based on Cohen's f as effect size measure.
 
     Warning: Methods in this class have the names df_num and df_denom reversed.
 
+    See Also
+    --------
+    FTestPowerF2 :
+        Class with Cohen's f-squared as effect size, corrected keyword names.
+
     Examples
     --------
     Sample size and power for multiple regression base on R-squared
@@ -1020,7 +1114,7 @@ def solve_power(self, effect_size=None, df_num=None, df_denom=None,
 
         Notes
         -----
-        The function uses scipy.optimize for finding the value that satisfies
+        The method uses scipy.optimize for finding the value that satisfies
         the power equation. It first uses ``brentq`` with a prior search for
         bounds. If this fails to find a root, ``fsolve`` is used. If ``fsolve``
         also fails, then, for ``alpha``, ``power`` and ``effect_size``,
@@ -1040,6 +1134,151 @@ def solve_power(self, effect_size=None, df_num=None, df_denom=None,
                                                       power=power,
                                                       ncc=ncc)
 
+
+
+class FTestPowerF2(Power):
+    """Statistical Power calculations for generic F-test of a constraint
+
+    This is based on Cohen's f^2 as effect size measure.
+
+    Examples
+    --------
+    Sample size and power for multiple regression base on R-squared
+
+    Compute effect size from R-squared
+
+    >>> r2 = 0.1
+    >>> f2 = r2 / (1 - r2)
+    >>> f = np.sqrt(f2)
+    >>> r2, f2, f
+    (0.1, 0.11111111111111112, 0.33333333333333337)
+
+    Find sample size by solving for denominator degrees of freedom.
+
+    >>> df1 = 1  # number of constraints in hypothesis test
+    >>> df2 = FTestPowerF2().solve_power(effect_size=f2, alpha=0.1, power=0.9,
+                                         df_num=df1)
+    >>> ncc = 1  # default
+    >>> nobs = df2 + df1 + ncc
+    >>> df2, nobs
+    (76.46459758305376, 78.46459758305376)
+
+    verify power at df2
+
+    >>> FTestPowerF2().power(effect_size=f, alpha=0.1, df_num=df1, df_denom=df2)
+    0.8999999972109698
+
+    """
+
+    def power(self, effect_size, df_num, df_denom, alpha, ncc=1):
+        '''Calculate the power of a F-test.
+
+        The effect size is Cohen's ``f^2``.
+
+        The sample size is given by ``nobs = df_denom + df_num + ncc``
+
+        Parameters
+        ----------
+        effect_size : float
+            The effect size is here Cohen's ``f2``. This is equal to
+            the noncentrality of an F-test divided by nobs.
+        df_num : int or float
+            Numerator degrees of freedom,
+            This corresponds to the number of constraints in Wald tests.
+        df_denom : int or float
+            Denominator degrees of freedom.
+            This corresponds to the df_resid in Wald tests.
+        alpha : float in interval (0,1)
+            Significance level, e.g. 0.05, is the probability of a type I
+            error, that is wrong rejections if the Null Hypothesis is true.
+        ncc : int
+            Degrees of freedom correction for non-centrality parameter.
+            see Notes
+
+        Returns
+        -------
+        power : float
+            Power of the test, e.g. 0.8, is one minus the probability of a
+            type II error. Power is the probability that the test correctly
+            rejects the Null Hypothesis if the Alternative Hypothesis is true.
+
+        Notes
+        -----
+        The sample size is given implicitly by df_denom
+
+        set ncc=0 to match t-test, or f-test in LikelihoodModelResults.
+        ncc=1 matches the non-centrality parameter in R::pwr::pwr.f2.test
+
+        ftest_power with ncc=0 should also be correct for f_test in regression
+        models, with df_num and d_denom as defined there. (not verified yet)
+        '''
+
+        pow_ = ftest_power_f2(effect_size, df_num, df_denom, alpha, ncc=ncc)
+        return pow_
+
+    #method is only added to have explicit keywords and docstring
+    def solve_power(self, effect_size=None, df_num=None, df_denom=None,
+                    alpha=None, power=None, ncc=1):
+        '''Solve for any one parameter of the power of a F-test
+
+        for the one sample F-test the keywords are:
+            effect_size, df_num, df_denom, alpha, power
+
+        Exactly one needs to be ``None``, all others need numeric values.
+
+        The effect size is Cohen's ``f2``.
+
+        The sample size is given by ``nobs = df_denom + df_num + ncc``, and
+        can be found by solving for df_denom.
+
+        Parameters
+        ----------
+        effect_size : float
+            The effect size is here Cohen's ``f2``. This is equal to
+            the noncentrality of an F-test divided by nobs.
+        df_num : int or float
+            Numerator degrees of freedom,
+            This corresponds to the number of constraints in Wald tests.
+        df_denom : int or float
+            Denominator degrees of freedom.
+            This corresponds to the df_resid in Wald tests.
+        alpha : float in interval (0,1)
+            significance level, e.g. 0.05, is the probability of a type I
+            error, that is wrong rejections if the Null Hypothesis is true.
+        power : float in interval (0,1)
+            power of the test, e.g. 0.8, is one minus the probability of a
+            type II error. Power is the probability that the test correctly
+            rejects the Null Hypothesis if the Alternative Hypothesis is true.
+        ncc : int
+            degrees of freedom correction for non-centrality parameter.
+            see Notes
+
+        Returns
+        -------
+        value : float
+            The value of the parameter that was set to None in the call. The
+            value solves the power equation given the remaining parameters.
+
+
+        Notes
+        -----
+        The function uses scipy.optimize for finding the value that satisfies
+        the power equation. It first uses ``brentq`` with a prior search for
+        bounds. If this fails to find a root, ``fsolve`` is used. If ``fsolve``
+        also fails, then, for ``alpha``, ``power`` and ``effect_size``,
+        ``brentq`` with fixed bounds is used. However, there can still be cases
+        where this fails.
+
+        '''
+
+        return super(FTestPowerF2, self).solve_power(effect_size=effect_size,
+                                                      df_num=df_num,
+                                                      df_denom=df_denom,
+                                                      alpha=alpha,
+                                                      power=power,
+                                                      ncc=ncc)
+
+
 class FTestAnovaPower(Power):
     '''Statistical Power calculations F-test for one factor balanced ANOVA
 


Test patch:
diff --git a/statsmodels/stats/tests/test_power.py b/statsmodels/stats/tests/test_power.py
--- a/statsmodels/stats/tests/test_power.py
+++ b/statsmodels/stats/tests/test_power.py
@@ -88,7 +88,7 @@ def test_roots(self):
 
     @pytest.mark.matplotlib
     def test_power_plot(self, close_figures):
-        if self.cls == smp.FTestPower:
+        if self.cls in [smp.FTestPower, smp.FTestPowerF2]:
             pytest.skip('skip FTestPower plot_power')
         fig = plt.figure()
         ax = fig.add_subplot(2,1,1)
@@ -701,6 +701,33 @@ def test_kwargs(self):
                 junk=3)
 
 
+class TestFtestPowerF2(CheckPowerMixin):
+
+    @classmethod
+    def setup_class(cls):
+        res2 = Holder()
+        #> rf = pwr.f2.test(u=5, v=19, f2=0.3**2, sig.level=0.1)
+        #> cat_items(rf, "res2.")
+        res2.u = 5
+        res2.v = 19
+        res2.f2 = 0.09
+        res2.sig_level = 0.1
+        res2.power = 0.235454222377575
+        res2.method = 'Multiple regression power calculation'
+
+        cls.res2 = res2
+        cls.kwds = {'effect_size': res2.f2, 'df_num': res2.u,
+                     'df_denom': res2.v, 'alpha': res2.sig_level,
+                     'power': res2.power}
+        # keyword for which we do not look for root:
+        # solving for n_bins does not work, will not be used in regular usage
+        cls.kwds_extra = {}
+        cls.args_names = ['effect_size', 'df_num', 'df_denom', 'alpha']
+        cls.cls = smp.FTestPowerF2
+        # precision for test_power
+        cls.decimal = 5
+
+
 
 def test_power_solver():
     # messing up the solver to trigger backup