Repository: scipy/scipy

Issue:
scipy.stats.rv_continuous.fit throws empty RuntimeError when input has NaNs
<!-- 
Thank you for taking the time to report a SciPy issue.
Please describe the issue in detail, and for bug reports
fill in the fields below. You can delete the sections that 
don't apply to your issue. You can view the final output
by clicking the preview button above.
-->

When using the `scipy.stats.rv_continuous.fit` function, a `RuntimeError` (without a message) is thrown if the array to which the distribution is being fit contains `NaN`s. It would be nice to have a warning (or error) for NaNs.

### Reproducing code example:
<!-- 
If you place your code between the triple backticks below, 
it will be marked as a code block automatically 
-->


```python
import numpy as np
import scipy.stats as stats

x = np.array([1.6483, 1.8223, 2.7169, 2.4667, 1.1791, 3.5433, 2.6561, 1.6631,
       2.412 , 6.3796, 1.7192, 1.438 , 4.5905, 2.832 , 1.86  , 2.1236,
       2.5366, 2.442 , 1.7216, 1.6853, 2.4713, 1.7739, 3.3199, 3.711 ,
       1.821 , 1.6071, 3.9321, 3.4891, 3.0009, 2.0428, 2.6311, 2.218 ,
       2.3838, 2.6662, 2.3853, 2.3079, 3.4989, 4.8993, 8.8305, 3.6952,
       1.9436, 5.7345, 3.1823, 2.8493, 2.2327, 3.2036, 2.3663, 3.0886,
       2.4526, 2.0095, 2.6788, 4.3908, 2.6042, 1.1514, 3.6923, 2.5201,
       2.8638, 1.7267, 6.2628, 2.3748, 2.3494, 4.6349, 2.8303, 2.1384,
       5.0081, 2.3307, 3.543 , 1.8633, 2.3803, 2.0992, 1.8173, 1.6052,
       2.4661, 3.3964, 2.4746, 2.1748, 1.673 , 3.1572, 3.7291, 2.2546,
       1.5754, 2.2669, 1.6383, 1.7696, 1.9079, 1.0955, 1.5637, 2.66  ,
       1.746 , 1.5319, 2.0958,    np.nan])
K = stats.exponnorm.fit(x)
```

### Error message:
<!-- If any, paste the *full* error message inside a code block
as above (starting from line Traceback)
-->

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-86-fb3d43a0acaa> in <module>
     13        1.5754, 2.2669, 1.6383, 1.7696, 1.9079, 1.0955, 1.5637, 2.66  ,
     14        1.746 , 1.5319, 2.0958,    np.nan])
---> 15 K = stats.exponnorm.fit(x)

/anaconda3/lib/python3.7/site-packages/scipy/stats/_distn_infrastructure.py in fit(self, data, *args, **kwds)
   2195                                          'scale' in kwds):
   2196             # get distribution specific starting locations
-> 2197             start = self._fitstart(data)
   2198             args += start[Narg:-2]
   2199         loc = kwds.pop('loc', start[-2])

/anaconda3/lib/python3.7/site-packages/scipy/stats/_distn_infrastructure.py in _fitstart(self, data, args)
   2034         if args is None:
   2035             args = (1.0,)*self.numargs
-> 2036         loc, scale = self._fit_loc_scale_support(data, *args)
   2037         return args + (loc, scale)
   2038 

/anaconda3/lib/python3.7/site-packages/scipy/stats/_distn_infrastructure.py in _fit_loc_scale_support(self, data, *args)
   2287             return (data_b - b) + margin, 1
   2288         else:
-> 2289             raise RuntimeError
   2290 
   2291     def fit_loc_scale(self, data, *args):

RuntimeError: 
```

### Scipy/Numpy/Python version information:
```
1.2.1 1.15.4 sys.version_info(major=3, minor=7, micro=1, releaselevel='final', serial=0)
```



Discussion:
Thanks for reporting this bug. Looks like the generic `fit` method of `rv_continuous` ends up at this runtime error if the distribution has support on the real line (a=-inf and b=inf).

Another issue:
```
x = np.array([1e6, 1e6, 1e6, np.nan])
stats.exponnorm.fit_loc_scale(x, 1) # (0, 1), does not make sense
```

Reason are the following lines in stats/_distn_infrastructure.py in `fit_loc_scale`: 
```
if not np.isfinite(Lhat):
    Lhat = 0
if not (np.isfinite(Shat) and (0 < Shat)):
    Shat = 1
```
I can try to fix this if no one else is working on it already. 

>Looks like the generic fit method of rv_continuous ends up at this runtime error if the distribution has support on the real line (a=-inf and b=inf).

Not exactly, for example:
```
import numpy as np
import scipy.stats as stats
x = np.array([1.6483, 1.8223, 2.7169, 2.4667, np.nan])
K = stats.norm.fit(x)
>>> K
(nan, nan)
```
I'd guess that is b/c `scipy.stats.norm` provides custom implementation for the `fit` and `fit_start`
Which is one output someone might expect. Alternative behavior that might be reasonable would be to ignore the `np.nan` values and emit a warning.

Unfortunately the behavior really varies across distributions, e.g. (using same `x` input as above)
```
>>> stats.betaprime.fit(x)
/Users/rlucas/scipy-dev/scipy/scipy/stats/_continuous_distns.py:667: RuntimeWarning: divide by zero encountered in true_divide
  a/(b-1.0),
/Users/rlucas/scipy-dev/scipy/scipy/stats/_continuous_distns.py:671: RuntimeWarning: divide by zero encountered in true_divide
  a*(a+1.0)/((b-2.0)*(b-1.0)),
/Users/rlucas/scipy-dev/scipy/scipy/stats/_distn_infrastructure.py:1061: RuntimeWarning: invalid value encountered in subtract
  mu2 = mu2p - mu * mu
/Users/rlucas/scipy-dev/scipy/scipy/stats/_distn_infrastructure.py:903: RuntimeWarning: invalid value encountered in greater
  return (a < x) & (x < b)
/Users/rlucas/scipy-dev/scipy/scipy/stats/_distn_infrastructure.py:903: RuntimeWarning: invalid value encountered in less
  return (a < x) & (x < b)
(1.2093829159613558, 1.3349245819681377, nan, 2.38258322474805e-28)
```
and for `scipy.stats.expon` on the first time
```
>>> K = stats.expon.fit(x)
/Users/rlucas/anaconda3/envs/scipy-dev/lib/python3.6/site-packages/numpy/core/_methods.py:32: RuntimeWarning: invalid value encountered in reduce
  return umr_minimum(a, axis, None, out, keepdims, initial)
``` 
amd afterwards no error but output is a tuple of `np.nan`

```
>>> K = stats.expon.fit(x)
>>> K
(nan, nan)
```
I'd propose the behavior of emit a runtime warning and return a tuple of `np.nan` as a solution. 
@chrisb83 @WarrenWeckesser @pvanmulbregt thoughts and/or recommendations of alternate solution(s)?  
I'd agree that if the input array has `np.nan`s, then a warning should be thrown. If the input array has only `np.nan`, then an error should be thrown.

I don't see a good use case for which returning a tuple of `np.nan` is useful, but this might be a shortsighted statement.
>I don't see a good use case for which returning a tuple of np.nan is useful, but this might be a shortsighted statement.

@kmundnic My thinking is that if you are getting `np.nan` that by propogating the warning through the code it is more likely to be surfaced. However, as I'm writing this I'm thinking maybe just throwing an error would be better. If the behavior is runtime warning and continue with a fit then to get something other than `np.nan` you'd need to ignore the `np.nan` values. However, once you get into that territory the number of non `np.nan` observations you need will differ across distributions, (e.g. single location parameter vs 2 parameter location-scale family. While the fit function might have that information available the logic can become fairly complicated. I'd prefer a simpler solution if possible which would be to emit `np.nan` as estimates. I know @WarrenWeckesser worked for a while on warnings in the SciPy libraries would like to get his recommendation before proceeding in any direction on fixing the defect.
> I'd propose the behavior of emit a runtime warning and return a tuple of np.nan as a solution

Returning nan is aligned to the default behaviour `nan_policy = 'propagate'` in other stats functions, raising an error corresponds to `'raise'`.

Any nonfinite value such as `Inf` probably need to be handled in a similar way.
Thanks for the input @chrisb83 It sounds (mostly) clear what to do except not sure if a PR should also address adding the nan_policy functionality or if that should be a separate PR? 

> On Jul 3, 2019, at 12:43 AM, Christoph Baumgarten <notifications@github.com> wrote:
> 
> I'd propose the behavior of emit a runtime warning and return a tuple of np.nan as a solution
> 
> Returning nan is aligned to the default behaviour nan_policy = 'propagate' in other stats functions, raising an error corresponds to 'raise'.
> 
> Any nonfinite value such as Inf probably need to be handled in a similar way.
> 
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub, or mute the thread.

If we were to add the `nan_policy` argument to the `fit` method, the behavior when the input contains `nan` would be:

* `nan_policy == 'raise'`: Raise an exception.
* `nan_policy == 'omit'`: Ignore any `nan` values in the input.  Do not generate a warning if the input contains `nan`.  E.g. `fit([1, 2, np.nan, 3])` should behave the same as `fit([1, 2, 3])`.  If *all* the values in the input are `nan`, the function should behave as if the input was empty.  That is, `fit([np.nan, np.nan, np.nan])` should behave the same as `fit([])`.  Unfortunately, the current behavior of the distributions in that case is inconsistent.  I just tried `norm` and `beta`.  `norm` generated a warning and returned `(nan, nan)`.  `beta` raised an exception.  I think the correct behavior is to raise an exception. The exception message should state explicitly that an empty input is not accepted.  (The error currently generated by `beta` is incidental, and the error message is not very helpful.)
* `nan_policy == 'propagate'`: Return `nan`s as the result.  Do not generate a warning.

Note that `nan_policy` controls how the floating point value `nan` is handled.  It does not affect how `inf` is handled.

If we *don't* add the argument `nan_policy`, then the two most reasonable options seem to be (1) generate a warning and return `nan`s, or (2) raise an exception.  My inclination is to raise an exception, but that's what I'll say if you ask me today.  Ask me tomorrow, and I might prefer a warning and returning `nan`s, or maybe even `nan`s with no warning.  (The argument in favor of the latter: many functions in NumPy pass through `nan`s without a warning, e.g. `np.sin([1, np.nan, 2])`, `np.mean([1, np.nan, 2])`, etc. Should we be consistent with that behavior?)  Given that, I'm *more* inclined to say we should implement the `nan_policy` argument (explicit is better than implicit, refuse the temptation to guess, etc.).  The default value should probably be `'propagate'`--that seems most consistent with the current behavior.  We can't expect complete consistency with the current behavior, because the current behavior is wildly inconsistent, as @rlucas7 noted above.  The `fit` methods were not written to take `nan` into account, and it shows.

For example,

    In [13]: from scipy.stats import norm, beta, exponnorm, gamma, weibull_min                  

    In [14]: x = np.array([0.1, 1, 2, 3, 3, np.nan, 5])

`norm.fit(x)` returns `(nan, nan)` with no warning:

    In [15]: norm.fit(x)                                                            
    Out[15]: (nan, nan)

`gamma.fit(x)` return a mix of `nan` and 1.0 (which makes no sense):

    In [16]: gamma.fit(x)                                                           
    Out[16]: (nan, nan, 1.0)

`beta.fit(x)` generates incidental warnings and returns a *different* mix of `nan`s and 1.0 :

    In [17]: beta.fit(x)                                                            
    /Users/warren/miniconda36b/lib/python3.6/site-packages/scipy/optimize/minpack.py:162: RuntimeWarning: The iteration is not making good progress, as measured by the 
      improvement from the last ten iterations.
      warnings.warn(msg, RuntimeWarning)
    /Users/warren/miniconda36b/lib/python3.6/site-packages/scipy/stats/_distn_infrastructure.py:874: RuntimeWarning: invalid value encountered in greater_equal
      return (self.a <= x) & (x <= self.b)
    /Users/warren/miniconda36b/lib/python3.6/site-packages/scipy/stats/_distn_infrastructure.py:874: RuntimeWarning: invalid value encountered in less_equal
      return (self.a <= x) & (x <= self.b)
    Out[17]: (1.0, 1.0, nan, nan)

Similarly, `weibull_min.fit(x)` generates incidental warnings, and returns yet another mix of `nan` and non-`nan` values:

    In [18]: weibull_min.fit(x)                                                     
    /Users/warren/miniconda36b/lib/python3.6/site-packages/scipy/stats/_distn_infrastructure.py:874: RuntimeWarning: invalid value encountered in greater_equal
      return (self.a <= x) & (x <= self.b)
    /Users/warren/miniconda36b/lib/python3.6/site-packages/scipy/stats/_distn_infrastructure.py:874: RuntimeWarning: invalid value encountered in less_equal
      return (self.a <= x) & (x <= self.b)
    Out[18]: (1.3053444453258112, nan, 3.902134995832234e-30)

And, as shown above, `exponnorm.fit(x)` raises a `RuntimeError`, with no error message in the exception:

    In [19]: exponnorm.fit(x)                                                       
    ---------------------------------------------------------------------------
    RuntimeError                              Traceback (most recent call last)
    <ipython-input-18-390b024fc8e6> in <module>
    ----> 1 exponnorm.fit(x)

    ~/miniconda36b/lib/python3.6/site-packages/scipy/stats/_distn_infrastructure.py in fit(self, data, *args, **kwds)
       2195                                          'scale' in kwds):
       2196             # get distribution specific starting locations
    -> 2197             start = self._fitstart(data)
       2198             args += start[Narg:-2]
       2199         loc = kwds.pop('loc', start[-2])

    ~/miniconda36b/lib/python3.6/site-packages/scipy/stats/_distn_infrastructure.py in _fitstart(self, data, args)
       2034         if args is None:
       2035             args = (1.0,)*self.numargs
    -> 2036         loc, scale = self._fit_loc_scale_support(data, *args)
       2037         return args + (loc, scale)
       2038 

    ~/miniconda36b/lib/python3.6/site-packages/scipy/stats/_distn_infrastructure.py in _fit_loc_scale_support(self, data, *args)
       2287             return (data_b - b) + margin, 1
       2288         else:
    -> 2289             raise RuntimeError
       2290 
       2291     def fit_loc_scale(self, data, *args):

    RuntimeError:

`nan_policy` has a few issues at the moment, see  #9558

I would favor just raising an error or return nan if there is any nonfinite value.The majority seems to favor raising an error, i think that is fine.
I agree that long term we’d like a nan_policy and I could do that but it would be blocked by the other issue and not clear when that would be resolved. So I’ll break this into 2 PRs, one that implements a default ‘propogate’ policy for nan, which could be merged as soon as approved. The I’ll open a 2nd PR which implements a full nan_policy and may be blocked by other side until resolved. Please chime in if any objections. The default policy I’ll likely have a PR out for by end of this coming weekend. 

Sincerely, 

-Lucas Roberts 

> On Jul 6, 2019, at 6:47 AM, Christoph Baumgarten <notifications@github.com> wrote:
> 
> nan_policy has a few issues at the moment, see #9558
> 
> I would favor just raising an error or return nan if there is any nonfinite value.The majority seems to favor raising an error, i think that is fine.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.

@chrisb83 and @WarrenWeckesser I'm looking at some of the other methods in `stats.py` to get an idea of what to do. I'm having a bit of difficulty identifying a function that has a correctly implemented version of `nan_policy='propagate'` for example:

```
>>> sc.moment([np.nan, np.nan, np.nan, 1, 2, 3,], moment=1, nan_policy='propagate')
0.0
```
which doesn't seem correct. I would expect the result to be `nan` I can open an issue for the moment function is this seems incorrect. Can either of you point me to a function which has a functioning `propagate` nan policy in the `stats.py` file (or somewhere else in scipy)?

Maybe `variation()`'s `nan_policy` works as intended?

It appears that for `moment=2` and higher the result is what I would expect. 

```
>>> sc.moment([np.nan, np.nan, np.nan, 1, 2, 3,], moment=2, nan_policy='propagate')
nan
>>> sc.moment([np.nan, np.nan, np.nan, 1, 2, 3,], moment=3, nan_policy='propagate')
nan
>>> 
```

Resolving patch:
diff --git a/scipy/stats/_continuous_distns.py b/scipy/stats/_continuous_distns.py
--- a/scipy/stats/_continuous_distns.py
+++ b/scipy/stats/_continuous_distns.py
@@ -284,6 +284,9 @@ def fit(self, data, **kwds):
 
         data = np.asarray(data)
 
+        if not np.isfinite(data).all():
+            raise RuntimeError("The data contains non-finite values.")
+
         if floc is None:
             loc = data.mean()
         else:
@@ -584,10 +587,14 @@ def fit(self, data, *args, **kwds):
         # "Two unknown parameters" in the section "Maximum likelihood" of
         # the Wikipedia article on the Beta distribution for the formulas.)
 
+        if not np.isfinite(data).all():
+            raise RuntimeError("The data contains non-finite values.")
+
         # Normalize the data to the interval [0, 1].
         data = (np.ravel(data) - floc) / fscale
         if np.any(data <= 0) or np.any(data >= 1):
             raise FitDataError("beta", lower=floc, upper=floc + fscale)
+
         xbar = data.mean()
 
         if f0 is not None or f1 is not None:
@@ -1440,7 +1447,12 @@ def fit(self, data, *args, **kwds):
                              "optimize.")
 
         data = np.asarray(data)
+
+        if not np.isfinite(data).all():
+            raise RuntimeError("The data contains non-finite values.")
+
         data_min = data.min()
+
         if floc is None:
             # ML estimate of the location is the minimum of the data.
             loc = data_min
@@ -2752,8 +2764,13 @@ def fit(self, data, *args, **kwds):
 
         # Fixed location is handled by shifting the data.
         data = np.asarray(data)
+
+        if not np.isfinite(data).all():
+            raise RuntimeError("The data contains non-finite values.")
+
         if np.any(data <= floc):
             raise FitDataError("gamma", lower=floc, upper=np.inf)
+
         if floc != 0:
             # Don't do the subtraction in-place, because `data` might be a
             # view of the input array.
@@ -4892,6 +4909,10 @@ def fit(self, data, *args, **kwds):
                              "optimize.")
 
         data = np.asarray(data)
+
+        if not np.isfinite(data).all():
+            raise RuntimeError("The data contains non-finite values.")
+
         floc = float(floc)
         if floc != 0:
             # Shifting the data by floc. Don't do the subtraction in-place,
@@ -7374,6 +7395,9 @@ def fit(self, data, *args, **kwds):
 
         data = np.asarray(data)
 
+        if not np.isfinite(data).all():
+            raise RuntimeError("The data contains non-finite values.")
+
         # MLE for the uniform distribution
         # --------------------------------
         # The PDF is
diff --git a/scipy/stats/_distn_infrastructure.py b/scipy/stats/_distn_infrastructure.py
--- a/scipy/stats/_distn_infrastructure.py
+++ b/scipy/stats/_distn_infrastructure.py
@@ -2232,6 +2232,8 @@ def fit(self, data, *args, **kwds):
         penalty applied for samples outside of range of the distribution. The
         returned answer is not guaranteed to be the globally optimal MLE, it
         may only be locally optimal, or the optimization may fail altogether.
+        If the data contain any of np.nan, np.inf, or -np.inf, the fit routine
+        will throw a RuntimeError.
 
         Examples
         --------
@@ -2275,6 +2277,9 @@ def fit(self, data, *args, **kwds):
         if Narg > self.numargs:
             raise TypeError("Too many input arguments.")
 
+        if not np.isfinite(data).all():
+            raise RuntimeError("The data contains non-finite values.")
+
         start = [None]*2
         if (Narg < self.numargs) or not ('loc' in kwds and
                                          'scale' in kwds):


Test patch:
diff --git a/scipy/stats/tests/test_distributions.py b/scipy/stats/tests/test_distributions.py
--- a/scipy/stats/tests/test_distributions.py
+++ b/scipy/stats/tests/test_distributions.py
@@ -1491,6 +1491,42 @@ def test_tail(self):  # Regression test for ticket 807
         assert_equal(stats.expon.cdf(1e-18), 1e-18)
         assert_equal(stats.expon.isf(stats.expon.sf(40)), 40)
 
+    def test_nan_raises_error(self):
+        # see gh-issue 10300
+        x = np.array([1.6483, 2.7169, 2.4667, 1.1791, 3.5433, np.nan])
+        assert_raises(RuntimeError, stats.expon.fit, x)
+
+    def test_inf_raises_error(self):
+        # see gh-issue 10300
+        x = np.array([1.6483, 2.7169, 2.4667, 1.1791, 3.5433, np.inf])
+        assert_raises(RuntimeError, stats.expon.fit, x)
+
+
+class TestNorm(object):
+    """gh-10300"""
+    def test_nan_raises_error(self):
+        # see gh-issue 10300
+        x = np.array([1.6483, 2.7169, 2.4667, 1.1791, 3.5433, np.nan])
+        assert_raises(RuntimeError, stats.norm.fit, x)
+
+    def test_inf_raises_error(self):
+        # see gh-issue 10300
+        x = np.array([1.6483, 2.7169, 2.4667, 1.1791, 3.5433, np.inf])
+        assert_raises(RuntimeError, stats.norm.fit, x)
+
+
+class TestUniform(object):
+    """gh-10300"""
+    def test_nan_raises_error(self):
+        # see gh-issue 10300
+        x = np.array([1.6483, 2.7169, 2.4667, 1.1791, 3.5433, np.nan])
+        assert_raises(RuntimeError, stats.uniform.fit, x)
+
+    def test_inf_raises_error(self):
+        # see gh-issue 10300
+        x = np.array([1.6483, 2.7169, 2.4667, 1.1791, 3.5433, np.inf])
+        assert_raises(RuntimeError, stats.uniform.fit, x)
+
 
 class TestExponNorm(object):
     def test_moments(self):
@@ -1520,6 +1556,16 @@ def get_moms(lam, sig, mu):
         sts = stats.exponnorm.stats(K, loc=mu, scale=sig, moments='mvsk')
         assert_almost_equal(sts, get_moms(lam, sig, mu))
 
+    def test_nan_raises_error(self):
+        # see gh-issue 10300
+        x = np.array([1.6483, 2.7169, 2.4667, 1.1791, 3.5433, np.nan])
+        assert_raises(RuntimeError, stats.exponnorm.fit, x, floc=0, fscale=1)
+
+    def test_inf_raises_error(self):
+        # see gh-issue 10300
+        x = np.array([1.6483, 2.7169, 2.4667, 1.1791, 3.5433, np.inf])
+        assert_raises(RuntimeError, stats.exponnorm.fit, x, floc=0, fscale=1)
+
     def test_extremes_x(self):
         # Test for extreme values against overflows
         assert_almost_equal(stats.exponnorm.pdf(-900, 1), 0.0)
@@ -2020,6 +2066,20 @@ class TestFitMethod(object):
     def setup_method(self):
         np.random.seed(1234)
 
+    # skip these b/c deprecated, or only loc and scale arguments
+    fitSkipNonFinite = ['frechet_l', 'frechet_r', 'expon', 'norm', 'uniform', ]
+
+    @pytest.mark.parametrize('dist,args', distcont)
+    def test_fit_w_non_finite_data_values(self, dist, args):
+        """gh-10300"""
+        if dist in self.fitSkipNonFinite:
+            pytest.skip("%s fit known to fail or deprecated" % dist)
+        x = np.array([1.6483, 2.7169, 2.4667, 1.1791, 3.5433, np.nan])
+        y = np.array([1.6483, 2.7169, 2.4667, 1.1791, 3.5433, np.inf])
+        distfunc = getattr(stats, dist)
+        assert_raises(RuntimeError, distfunc.fit, x, floc=0, fscale=1)
+        assert_raises(RuntimeError, distfunc.fit, y, floc=0, fscale=1)
+
     def test_fix_fit_2args_lognorm(self):
         # Regression test for #1551.
         np.random.seed(12345)