Repository: scipy/scipy

Issue:
WIP: Addition of MLE for stats.invgauss/wald
Part of the #11782 issue.

@mdhaber,


I have been investigating the maximum likelihood estimators given in Chapter 25, Inverse Gaussian (Wald) Distribution, but the results from the MLEs don’t seem to come close to the results of either the [wald](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.wald.html#scipy.stats.wald) or [invgauss](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.invgauss.html#scipy.stats.invgauss) distributions in scipy. 

Here is how the PDF is described in the textbook we have been using, on page 120: 
<img width="509" alt="Screen Shot 2020-07-02 at 5 01 53 PM" src="https://user-images.githubusercontent.com/44255917/86418668-bb768380-bc85-11ea-81bc-192776c5f1eb.png">

And here are the equations for MLE. 
<img width="495" alt="Screen Shot 2020-07-02 at 5 02 22 PM" src="https://user-images.githubusercontent.com/44255917/86418686-cdf0bd00-bc85-11ea-9729-728e40c779e5.png">

It may be another issue with terminology, since the use of mu to represent the location, as the textbook describes, is implemented as a shape parameter in [`stats.invgauss`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.invgauss.html#scipy.stats.invgauss). However, the mean of a random generate only appears to represent the `mu` used in generation when `loc` and `scale` are the standard `loc=0, scale=1`. 

For example, 

```
from scipy.stats import invgauss
data = invgauss.rvs(mu=3.25, size=10000)
print(data.mean())
```
=> `3.223328...`. Just one example, but it appears that in general it matches.


With other location and scale values it doesn't match. 

```
from scipy.stats import invgauss
data = invgauss.rvs(mu=3.25, scale=2, size=10000)
print(data.mean())
```
=> `6.62447...`. Just one example, but in general they don't match.


The code for the scale is

```scale = len(data) / (np.sum(data**(-1) - mu**(-1)))```

and I played around with both of these equations quite a bit to try to get them to match up with / or be better than the default fit method, but other than the relation I described above, I'm not sure that these will work for these distributions. It may be possible to use the mean to determine the shape parameter in `invgauss` if the person has already fixed `floc=0, scale=1`, but that seems like a very niche usage. 

What are your thoughts? 


Discussion:
I agree that there is not an obvious direct way to convert between $\lambda, \mu$ from the textbook and `mu`, `loc`, `scale` of SciPy in general, and it's not just because SciPy adds `loc`, but also because `scale` $\neq \lambda$. Could you write out SciPy's full formula for the PDF by making the substitution mentioned in the SciPy documentation:

> Specifically, invgauss.pdf(x, mu, loc, scale) is identically equivalent to invgauss.pdf(y, mu) / scale with y = (x - loc) / scale.

And put it side by side with the book's formula for the PDF so we can more easily compare?  Please render them both  with SciPy's `mu`, `loc`, `scale` as ![image](https://user-images.githubusercontent.com/6570539/86526685-e6aacf80-be4b-11ea-83f4-a739230abf04.png) (e.g. using https://quicklatex.com/)

scipy's Invgauss after applying 

> `invgauss.pdf(x, mu, loc, scale)` is identically equivalent to `invgauss.pdf(y, mu) / scale with y = (x - loc) / scale.`



![image](https://user-images.githubusercontent.com/44255917/86721472-de34cf00-bfda-11ea-92ed-f3e73bd253d1.png)


And then SciPy's wald pdf after the same:

> `wald.pdf(x, loc, scale)` is identically equivalent to `wald.pdf(y) / scale with y = (x - loc) / scale`.

![image](https://user-images.githubusercontent.com/44255917/86713694-2fd95b80-bfd3-11ea-9b57-79a0f9382850.png)


All in comparison to the PDF found in the textbook:
<img width="297" alt="Screen Shot 2020-07-06 at 9 33 27 PM" src="https://user-images.githubusercontent.com/44255917/86710966-55b13100-bfd0-11ea-858b-5791b546c0d6.png">

or, rewriting to be more similar style as scipy: 
![image](https://user-images.githubusercontent.com/44255917/86719947-67e39d00-bfd9-11ea-9531-e2776dbafd1d.png)


So, in all honesty, I don't see a way to refactor that to match wald or invgauss. Should I 'call it' trying to apply MLEs for this one?



I think if you go a bit further, you'll find with `loc=0` that they are equivalent with
![image](https://user-images.githubusercontent.com/6570539/86788426-047a5f00-c01b-11ea-9bdf-1978c22eb885.png)

So if the user passes in `floc=0`, you could use the equations from the book to get the book's version of the parameters, then use the relationships above to get SciPy's parameters. 

_Update: Yes:_
```
import numpy as np
from scipy.stats import invgauss
data = invgauss.rvs(mu=3.25, scale=2, size=1000000)

mu = np.mean(data)
s = len(data) / (np.sum(data**(-1) - mu**(-1)))
mu_s = mu/s
print(mu_s, s)
```
gives
```
3.2698217368812785 1.9933346841496646
```
If the user doesn't pass in `floc=0`, then you could at least use the analytical solution above (assuming `floc=0`) as a guess to the `super` `fit` method. You might try following @WarrenWeckesser's argument [here](https://github.com/scipy/scipy/issues/11806#issuecomment-651242564) about `weibull_min` to see if it applies to this distribution.  
@mdhaber, great, that conversion between the SciPy parameters (when `loc` is 0) and the text's parameters checks out, so we can use those MLE formulas when `floc` is given.

By the way (maybe you are already aware of this, and just use `floc=0` as shorthand): we don't have to restrict this to just `0` for `floc`.  For any constant `c`, if the user gives `floc=c`, we can apply the analytical formulas to `x - c`.  We just have to first check that each value in `x` is greater than `c`, and if not, raise an exception.

> By the way

I was having thoughts along those lines but not fully fleshed out. Thanks for clarifying @WarrenWeckesser!

Sound good, @swallan?
@mdhaber @WarrenWeckesser 

Thanks for the pointers! I really appreciate it. I’ll create a PR for this soon. 

Resolving patch:
diff --git a/scipy/stats/_continuous_distns.py b/scipy/stats/_continuous_distns.py
--- a/scipy/stats/_continuous_distns.py
+++ b/scipy/stats/_continuous_distns.py
@@ -3392,6 +3392,37 @@ def _cdf(self, x, mu):
     def _stats(self, mu):
         return mu, mu**3.0, 3*np.sqrt(mu), 15*mu
 
+    def fit(self, data, *args, **kwds):
+
+        if type(self) == wald_gen:
+            return super(invgauss_gen, self).fit(data, *args, **kwds)
+
+        data, fshape_s, floc, fscale = _check_fit_input_parameters(self, data,
+                                                                   args, kwds)
+        '''
+        Source: Statistical Distributions, 3rd Edition. Evans, Hastings,
+        and Peacock (2000), Page 121. Their shape parameter is equivilent to
+        SciPy's with the conversion `fshape_s = fshape / scale`.
+
+        MLE formulas are not used in 3 condtions:
+        - `loc` is not fixed
+        - `mu` is fixed
+        These cases fall back on the superclass fit method.
+        - `loc` is fixed but translation results in negative data raises
+          a `FitDataError`.
+        '''
+        if floc is None or fshape_s is not None:
+            return super(invgauss_gen, self).fit(data, *args, **kwds)
+        elif np.any(data - floc < 0):
+            raise FitDataError("invgauss", lower=0, upper=np.inf)
+        else:
+            data = data - floc
+            fshape_n = np.mean(data)
+            if fscale is None:
+                fscale = len(data) / (np.sum(data ** -1 - fshape_n ** -1))
+            fshape_s = fshape_n / fscale
+        return fshape_s, floc, fscale
+
 
 invgauss = invgauss_gen(a=0.0, name='invgauss')
 
@@ -3991,7 +4022,7 @@ def _check_fit_input_parameters(dist, data, args, kwds):
     floc = kwds.get('floc', None)
     fscale = kwds.get('fscale', None)
 
-    num_shapes = len(dist.shapes) if dist.shapes else 0
+    num_shapes = len(dist.shapes.split(",")) if dist.shapes else 0
     fshape_keys = []
     fshapes = []
 


Test patch:
diff --git a/scipy/stats/tests/test_distributions.py b/scipy/stats/tests/test_distributions.py
--- a/scipy/stats/tests/test_distributions.py
+++ b/scipy/stats/tests/test_distributions.py
@@ -77,6 +77,37 @@ def test_vonmises_numerical():
     assert_almost_equal(vm.cdf(0), 0.5)
 
 
+def _assert_lessthan_loglike(dist, data, func, **kwds):
+    mle_analytical = dist.fit(data, **kwds)
+    numerical_opt = super(type(dist), dist).fit(data, **kwds)
+    ll_mle_analytical = func(mle_analytical, data)
+    ll_numerical_opt = func(numerical_opt, data)
+    assert ll_mle_analytical < ll_numerical_opt
+
+
+def assert_fit_warnings(dist):
+    param = ['floc', 'fscale']
+    if dist.shapes:
+        nshapes = len(dist.shapes.split(","))
+        param += ['f0', 'f1', 'f2'][:nshapes]
+    all_fixed = dict(zip(param, np.arange(len(param))))
+    data = [1, 2, 3]
+    with pytest.raises(RuntimeError,
+                       match="All parameters fixed. There is nothing "
+                       "to optimize."):
+        dist.fit(data, **all_fixed)
+    with pytest.raises(RuntimeError,
+                       match="The data contains non-finite values"):
+        dist.fit([np.nan])
+    with pytest.raises(RuntimeError,
+                       match="The data contains non-finite values"):
+        dist.fit([np.inf])
+    with pytest.raises(TypeError, match="Unknown keyword arguments:"):
+        dist.fit(data, extra_keyword=2)
+    with pytest.raises(TypeError, match="Too many positional arguments."):
+        dist.fit(data, *[1]*(len(param) - 1))
+
+
 @pytest.mark.parametrize('dist',
                          ['alpha', 'betaprime',
                           'fatiguelife', 'invgamma', 'invgauss', 'invweibull',
@@ -1030,45 +1061,28 @@ def test_fit_MLE_comp_optimzer(self, rvs_shape, rvs_loc, rvs_scale):
         args = [data, (stats.pareto._fitstart(data), )]
         func = stats.pareto._reduce_func(args, {})[1]
 
-        def _assert_lessthan_loglike(dist, data, func, **kwds):
-            mle_analytical = dist.fit(data, **kwds)
-            numerical_opt = super(type(dist), dist).fit(data, **kwds)
-            ll_mle_analytical = func(mle_analytical, data)
-            ll_numerical_opt = func(numerical_opt, data)
-            assert ll_mle_analytical < ll_numerical_opt
-
-        # fixed `floc` to actual location provides as good or better fit.
+        # fixed `floc` to actual location provides a better fit than the
+        # super method
         _assert_lessthan_loglike(stats.pareto, data, func, floc=rvs_loc)
 
-        # fixing `floc` to an arbitrary number, 0, still provides an as good
-        # or better fit.
+        # fixing `floc` to an arbitrary number, 0, still provides a better
+        # fit than the super method
         _assert_lessthan_loglike(stats.pareto, data, func, floc=0)
 
-        # fixed shape still uses analytical MLE and provides
-        # an as good or better fit.
+        # fixed shape still uses MLE formula and provides a better fit than
+        # the super method
         _assert_lessthan_loglike(stats.pareto, data, func, floc=0, f0=4)
 
-        # valid fixed fscale still uses analytical MLE and provides
-        # an as good or better fit.
+        # valid fixed fscale still uses MLE formulas and provides a better
+        # fit than the super method
         _assert_lessthan_loglike(stats.pareto, data, func, floc=0,
                                  fscale=rvs_scale/2)
 
     def test_fit_warnings(self):
-        with pytest.raises(RuntimeError,
-                           match="All parameters fixed. There is nothing "
-                           "to optimize."):
-            stats.pareto.fit([1, 2, 3], f0=2, floc=1, fscale=1)
-        with pytest.raises(RuntimeError,
-                           match="The data contains non-finite values"):
-            stats.pareto.fit([np.nan])
-        with pytest.raises(RuntimeError,
-                           match="The data contains non-finite values"):
-            stats.pareto.fit([np.inf])
-        with pytest.raises(TypeError, match="Unknown keyword arguments:"):
-            stats.pareto.fit([2, 2, 3], floc=1, extra=2)
-        with pytest.raises(TypeError, match="Too many positional arguments."):
-            stats.pareto.fit([1, 2, 3], 1, 4)
+        assert_fit_warnings(stats.pareto)
+        # `floc` that causes invalid negative data
         assert_raises(FitDataError, stats.pareto.fit, [1, 2, 3], floc=2)
+        # `floc` and `fscale` combination causes invalid data
         assert_raises(FitDataError, stats.pareto.fit, [5, 2, 3], floc=1,
                       fscale=3)
 
@@ -1540,6 +1554,87 @@ def test_stats2(self):
         assert_allclose((v, k), (4., 3.25))
 
 
+class TestInvgauss(object):
+    def setup_method(self):
+        np.random.seed(1234)
+
+    @pytest.mark.parametrize("rvs_mu,rvs_loc,rvs_scale",
+                             [(2, 0, 1), (np.random.rand(3)*10)])
+    def test_fit(self, rvs_mu, rvs_loc, rvs_scale):
+        data = stats.invgauss.rvs(size=100, mu=rvs_mu,
+                                  loc=rvs_loc, scale=rvs_scale)
+        # Analytical MLEs are calculated with formula when `floc` is fixed
+        mu, loc, scale = stats.invgauss.fit(data, floc=rvs_loc)
+
+        data = data - rvs_loc
+        mu_temp = np.mean(data)
+        scale_mle = len(data) / (np.sum(data**(-1) - mu_temp**(-1)))
+        mu_mle = mu_temp/scale_mle
+
+        # `mu` and `scale` match analytical formula
+        assert_allclose(mu_mle, mu, atol=1e-15, rtol=1e-15)
+        assert_allclose(scale_mle, scale, atol=1e-15, rtol=1e-15)
+        assert_equal(loc, rvs_loc)
+        data = stats.invgauss.rvs(size=100, mu=rvs_mu,
+                                  loc=rvs_loc, scale=rvs_scale)
+        # fixed parameters are returned
+        mu, loc, scale = stats.invgauss.fit(data, floc=rvs_loc - 1,
+                                            fscale=rvs_scale + 1)
+        assert_equal(rvs_scale + 1, scale)
+        assert_equal(rvs_loc - 1, loc)
+
+        # shape can still be fixed with multiple names
+        shape_mle1 = stats.invgauss.fit(data, fmu=1.04)[0]
+        shape_mle2 = stats.invgauss.fit(data, fix_mu=1.04)[0]
+        shape_mle3 = stats.invgauss.fit(data, f0=1.04)[0]
+        assert shape_mle1 == shape_mle2 == shape_mle3 == 1.04
+
+    @pytest.mark.parametrize("rvs_mu,rvs_loc,rvs_scale",
+                             [(2, 0, 1), (np.random.rand(3)*10)])
+    def test_fit_MLE_comp_optimzer(self, rvs_mu, rvs_loc, rvs_scale):
+        data = stats.invgauss.rvs(size=100, mu=rvs_mu,
+                                  loc=rvs_loc, scale=rvs_scale)
+
+        super_fit = super(type(stats.invgauss), stats.invgauss).fit
+        # fitting without `floc` uses superclass fit method
+        super_fitted = super_fit(data)
+        invgauss_fit = stats.invgauss.fit(data)
+        assert_equal(super_fitted, invgauss_fit)
+
+        # fitting with `fmu` is uses superclass fit method
+        super_fitted = super_fit(data, floc=0, fmu=2)
+        invgauss_fit = stats.invgauss.fit(data, floc=0, fmu=2)
+        assert_equal(super_fitted, invgauss_fit)
+
+        # obtain log-likelihood objective function to compare results
+        args = [data, (stats.invgauss._fitstart(data), )]
+        func = stats.invgauss._reduce_func(args, {})[1]
+
+        # fixed `floc` uses analytical formula and provides better fit than
+        # super method
+        _assert_lessthan_loglike(stats.invgauss, data, func, floc=rvs_loc)
+
+        # fixed `floc` not resulting in invalid data < 0 uses analytical
+        # formulas and provides a better fit than the super method
+        assert np.all((data - (rvs_loc - 1)) > 0)
+        _assert_lessthan_loglike(stats.invgauss, data, func, floc=rvs_loc - 1)
+
+        # fixed `floc` to an arbitrary number, 0, still provides a better fit
+        # than the super method
+        _assert_lessthan_loglike(stats.invgauss, data, func, floc=0)
+
+        # fixed `fscale` to an arbitrary number still provides a better fit
+        # than the super method
+        _assert_lessthan_loglike(stats.invgauss, data, func, floc=rvs_loc,
+                                 fscale=np.random.rand(1)[0])
+
+    def test_fit_raise_errors(self):
+        assert_fit_warnings(stats.invgauss)
+        # FitDataError is raised when negative invalid data
+        with pytest.raises(FitDataError):
+            stats.invgauss.fit([1, 2, 3], floc=2)
+
+
 class TestLaplace(object):
     @pytest.mark.parametrize("rvs_loc", [-5, 0, 1, 2])
     @pytest.mark.parametrize("rvs_scale", [1, 2, 3, 10])
@@ -1552,12 +1647,12 @@ def test_fit(self, rvs_loc, rvs_scale):
         loc_mle = np.median(data)
         scale_mle = np.sum(np.abs(data - loc_mle)) / len(data)
 
-        # standard outputs should match MLE
+        # standard outputs should match analytical MLE formulas
         loc, scale = stats.laplace.fit(data)
         assert_allclose(loc, loc_mle, atol=1e-15, rtol=1e-15)
         assert_allclose(scale, scale_mle, atol=1e-15, rtol=1e-15)
 
-        # fixed parameter should use MLE for other
+        # fixed parameter should use analytical formula for other
         loc, scale = stats.laplace.fit(data, floc=loc_mle)
         assert_allclose(scale, scale_mle, atol=1e-15, rtol=1e-15)
         loc, scale = stats.laplace.fit(data, fscale=scale_mle)
@@ -1571,12 +1666,12 @@ def test_fit(self, rvs_loc, rvs_scale):
         # fixed loc to non median, scale should match
         # scale calculation with modified loc
         loc, scale = stats.laplace.fit(data, floc=loc)
-        assert_allclose(scale, scale_mle, atol=1e-15, rtol=1e-15)
+        assert_equal(scale_mle, scale)
 
         # fixed scale created with non median loc,
         # loc output should still be the data median.
         loc, scale = stats.laplace.fit(data, fscale=scale_mle)
-        assert_allclose(loc_mle, loc, atol=1e-15, rtol=1e-15)
+        assert_equal(loc_mle, loc)
 
         # error raised when both `floc` and `fscale` are fixed
         assert_raises(RuntimeError, stats.laplace.fit, data, floc=loc_mle,