Repository: scipy/scipy

Issue:
Discrete Distribution Yule-Simon support
<<Please describe the issue in detail here, and for bug reports fill in the fields below.>>
I am what some might call a domain expert in statistics (my PhD is in statistics). I've derived some results on the yule simon distribution and would like to implement them in the scipy library. 

I'm willing to implement the distribution myself and do a pull request. However, I want to confirm support for the distribution with the core developer team before I begin the work of coding. Can someone confirm (or deny)? 

Here is [the link to the paper](https://arxiv.org/pdf/1710.08511.pdf) 

where we derive the variance of the estimator, and give a new framework to think about the procedure for estimation (same algorithm was derived in a previous paper via a maximum likelihood argument, we derive via an em algorithm which gives a variance estimate and a convergence proof for a subset of the values of the parameter)

here are some related papers on [yule simon](https://scholar.google.com/scholar?q=related:F6IAd0FjgsUJ:scholar.google.com/&hl=en&as_sdt=0,33)

and [a paper by Devroye](http://luc.devroye.org/digammapaper.ps) to simulate the random variable (yule simon as a special case of some more general random variate):

In the meantime I'm more than happy to do one of the "good first issue" issues in the scipy repo to get started contributing. 
### Reproducing code example:
```
<<A short code example that reproduces the problem. It should be self-contained, i.e., possible to run as-is via 'python myproblem.py'>>
```

### Error message:
```
<<Full error message, if any (starting from line Traceback: ...)>>
```

### Scipy/Numpy/Python version information:
```
<<Output from 'import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info)'>>
```




Discussion:
Hi @rlucas7, thanks for your interest. The Yule-Simon distribution looks like a useful addition to `scipy.stats` to me, we'd be happy to get a PR for it.

In SciPy we tend to prefer not to include code based on new research/papers, but only after it's been proven (whatever that means, normally peer reviewed and some evidence in the form of usage/citations). However in this case the distribution is well-known, fits in nicely with our distributions framework, and the results from the EM algorithm can be verified because they should roughly match the generic implementation of the `var` method. So as long as the code is relatively concise, an implementation based on your paper is fine here I think.

Nice paper by the way.
@rgommers,  Thanks for your encouragement! we'll start working on the implementation. 
@rgommers, I've been hacking away on my forked repo: 
I think the code is working as intended and I'm thinking about unit tests. 
This came up as I poked around the api:

Lucass-MacBook:tests rlucas$ python -V
Python 3.6.6 :: Anaconda, Inc.
Lucass-MacBook:tests rlucas$ python
Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 11:07:29)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from scipy.stats import cauchy
>>> cauchy(0).mean()
nan
>>> cauchy(0).expect()
0.0

The issue would arise w/Yule-Simon as well if the Yule-Simon parameter \alpha is such that 0 < \alpha < 1 where the mean in infinite. The expect() method defaults to numeric integration w/o checking the result of mean()-if there is an implementation that overrides the expect(). Do you think this expected behavior or should I open an issue? 

I've been trying to send this email to the scipy-dev@python.org email list but my email keeps getting returned as undeliverable. 
> I've been trying to send this email to the scipy-dev@python.org email list but my email keeps getting returned as undeliverable.

Did you sign up successfully? Can be done at https://mail.python.org/mailman/listinfo/scipy-dev. I'm not aware of issues with that recently.
> cauchy(0).mean()
> nan

Well, the mean for cauchy is undefined (see e.g. https://en.wikipedia.org/wiki/Cauchy_distribution). The `0` in there does nothing (it's equivalent to `cauchy(loc=0)` which is the default value for `loc`). So that looks right to me.

And the Cauchy distribution is symmetric, so `0` looks also like the right answer to me for `cauchy.expect()`.

Am I missing something?


The default function for the `expect` method is `x`, so from the mathematical point of view, `distr.expect()` should return the same as `distr.mean()`, and that should be `nan` for the Cauchy distribution.
@WarrenWeckesser, yes, also any r.v. with an mean() of inf as return value has the same issue. 
it sounds like this is an issue.  I'll open an issue and put this context in the issue. 

I was confused about whether there was a specific use case where a numeric value was needed even if the expected value is inf or nan. After reading your reply, I'm clear that this is a conceptual issue. If a numeric value is needed another method (not mean() or expect() ) should be used. 
Been working on this over here:
https://github.com/rlucas7/scipy/tree/yule_simon_dev

given the issue with the infinite variances I've taken the `.stats()` method out of the distribution and inherited the default. I've got unit tests and documentation (including doc strings).  

I think I'm ready for the pull request, anything obvious I'm missing? 


Resolving patch:
diff --git a/scipy/stats/__init__.py b/scipy/stats/__init__.py
--- a/scipy/stats/__init__.py
+++ b/scipy/stats/__init__.py
@@ -159,6 +159,7 @@
    randint           -- Discrete Uniform
    skellam           -- Skellam
    zipf              -- Zipf
+   yulesimon         -- Yule-Simon
 
 An overview of statistical functions is given below.
 Several of these functions have a similar version in
diff --git a/scipy/stats/_discrete_distns.py b/scipy/stats/_discrete_distns.py
--- a/scipy/stats/_discrete_distns.py
+++ b/scipy/stats/_discrete_distns.py
@@ -887,6 +887,84 @@ def _stats(self, mu1, mu2):
 skellam = skellam_gen(a=-np.inf, name="skellam", longname='A Skellam')
 
 
+class yulesimon_gen(rv_discrete):
+    r"""A Yule-Simon discrete random variable.
+
+    %(before_notes)s
+
+    Notes
+    -----
+
+    The probability mass function for the `yulesimon` is:
+
+    .. math::
+
+        f(k) =  \alpha B(k, \alpha+1)
+
+    for :math:`k=1,2,3,...`, where :math:`\alpha>0`.
+    Here :math:`B` refers to the `scipy.special.beta` function.
+
+    The sampling of random variates is based on pg 553, Section 6.3 of [1]_.
+    Our notation maps to the referenced logic via :math:`\alpha=a-1`.
+
+    For details see the wikipedia entry [2]_.
+
+    References
+    ----------
+    .. [1] Devroye, Luc. "Non-uniform Random Variate Generation",
+         (1986) Springer, New York.
+
+    .. [2] https://en.wikipedia.org/wiki/Yule-Simon_distribution
+
+    %(after_notes)s
+
+    %(example)s
+
+    """
+    def _rvs(self, alpha):
+        E1 = self._random_state.standard_exponential(self._size)
+        E2 = self._random_state.standard_exponential(self._size)
+        ans = ceil(-E1 / log1p(-exp(-E2 / alpha)))
+        return ans
+
+    def _pmf(self, x, alpha):
+        return alpha * special.beta(x, alpha + 1)
+
+    def _argcheck(self, alpha):
+        return (alpha > 0)
+
+    def _logpmf(self, x, alpha):
+        return log(alpha) + special.betaln(x, alpha + 1)
+
+    def _cdf(self, x, alpha):
+        return 1 - x * special.beta(x, alpha + 1)
+
+    def _sf(self, x, alpha):
+        return x * special.beta(x, alpha + 1)
+
+    def _logsf(self, x, alpha):
+        return log(x) + special.betaln(x, alpha + 1)
+
+    def _stats(self, alpha):
+        mu = np.where(alpha <= 1, np.inf, alpha / (alpha - 1))
+        mu2 = np.where(alpha > 2,
+                alpha**2 / ((alpha - 2.0) * (alpha - 1)**2),
+                np.inf)
+        mu2 = np.where(alpha <= 1, np.nan, mu2)
+        g1 = np.where(alpha > 3,
+                sqrt(alpha - 2) * (alpha + 1)**2 / (alpha * (alpha - 3)),
+                np.inf)
+        g1 = np.where(alpha <= 2, np.nan, g1)
+        g2 = np.where(alpha > 4,
+                (alpha + 3) + (alpha**3 - 49 * alpha - 22) / (alpha *
+                        (alpha - 4) * (alpha - 3)), np.inf)
+        g2 = np.where(alpha <= 2, np.nan, g2)
+        return mu, mu2, g1, g2
+
+
+yulesimon = yulesimon_gen(name='yulesimon', a=1)
+
+
 # Collect names of classes and objects in this module.
 pairs = list(globals().items())
 _distn_names, _distn_gen_names = get_distribution_names(pairs, rv_discrete)
diff --git a/scipy/stats/_distr_params.py b/scipy/stats/_distr_params.py
--- a/scipy/stats/_distr_params.py
+++ b/scipy/stats/_distr_params.py
@@ -125,5 +125,6 @@
     ['poisson', (0.6,)],
     ['randint', (7, 31)],
     ['skellam', (15, 8)],
-    ['zipf', (6.5,)]
+    ['zipf', (6.5,)],
+    ['yulesimon',(11.0,)]
 ]


Test patch:
diff --git a/scipy/stats/tests/test_discrete_basic.py b/scipy/stats/tests/test_discrete_basic.py
--- a/scipy/stats/tests/test_discrete_basic.py
+++ b/scipy/stats/tests/test_discrete_basic.py
@@ -80,7 +80,7 @@ def test_moments(distname, arg):
     check_mean_expect(distfn, arg, m, distname)
     check_var_expect(distfn, arg, m, v, distname)
     check_skew_expect(distfn, arg, m, v, s, distname)
-    if distname not in ['zipf']:
+    if distname not in ['zipf', 'yulesimon']:
         check_kurt_expect(distfn, arg, m, v, k, distname)
 
     # frozen distr moments
@@ -100,7 +100,7 @@ def test_rvs_broadcast(dist, shape_args):
     # implementation detail of the distribution, not a requirement.  If
     # the implementation the rvs() method of a distribution changes, this
     # test might also have to be changed.
-    shape_only = dist in ['skellam']
+    shape_only = dist in ['skellam', 'yulesimon']
 
     try:
         distfunc = getattr(stats, dist)