Repository: statsmodels/statsmodels

Issue:
ENH: simultaneous confidence intervals for multinomial proportions
I've been looking around for a pythonic way of computing this, and have found nothing.

The [MultinomialCI](https://cran.r-project.org/web/packages/MultinomialCI/index.html) R package implements the method from [Sison & Glaz (1995), Simultaneous confidence intervals and sample size determination for multinomial proportions](http://tx.liberal.ntu.edu.tw/~purplewoo/Literature/!Methodology/!Distribution_SampleSize/SimultConfidIntervJASA.pdf), which seems to be the reference (for asymptotic confidence intervals) in the literature.

Would a port of this package in `statsmodels.stats.proportion.multinomial_proportion_confint()` interest the statsmodels team?



Discussion:
A direct port will not be useful to us because of the licence incompatibility.

Based on a brief look at the Sison and Glaz article it might not be too difficult to implement it from scratch.

I have never looked at simultaneous confidence intervals in this area, so right now I don't have any background information for this. It sounds interesting and like a good addition.

All right. I'll try and do an original implementation based on Sison & Glaz then.

The R package linked to above is in fact "an R translation of the SAS code implemented by May and Johnson in their paper: [May, W.L. and W.D. Johnson. Constructing two-sided simultaneous confidence intervals for multinomial proportions for small counts in a large number of cells](https://www.jstatsoft.org/index.php/jss/article/view/v005i06/Sisonfinal.pdf)". [May & Johnson (1997)](https://www.researchgate.net/profile/Warren_May/publication/13990121_A_SASR_macro_for_constructing_simultaneous_confidence_intervals_for_multinomial_proportions/links/5405ea520cf2c48563b1f81d.pdf) also implements other methods (for e.g. lower number of cells) for multinomial CIs.

I can't find any licensing information on those two papers, but it seems to me rather standard to adapt code from published papers as long as they're cited. Would that be okay license-wise?

papers and published articles are still copyrighted. (usual disclaimer: I'm not a lawyer.)
When I started with scipy around 8 years ago, the answer was negative. Without a compatible license statement or explicit permission by the authors it is not allowed to translate the code and redistribute it under a different license. Some journals that publish more software have a, most of the time restrictive, default copyright statement.

I'm personally not completely restrictive anymore and sometimes check part of the code that is in an article when something is not clear from the main text of the article.

---

I went over some of the papers after this issue was opened.
May and Johson 1997 looks relatively straightforward. In section for they list the earlier confidence intervals which are base mostly on roots of quadratic functions.

Sison and Glaz have a few tricky part, but the description in the later May Johnson sounded ok, and explained where they deviated in the implementation from the original Sison/Glaz paper. (I don't see that I have a printed copy, so I guess I only skimmed and read parts of the second May Johnson paper in pdf.)

I also found a few recent articles that describe fast exact algorithms for multinomial box probabilities, but they looked a lot more computation and more for someone that is interested in implementing numerical algorithms (like scipy special).

If you want to get started with this, I think it would be best to start with the quadratic equations in May Johnson 1997 and some test cases. I can look into Sison/Glaz and the second May/Johnson again. (My guess would be that searching for c in Sison/Glaz might be faster when starting with something like Quesenberry and Hurst intervals (approximately because the shape of the confidence region is different)).


Resolving patch:
diff --git a/statsmodels/stats/proportion.py b/statsmodels/stats/proportion.py
--- a/statsmodels/stats/proportion.py
+++ b/statsmodels/stats/proportion.py
@@ -120,6 +120,190 @@ def func(qi):
         raise NotImplementedError('method "%s" is not available' % method)
     return ci_low, ci_upp
 
+def multinomial_proportions_confint(counts, alpha=0.05, method='goodman'):
+    '''Confidence intervals for multinomial proportions.
+
+    Parameters
+    ----------
+    counts : array_like of int, 1-D
+        Number of observations in each category.
+    alpha : float in (0, 1), optional
+        Significance level, defaults to 0.05.
+    method : {'goodman', 'sison-glaz'}, optional
+        Method to use to compute the confidence intervals; available methods
+        are:
+
+         - `goodman`: based on a chi-squared approximation, valid if all
+           values in `counts` are greater or equal to 5 [2]_
+         - `sison-glaz`: less conservative than `goodman`, but only valid if
+           `counts` has 7 or more categories (``len(counts) >= 7``) [3]_
+
+    Returns
+    -------
+    confint : ndarray, 2-D
+        Array of [lower, upper] confidence levels for each category, such that
+        overall coverage is (approximately) `1-alpha`.
+
+    Raises
+    ------
+    ValueError
+        If `alpha` is not in `(0, 1)` (bounds excluded), or if the values in
+        `counts` are not all positive or null.
+    NotImplementedError
+        If `method` is not kown.
+    Exception
+        When ``method == 'sison-glaz'``, if for some reason `c` cannot be
+        computed; this signals a bug and should be reported.
+
+    Notes
+    -----
+    The `goodman` method [2]_ is based on approximating a statistic based on
+    the multinomial as a chi-squared random variable. The usual recommendation
+    is that this is valid if all the values in `counts` are greater than or
+    equal to 5. There is no condition on the number of categories for this
+    method.
+
+    The `sison-glaz` method [3]_ approximates the multinomial probabilities,
+    and evaluates that with a maximum-likelihood estimator. The first
+    approximation is an Edgeworth expansion that converges when the number of
+    categories goes to infinity, and the maximum-likelihood estimator converges
+    when the number of observations (``sum(counts)``) goes to infinity. In
+    their paper, Sison & Glaz demo their method with at least 7 categories, so
+    ``len(counts) >= 7`` with all values in `counts` at or above 5 can be used
+    as a rule of thumb for the validity of this method. This method is less
+    conservative than the `goodman` method (i.e. it will yield confidence
+    intervals closer to the desired significance level), but produces
+    confidence intervals of uniform width over all categories (except when the
+    intervals reach 0 or 1, in which case they are truncated), which makes it
+    most useful when proportions are of similar magnitude.
+
+    Aside from the original sources ([1]_, [2]_, and [3]_), the implementation
+    uses the formulas (though not the code) presented in [4]_ and [5]_.
+
+    References
+    ----------
+    .. [1] Levin, Bruce, "A representation for multinomial cumulative
+           distribution functions," The Annals of Statistics, Vol. 9, No. 5,
+           1981, pp. 1123-1126.
+
+    .. [2] Goodman, L.A., "On simultaneous confidence intervals for multinomial
+           proportions," Technometrics, Vol. 7, No. 2, 1965, pp. 247-254.
+
+    .. [3] Sison, Cristina P., and Joseph Glaz, "Simultaneous Confidence
+           Intervals and Sample Size Determination for Multinomial
+           Proportions," Journal of the American Statistical Association,
+           Vol. 90, No. 429, 1995, pp. 366-369.
+
+    .. [4] May, Warren L., and William D. Johnson, "A SAS® macro for
+           constructing simultaneous confidence intervals  for multinomial
+           proportions," Computer methods and programs in Biomedicine, Vol. 53,
+           No. 3, 1997, pp. 153-162.
+
+    .. [5] May, Warren L., and William D. Johnson, "Constructing two-sided
+           simultaneous confidence intervals for multinomial proportions for
+           small counts in a large number of cells," Journal of Statistical
+           Software, Vol. 5, No. 6, 2000, pp. 1-24.
+    '''
+    if alpha <= 0 or alpha >= 1:
+        raise ValueError('alpha must be in (0, 1), bounds excluded')
+    counts = np.array(counts, dtype=np.float)
+    if (counts < 0).any():
+        raise ValueError('counts must be > 0')
+    elif (counts == 0).any():
+        # This case is separated from (counts < 0).any() so we can give clearer
+        # advice.
+        raise ValueError('The behavior is not undefined for 0-valued counts; '
+                         'try adding 0.5 to your 0-valued counts to run this')
+    n = counts.sum()
+    k = len(counts)
+    proportions = counts / n
+    if method == 'goodman':
+        chi2 = stats.chi2.ppf(1 - alpha / k, 1)
+        delta = chi2 ** 2 + (4 * n * proportions * chi2 * (1 - proportions))
+        region = ((2 * n * proportions + chi2 +
+                   np.array([- np.sqrt(delta), np.sqrt(delta)])) /
+                  (2 * (chi2 + n))).T
+    elif method[:5] == 'sison':  # We accept any name starting with 'sison'
+        # Define a few functions we'll use a lot.
+        def poisson_interval(interval, p):
+            """Compute P(b <= Z <= a) where Z ~ Poisson(p) and
+            `interval = (b, a)`."""
+            b, a = interval
+            return stats.poisson.cdf(a, p) - stats.poisson.cdf(b - 1, p)
+
+        def truncated_poisson_factorial_moment(interval, r, p):
+            """Compute mu_r, the r-th factorial moment of a poisson random
+            variable of parameter `p` truncated to `interval = (b, a)`."""
+            b, a = interval
+            return p ** r * (1 - ((poisson_interval((a - r + 1, a), p) -
+                                   poisson_interval((b - r, b - 1), p)) /
+                                  poisson_interval((b, a), p)))
+
+        def edgeworth(intervals):
+            """Compute the Edgeworth expansion term of Sison & Glaz's formula
+            (1) (approximated probability for multinomial proportions in a
+            given box)."""
+            # Compute means and central moments of the truncated poisson
+            # variables.
+            mu_r1, mu_r2, mu_r3, mu_r4 = [
+                np.array([truncated_poisson_factorial_moment(interval, r, p)
+                          for (interval, p) in zip(intervals, counts)])
+                for r in range(1, 5)
+            ]
+            mu = mu_r1
+            mu2 = mu_r2 + mu - mu ** 2
+            mu3 = mu_r3 + mu_r2 * (3 - 3 * mu) + mu - 3 * mu ** 2 + 2 * mu ** 3
+            mu4 = (mu_r4 + mu_r3 * (6 - 4 * mu) +
+                   mu_r2 * (7 - 12 * mu + 6 * mu ** 2) +
+                   mu - 4 * mu ** 2 + 6 * mu ** 3 - 3 * mu ** 4)
+
+            # Compute expansion factors, gamma_1 and gamma_2.
+            g1 = mu3.sum() / mu2.sum() ** 1.5
+            g2 = (mu4.sum() - 3 * (mu2 ** 2).sum()) / mu2.sum() ** 2
+
+            # Compute the expansion itself.
+            x = (n - mu.sum()) / np.sqrt(mu2.sum())
+            phi = np.exp(- x ** 2 / 2) / np.sqrt(2 * np.pi)
+            H3 = x ** 3 - 3 * x
+            H4 = x ** 4 - 6 * x ** 2 + 3
+            H6 = x ** 6 - 15 * x ** 4 + 45 * x ** 2 - 15
+            f = phi * (1 + g1 * H3 / 6 + g2 * H4 / 24 + g1 ** 2 * H6 / 72)
+            return f / np.sqrt(mu2.sum())
+
+        def approximated_multinomial_interval(intervals):
+            """Compute approximated probability for Multinomial(n, proportions)
+            to be in `intervals` (Sison & Glaz's formula (1))."""
+            return np.exp(
+                np.sum(np.log([poisson_interval(interval, p)
+                               for (interval, p) in zip(intervals, counts)])) +
+                np.log(edgeworth(intervals)) -
+                np.log(stats.poisson.pmf(n, n))
+            )
+
+        def nu(c):
+            """Compute interval coverage for a given `c` (Sison & Glaz's
+            formula (7))."""
+            return approximated_multinomial_interval(
+                [(count - c, count + c) for count in counts])
+
+        # Find the value of `c` that will give us the confidence intervals
+        # (solving nu(c) <= 1 - alpha < nu(c + 1).
+        c = 1.0
+        while not (nu(c) <= (1 - alpha) < nu(c + 1)):
+            if c > n:
+                raise Exception("Couldn't find a value for `c` that "
+                                "solves nu(c) <= 1 - alpha < nu(c + 1)")
+            c += 1
+
+        # Compute gamma and the corresponding confidence intervals.
+        g = (1 - alpha - nu(c)) / (nu(c + 1) - nu(c))
+        ci_lower = np.maximum(proportions - c / n, 0)
+        ci_upper = np.minimum(proportions + (c + 2 * g) / n, 1)
+        region = np.array([ci_lower, ci_upper]).T
+    else:
+        raise NotImplementedError('method "%s" is not available' % method)
+    return region
+
 def samplesize_confint_proportion(proportion, half_length, alpha=0.05,
                                   method='normal'):
     '''find sample size to get desired confidence interval length


Test patch:
diff --git a/statsmodels/stats/tests/results/results_multinomial_proportions.py b/statsmodels/stats/tests/results/results_multinomial_proportions.py
new file mode 100644
--- /dev/null
+++ b/statsmodels/stats/tests/results/results_multinomial_proportions.py
@@ -0,0 +1,71 @@
+# -*- coding: utf-8 -*-
+"""Test values for multinomial_proportion_confint.
+
+Author: Sébastien Lerique
+"""
+
+
+import collections
+import numpy as np
+
+
+class Holder(object):
+    pass
+
+
+res_multinomial = collections.defaultdict(Holder)
+
+# The following examples come from the Sison & Glaz paper, and the values were
+# computed using the R MultinomialCI package.
+
+# Floating-point arithmetic errors get blown up in the Edgeworth expansion
+# (starting in g1 and g2, but mostly when computing f, because of the
+# polynomials), which explains why we only obtain a precision of 4 decimals
+# when comparing to values computed in R.
+
+# We test with any method name that starts with 'sison', as that is the
+# criterion.
+res_multinomial[('sison', 'Sison-Glaz example 1')]\
+    .proportions = [56, 72, 73, 59, 62, 87, 58]
+res_multinomial[('sison', 'Sison-Glaz example 1')]\
+    .cis = np.array([[.07922912, .1643361], [.11349036, .1985973],
+                     [.11563169, .2007386], [.08565310, .1707601],
+                     [.09207709, .1771840], [.14561028, .2307172],
+                     [.08351178, .1686187]])
+res_multinomial[('sison', 'Sison-Glaz example 1')].precision = 4
+
+res_multinomial[('sisonandglaz', 'Sison-Glaz example 2')].proportions = [5] * 50
+res_multinomial[('sisonandglaz', 'Sison-Glaz example 2')]\
+    .cis = [0, .05304026] * np.ones((50, 2))
+res_multinomial[('sisonandglaz', 'Sison-Glaz example 2')].precision = 4
+
+res_multinomial[('sison-whatever', 'Sison-Glaz example 3')]\
+    .proportions = [1] * 10 + [12] * 10 + [5] * 10 + [3] * 10 + [4] * 10
+res_multinomial[('sison-whatever', 'Sison-Glaz example 3')].cis = np.concatenate([
+    [0, .04120118] * np.ones((10, 2)), [.012, .08520118] * np.ones((10, 2)),
+    [0, .05720118] * np.ones((10, 2)), [0, .04920118] * np.ones((10, 2)),
+    [0, .05320118] * np.ones((10, 2))
+])
+res_multinomial[('sison-whatever', 'Sison-Glaz example 3')].precision = 4
+
+# The examples from the Sison & Glaz paper only include 3 decimals.
+res_multinomial[('goodman', 'Sison-Glaz example 1')]\
+    .proportions = [56, 72, 73, 59, 62, 87, 58]
+res_multinomial[('goodman', 'Sison-Glaz example 1')]\
+    .cis = np.array([[.085, .166], [.115, .204], [.116, .207], [.091, .173],
+                     [.096, .181], [.143, .239], [.089, .171]])
+res_multinomial[('goodman', 'Sison-Glaz example 1')].precision = 3
+
+res_multinomial[('goodman', 'Sison-Glaz example 2')].proportions = [5] * 50
+res_multinomial[('goodman', 'Sison-Glaz example 2')]\
+    .cis = [.005, .075] * np.ones((50, 2))
+res_multinomial[('goodman', 'Sison-Glaz example 2')].precision = 3
+
+res_multinomial[('goodman', 'Sison-Glaz example 3')]\
+    .proportions = [1] * 10 + [12] * 10 + [5] * 10 + [3] * 10 + [4] * 10
+res_multinomial[('goodman', 'Sison-Glaz example 3')].cis = np.concatenate([
+    [0, .049] * np.ones((10, 2)), [.019, .114] * np.ones((10, 2)),
+    [.005, .075] * np.ones((10, 2)), [.002, .062] * np.ones((10, 2)),
+    [.004, .069] * np.ones((10, 2))
+])
+res_multinomial[('goodman', 'Sison-Glaz example 3')].precision = 3
diff --git a/statsmodels/stats/tests/test_proportion.py b/statsmodels/stats/tests/test_proportion.py
--- a/statsmodels/stats/tests/test_proportion.py
+++ b/statsmodels/stats/tests/test_proportion.py
@@ -11,7 +11,8 @@
 from numpy.testing import (assert_almost_equal, assert_equal, assert_array_less,
                            assert_raises)
 
-from statsmodels.stats.proportion import proportion_confint
+from statsmodels.stats.proportion import (proportion_confint,
+                                          multinomial_proportions_confint)
 import statsmodels.stats.proportion as smprop
 from statsmodels.tools.sm_exceptions import HypothesisTestWarning
 
@@ -55,6 +56,31 @@ def test_proportion_effect_size():
     es = smprop.proportion_effectsize(0.5, 0.4)
     assert_almost_equal(es, 0.2013579207903309, decimal=13)
 
+def test_confint_multinomial_proportions():
+    from .results.results_multinomial_proportions import res_multinomial
+
+    for ((method, description), values) in res_multinomial.items():
+        cis = multinomial_proportions_confint(values.proportions, 0.05,
+                                              method=method)
+        assert_almost_equal(
+            values.cis, cis, decimal=values.precision,
+            err_msg='"%s" method, %s' % (method, description))
+
+def test_multinomial_proportions_errors():
+    # Out-of-bounds values for alpha raise a ValueError
+    for alpha in [-.1, 0, 1, 1.1]:
+        assert_raises(ValueError, multinomial_proportions_confint,
+                      [5] * 50, alpha=alpha)
+    # Null values in `counts` are okay, but negative or zero values are not.
+    assert_raises(ValueError, multinomial_proportions_confint,
+                  np.arange(50))
+    assert_raises(ValueError, multinomial_proportions_confint,
+                  np.arange(50) - 1)
+    # Any unknown method is reported.
+    for method in ['unknown_method', 'sisok_method', 'unknown-glaz']:
+        assert_raises(NotImplementedError, multinomial_proportions_confint,
+                      [5] * 50, method=method)
+
 class CheckProportionMixin(object):
     def test_proptest(self):
         # equality of k-samples