Repository: statsmodels/statsmodels

Issue:
ENH: Implementation of the robust Qn scale estimator
- [X] related to #6798
- [X] tests added / passed. tested against robustbase from R and also against a naive implementation
- [X] code/documentation is well formatted.  
- [X] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

This is an implementation of the Qn robust estimator of scale. The estimator was first proposed in Rousseeuw and Croux (1993), Alternatives to the Median Absolute Deviation. The main appeal of the estimator is that it has maximal breakdown point (50%), a high asymptotic efficiency at the normal distribution (~80%) and that it does not require an auxiliary location estimator. The implementation here follows the one described in Croux and Rousseeuw (1992), Time-Efficient Algorithms for Two Highly Robust Estimators of Scale. The implementation is tested against a naive (and slow) implementation of the estimator and also against the implementation available in the robustbase R package for a couple of datasets. The implementation in robustbase also includes some finite sample corrections, which were derived (according to the documentation) from a simulation study that is not referenced. I did not include these finite sample corrections in my implementation, since I am not sure of how they were constructed, and in my opinion, they make the estimator less transparent.
<details>


**Notes**:

* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in master and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/master -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -159,6 +159,7 @@
     _kim_smoother={'source': 'statsmodels/tsa/regime_switching/_kim_smoother.pyx.in'},  # noqa: E501
     _arma_innovations={'source': 'statsmodels/tsa/innovations/_arma_innovations.pyx.in'},  # noqa: E501
     linbin={'source': 'statsmodels/nonparametric/linbin.pyx'},
+    _qn={'source': 'statsmodels/robust/_qn.pyx'},
     _smoothers_lowess={'source': 'statsmodels/nonparametric/_smoothers_lowess.pyx'},  # noqa: E501
     kalman_loglike={'source': 'statsmodels/tsa/kalmanf/kalman_loglike.pyx',
                     'include_dirs': ['statsmodels/src'],
diff --git a/statsmodels/robust/scale.py b/statsmodels/robust/scale.py
--- a/statsmodels/robust/scale.py
+++ b/statsmodels/robust/scale.py
@@ -7,12 +7,16 @@
 
 R Venables, B Ripley. 'Modern Applied Statistics in S'
     Springer, New York, 2002.
+
+C Croux, PJ Rousseeuw, 'Time-efficient algorithms for two highly robust
+estimators of scale' Computational statistics. Physica, Heidelberg, 1992.
 """
 import numpy as np
 from scipy.stats import norm as Gaussian
 from . import norms
 from statsmodels.tools import tools
 from statsmodels.tools.validation import array_like, float_like
+from ._qn import _qn
 
 
 def mad(a, c=Gaussian.ppf(3/4.), axis=0, center=np.median):
@@ -81,6 +85,79 @@ def iqr(a, c=Gaussian.ppf(3/4) - Gaussian.ppf(1/4), axis=0):
         return np.squeeze(np.diff(quantiles, axis=0) / c)
 
 
+def qn_scale(a, c=1 / (np.sqrt(2) * Gaussian.ppf(5 / 8)), axis=0):
+    """
+    Computes the Qn robust estimator of scale
+
+    The Qn scale estimator is a more efficient alternative to the MAD.
+    The Qn scale estimator of an array a of length n is defined as
+    c * {abs(a[i] - a[j]): i<j}_(k), for k equal to [n/2] + 1 choose 2. Thus,
+    the Qn estimator is the k-th order statistic of the absolute differences
+    of the array. The optional constant is used to normalize the estimate
+    as explained below. The implementation follows the algorithm described
+    in Croux and Rousseeuw (1992).
+
+    Parameters
+    ----------
+    a : array_like
+        Input array.
+    c : float, optional
+        The normalization constant. The default value is used to get consistent
+        estimates of the standard deviation at the normal distribution.
+    axis : int, optional
+        The default is 0.
+
+    Returns
+    -------
+    {float, ndarray}
+        The Qn robust estimator of scale
+    """
+    a = array_like(a, 'a', ndim=None, dtype=np.float64, contiguous=True,
+                   order='C')
+    c = float_like(c, 'c')
+    if a.ndim == 0:
+        raise ValueError("a should have at least one dimension")
+    elif a.size == 0:
+        return np.nan
+    else:
+        out = np.apply_along_axis(_qn, axis=axis, arr=a, c=c)
+        if out.ndim == 0:
+            return float(out)
+        return out
+
+
+def _qn_naive(a, c=1 / (np.sqrt(2) * Gaussian.ppf(5 / 8))):
+    """
+    A naive implementation of the Qn robust estimator of scale, used solely
+    to test the faster, more involved one
+
+    Parameters
+    ----------
+    a : array_like
+        Input array.
+    c : float, optional
+        The normalization constant, used to get consistent estimates of the
+        standard deviation at the normal distribution.  Defined as
+        1/(np.sqrt(2) * scipy.stats.norm.ppf(5/8)), which is 2.219144.
+
+    Returns
+    -------
+    The Qn robust estimator of scale
+    """
+    a = np.squeeze(a)
+    n = a.shape[0]
+    if a.size == 0:
+        return np.nan
+    else:
+        h = int(n // 2 + 1)
+        k = int(h * (h - 1) / 2)
+        idx = np.triu_indices(n, k=1)
+        diffs = np.abs(a[idx[0]] - a[idx[1]])
+        output = np.partition(diffs, kth=k - 1)[k - 1]
+        output = c * output
+        return output
+
+
 class Huber(object):
     """
     Huber's proposal 2 for estimating location and scale jointly.


Test patch:
diff --git a/statsmodels/robust/tests/test_scale.py b/statsmodels/robust/tests/test_scale.py
--- a/statsmodels/robust/tests/test_scale.py
+++ b/statsmodels/robust/tests/test_scale.py
@@ -9,6 +9,7 @@
 # Example from Section 5.5, Venables & Ripley (2002)
 
 import statsmodels.robust.scale as scale
+import statsmodels.api as sm
 
 DECIMAL = 4
 # TODO: Can replicate these tests using stackloss data and R if this
@@ -35,6 +36,9 @@ def test_mad(self):
     def test_iqr(self):
         assert_almost_equal(scale.iqr(self.chem), 0.68570, DECIMAL)
 
+    def test_qn(self):
+        assert_almost_equal(scale.qn_scale(self.chem), 0.73231, DECIMAL)
+
     def test_huber_scale(self):
         assert_almost_equal(scale.huber(self.chem)[0], 3.20549, DECIMAL)
 
@@ -147,6 +151,69 @@ def test_axisneg1(self):
         assert_equal(m.shape, (40, 10))
 
 
+class TestQn(object):
+    @classmethod
+    def setup_class(cls):
+        np.random.seed(54321)
+        cls.normal = standard_normal(size=40)
+        cls.range = np.arange(0, 40)
+        cls.exponential = np.random.exponential(size=40)
+        cls.stackloss = sm.datasets.stackloss.load_pandas().data
+        cls.sunspot = sm.datasets.sunspots.load_pandas().data.SUNACTIVITY
+
+    def test_qn_naive(self):
+        assert_almost_equal(scale.qn_scale(self.normal),
+                            scale._qn_naive(self.normal), DECIMAL)
+        assert_almost_equal(scale.qn_scale(self.range),
+                            scale._qn_naive(self.range), DECIMAL)
+        assert_almost_equal(scale.qn_scale(self.exponential),
+                            scale._qn_naive(self.exponential), DECIMAL)
+
+    def test_qn_robustbase(self):
+        # from R's robustbase with finite.corr = FALSE
+        assert_almost_equal(scale.qn_scale(self.range), 13.3148, DECIMAL)
+        assert_almost_equal(scale.qn_scale(self.stackloss),
+                            np.array([8.87656, 8.87656, 2.21914, 4.43828]),
+                            DECIMAL)
+        # sunspot.year from datasets in R only goes up to 289
+        assert_almost_equal(scale.qn_scale(self.sunspot[0:289]), 33.50901,
+                            DECIMAL)
+
+    def test_qn_empty(self):
+        empty = np.empty(0)
+        assert np.isnan(scale.qn_scale(empty))
+        empty = np.empty((10, 100, 0))
+        assert_equal(scale.qn_scale(empty, axis=1), np.empty((10, 0)))
+        empty = np.empty((100, 100, 0, 0))
+        assert_equal(scale.qn_scale(empty, axis=-1), np.empty((100, 100, 0)))
+        empty = np.empty(shape=())
+        with pytest.raises(ValueError):
+            scale.iqr(empty)
+
+
+class TestQnAxes(object):
+    @classmethod
+    def setup_class(cls):
+        np.random.seed(54321)
+        cls.X = standard_normal((40, 10, 30))
+
+    def test_axis0(self):
+        m = scale.qn_scale(self.X, axis=0)
+        assert_equal(m.shape, (10, 30))
+
+    def test_axis1(self):
+        m = scale.qn_scale(self.X, axis=1)
+        assert_equal(m.shape, (40, 30))
+
+    def test_axis2(self):
+        m = scale.qn_scale(self.X, axis=2)
+        assert_equal(m.shape, (40, 10))
+
+    def test_axisneg1(self):
+        m = scale.qn_scale(self.X, axis=-1)
+        assert_equal(m.shape, (40, 10))
+
+
 class TestHuber(object):
     @classmethod
     def setup_class(cls):