Repository: statsmodels/statsmodels

Issue:
BUG: error on passing axis=None to mad
#### Describe the bug

Setting axis=None in the mad function raises "TypeError: '<' not supported between instances of 'NoneType' and 'int'". According to the docs, axis=None is supported. The docs don't say so, but I guess the expected behaviour would be to have the mad applied on the flattened input array (like what np.sum does for axis=None for example). I'd be happy to write a fix for this.

#### Code Sample


```python
import numpy as np
from statsmodels.robust.scale import mad
a = np.array([[0, 1, 2],[2, 3, 2]])

mad(a=a, axis=None) # raises TypeError

mad(a.flatten()) # this is what I would expect the output to be
```
<details>

**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `master`. If your problem has been fixed in an unreleased version, you might be able to use `master` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the master branch of this repository? It helps the limited resources if we know problems exist in the current master so that they do not need to check whether the code sample produces a bug in the next release.

</details>



#### Expected Output

The mad function applied to the flattened input array.
#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

INSTALLED VERSIONS
------------------
Python: 3.8.2.final.0
OS: Linux 5.4.0-45-generic #49-Ubuntu SMP Wed Aug 26 13:38:52 UTC 2020 x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

statsmodels
===========
Installed: v0.13.0.dev0+18.g3d6a49310 (/home/esmucler/Packages/statsmodels/statsmodels)

Required Dependencies
=====================
cython: 0.29.21 (/home/esmucler/Packages/statsmodels/venv/lib/python3.8/site-packages/Cython)
numpy: 1.19.1 (/home/esmucler/Packages/statsmodels/venv/lib/python3.8/site-packages/numpy)
scipy: 1.5.2 (/home/esmucler/Packages/statsmodels/venv/lib/python3.8/site-packages/scipy)
pandas: 1.1.0 (/home/esmucler/Packages/statsmodels/venv/lib/python3.8/site-packages/pandas)
    dateutil: 2.8.1 (/home/esmucler/Packages/statsmodels/venv/lib/python3.8/site-packages/dateutil)
patsy: 0.5.1 (/home/esmucler/Packages/statsmodels/venv/lib/python3.8/site-packages/patsy)

Optional Dependencies
=====================
matplotlib: 3.3.1 (/home/esmucler/Packages/statsmodels/venv/lib/python3.8/site-packages/matplotlib)
    backend: agg 
cvxopt: Not installed
joblib: Not installed

Developer Tools
================
IPython: 7.17.0 (/home/esmucler/Packages/statsmodels/venv/lib/python3.8/site-packages/IPython)
    jinja2: 2.11.2 (/home/esmucler/Packages/statsmodels/venv/lib/python3.8/site-packages/jinja2)
sphinx: 3.2.1 (/home/esmucler/Packages/statsmodels/venv/lib/python3.8/site-packages/sphinx)
    pygments: 2.6.1 (/home/esmucler/Packages/statsmodels/venv/lib/python3.8/site-packages/pygments)
pytest: 6.0.1 (/home/esmucler/Packages/statsmodels/venv/lib/python3.8/site-packages/pytest)
virtualenv: Not installed

</details>



Discussion:

Resolving patch:
diff --git a/statsmodels/robust/scale.py b/statsmodels/robust/scale.py
--- a/statsmodels/robust/scale.py
+++ b/statsmodels/robust/scale.py
@@ -13,14 +13,15 @@
 """
 import numpy as np
 from scipy.stats import norm as Gaussian
-from . import norms
+
 from statsmodels.tools import tools
 from statsmodels.tools.validation import array_like, float_like
+
+from . import norms
 from ._qn import _qn
 
 
-def mad(a, c=Gaussian.ppf(3/4.), axis=0, center=np.median):
-    # c \approx .6745
+def mad(a, c=Gaussian.ppf(3 / 4.0), axis=0, center=np.median):
     """
     The Median Absolute Deviation along given axis of an array
 
@@ -30,7 +31,7 @@ def mad(a, c=Gaussian.ppf(3/4.), axis=0, center=np.median):
         Input array.
     c : float, optional
         The normalization constant.  Defined as scipy.stats.norm.ppf(3/4.),
-        which is approximately .6745.
+        which is approximately 0.6745.
     axis : int, optional
         The default is 0. Can also be None.
     center : callable or float
@@ -43,19 +44,22 @@ def mad(a, c=Gaussian.ppf(3/4.), axis=0, center=np.median):
     mad : float
         `mad` = median(abs(`a` - center))/`c`
     """
-    a = array_like(a, 'a', ndim=None)
-    c = float_like(c, 'c')
+    a = array_like(a, "a", ndim=None)
+    c = float_like(c, "c")
     if not a.size:
-        center = 0.0
+        center_val = 0.0
     elif callable(center):
-        center = np.apply_over_axes(center, a, axis)
+        if axis is not None:
+            center_val = np.apply_over_axes(center, a, axis)
+        else:
+            center_val = center(a.ravel())
     else:
-        center = float_like(center, "center")
+        center_val = float_like(center, "center")
 
-    return np.median((np.abs(a-center)) / c, axis=axis)
+    return np.median((np.abs(a - center_val)) / c, axis=axis)
 
 
-def iqr(a, c=Gaussian.ppf(3/4) - Gaussian.ppf(1/4), axis=0):
+def iqr(a, c=Gaussian.ppf(3 / 4) - Gaussian.ppf(1 / 4), axis=0):
     """
     The normalized interquartile range along given axis of an array
 
@@ -75,8 +79,8 @@ def iqr(a, c=Gaussian.ppf(3/4) - Gaussian.ppf(1/4), axis=0):
     -------
     The normalized interquartile range
     """
-    a = array_like(a, 'a', ndim=None)
-    c = float_like(c, 'c')
+    a = array_like(a, "a", ndim=None)
+    c = float_like(c, "c")
 
     if a.ndim == 0:
         raise ValueError("a should have at least one dimension")
@@ -114,9 +118,10 @@ def qn_scale(a, c=1 / (np.sqrt(2) * Gaussian.ppf(5 / 8)), axis=0):
     {float, ndarray}
         The Qn robust estimator of scale
     """
-    a = array_like(a, 'a', ndim=None, dtype=np.float64, contiguous=True,
-                   order='C')
-    c = float_like(c, 'c')
+    a = array_like(
+        a, "a", ndim=None, dtype=np.float64, contiguous=True, order="C"
+    )
+    c = float_like(c, "c")
     if a.ndim == 0:
         raise ValueError("a should have at least one dimension")
     elif a.size == 0:
@@ -197,7 +202,7 @@ def __init__(self, c=1.5, tol=1.0e-08, maxiter=30, norm=None):
         self.tol = tol
         self.norm = norm
         tmp = 2 * Gaussian.cdf(c) - 1
-        self.gamma = tmp + c**2 * (1 - tmp) - 2 * c * Gaussian.pdf(c)
+        self.gamma = tmp + c ** 2 * (1 - tmp) - 2 * c * Gaussian.pdf(c)
 
     def __call__(self, a, mu=None, initscale=None, axis=0):
         """
@@ -262,35 +267,44 @@ def _estimate_both(self, a, scale, mu, axis, est_mu, n):
                     # This is a one-step fixed-point estimator
                     # if self.norm == norms.HuberT
                     # It should be faster than using norms.HuberT
-                    nmu = np.clip(a, mu-self.c*scale,
-                                  mu+self.c*scale).sum(axis) / a.shape[axis]
+                    nmu = (
+                        np.clip(
+                            a, mu - self.c * scale, mu + self.c * scale
+                        ).sum(axis)
+                        / a.shape[axis]
+                    )
                 else:
-                    nmu = norms.estimate_location(a, scale, self.norm, axis,
-                                                  mu, self.maxiter, self.tol)
+                    nmu = norms.estimate_location(
+                        a, scale, self.norm, axis, mu, self.maxiter, self.tol
+                    )
             else:
                 # Effectively, do nothing
                 nmu = mu.squeeze()
             nmu = tools.unsqueeze(nmu, axis, a.shape)
 
-            subset = np.less_equal(np.abs((a - mu)/scale), self.c)
+            subset = np.less_equal(np.abs((a - mu) / scale), self.c)
             card = subset.sum(axis)
 
-            scale_num = np.sum(subset * (a - nmu)**2, axis)
-            scale_denom = (n * self.gamma - (a.shape[axis] - card) * self.c**2)
+            scale_num = np.sum(subset * (a - nmu) ** 2, axis)
+            scale_denom = n * self.gamma - (a.shape[axis] - card) * self.c ** 2
             nscale = np.sqrt(scale_num / scale_denom)
             nscale = tools.unsqueeze(nscale, axis, a.shape)
 
-            test1 = np.alltrue(np.less_equal(np.abs(scale - nscale),
-                                             nscale * self.tol))
+            test1 = np.alltrue(
+                np.less_equal(np.abs(scale - nscale), nscale * self.tol)
+            )
             test2 = np.alltrue(
-                np.less_equal(np.abs(mu - nmu), nscale * self.tol))
+                np.less_equal(np.abs(mu - nmu), nscale * self.tol)
+            )
             if not (test1 and test2):
                 mu = nmu
                 scale = nscale
             else:
                 return nmu.squeeze(), nscale.squeeze()
-        raise ValueError('joint estimation of location and scale failed '
-                         'to converge in %d iterations' % self.maxiter)
+        raise ValueError(
+            "joint estimation of location and scale failed "
+            "to converge in %d iterations" % self.maxiter
+        )
 
 
 huber = Huber()
@@ -331,16 +345,22 @@ class HuberScale(object):
     and the Huber constant h = (n-p)/n*(d**2 + (1-d**2)*\
             scipy.stats.norm.cdf(d) - .5 - d*sqrt(2*pi)*exp(-0.5*d**2)
     """
+
     def __init__(self, d=2.5, tol=1e-08, maxiter=30):
         self.d = d
         self.tol = tol
         self.maxiter = maxiter
 
     def __call__(self, df_resid, nobs, resid):
-        h = df_resid / nobs * (
-            self.d ** 2
-            + (1 - self.d ** 2) * Gaussian.cdf(self.d)
-            - .5 - self.d / (np.sqrt(2 * np.pi)) * np.exp(-.5 * self.d ** 2)
+        h = (
+            df_resid
+            / nobs
+            * (
+                self.d ** 2
+                + (1 - self.d ** 2) * Gaussian.cdf(self.d)
+                - 0.5
+                - self.d / (np.sqrt(2 * np.pi)) * np.exp(-0.5 * self.d ** 2)
+            )
         )
         s = mad(resid)
 
@@ -348,15 +368,22 @@ def subset(x):
             return np.less(np.abs(resid / x), self.d)
 
         def chi(s):
-            return subset(s) * (resid / s) ** 2 / 2 + (1 - subset(s)) * \
-                   (self.d ** 2 / 2)
+            return subset(s) * (resid / s) ** 2 / 2 + (1 - subset(s)) * (
+                self.d ** 2 / 2
+            )
 
         scalehist = [np.inf, s]
         niter = 1
-        while (np.abs(scalehist[niter - 1] - scalehist[niter]) > self.tol
-               and niter < self.maxiter):
-            nscale = np.sqrt(1 / (nobs * h) * np.sum(chi(scalehist[-1])) *
-                             scalehist[-1] ** 2)
+        while (
+            np.abs(scalehist[niter - 1] - scalehist[niter]) > self.tol
+            and niter < self.maxiter
+        ):
+            nscale = np.sqrt(
+                1
+                / (nobs * h)
+                * np.sum(chi(scalehist[-1]))
+                * scalehist[-1] ** 2
+            )
             scalehist.append(nscale)
             niter += 1
             # TODO: raise on convergence failure?


Test patch:
diff --git a/statsmodels/robust/tests/test_scale.py b/statsmodels/robust/tests/test_scale.py
--- a/statsmodels/robust/tests/test_scale.py
+++ b/statsmodels/robust/tests/test_scale.py
@@ -5,25 +5,52 @@
 import numpy as np
 from numpy.random import standard_normal
 from numpy.testing import assert_almost_equal, assert_equal
-from scipy.stats import norm as Gaussian
 import pytest
-# Example from Section 5.5, Venables & Ripley (2002)
+from scipy.stats import norm as Gaussian
 
-import statsmodels.robust.scale as scale
 import statsmodels.api as sm
+import statsmodels.robust.scale as scale
+from statsmodels.robust.scale import mad
+
+# Example from Section 5.5, Venables & Ripley (2002)
+
 
 DECIMAL = 4
 # TODO: Can replicate these tests using stackloss data and R if this
-# data is a problem
+#  data is a problem
 
 
 class TestChem(object):
     @classmethod
     def setup_class(cls):
-        cls.chem = np.array([
-            2.20, 2.20, 2.4, 2.4, 2.5, 2.7, 2.8, 2.9, 3.03,
-            3.03, 3.10, 3.37, 3.4, 3.4, 3.4, 3.5, 3.6, 3.7, 3.7, 3.7, 3.7,
-            3.77, 5.28, 28.95])
+        cls.chem = np.array(
+            [
+                2.20,
+                2.20,
+                2.4,
+                2.4,
+                2.5,
+                2.7,
+                2.8,
+                2.9,
+                3.03,
+                3.03,
+                3.10,
+                3.37,
+                3.4,
+                3.4,
+                3.4,
+                3.5,
+                3.6,
+                3.7,
+                3.7,
+                3.7,
+                3.7,
+                3.77,
+                5.28,
+                28.95,
+            ]
+        )
 
     def test_mean(self):
         assert_almost_equal(np.mean(self.chem), 4.2804, DECIMAL)
@@ -50,10 +77,12 @@ def test_huber_huberT(self):
         n = scale.norms.HuberT()
         n.t = 1.5
         h = scale.Huber(norm=n)
-        assert_almost_equal(scale.huber(self.chem)[0], h(self.chem)[0],
-                            DECIMAL)
-        assert_almost_equal(scale.huber(self.chem)[1], h(self.chem)[1],
-                            DECIMAL)
+        assert_almost_equal(
+            scale.huber(self.chem)[0], h(self.chem)[0], DECIMAL
+        )
+        assert_almost_equal(
+            scale.huber(self.chem)[1], h(self.chem)[1], DECIMAL
+        )
 
     def test_huber_Hampel(self):
         hh = scale.Huber(norm=scale.norms.Hampel())
@@ -84,10 +113,11 @@ def test_mad_center(self):
         assert_equal(n.shape, (10,))
         with pytest.raises(TypeError):
             scale.mad(self.X, center=None)
-        assert_almost_equal(scale.mad(self.X, center=1),
-                            np.median(np.abs(self.X - 1),
-                                      axis=0)/Gaussian.ppf(3/4.),
-                            DECIMAL)
+        assert_almost_equal(
+            scale.mad(self.X, center=1),
+            np.median(np.abs(self.X - 1), axis=0) / Gaussian.ppf(3 / 4.0),
+            DECIMAL,
+        )
 
 
 class TestMadAxes(object):
@@ -169,22 +199,30 @@ def setup_class(cls):
         cls.sunspot = sm.datasets.sunspots.load_pandas().data.SUNACTIVITY
 
     def test_qn_naive(self):
-        assert_almost_equal(scale.qn_scale(self.normal),
-                            scale._qn_naive(self.normal), DECIMAL)
-        assert_almost_equal(scale.qn_scale(self.range),
-                            scale._qn_naive(self.range), DECIMAL)
-        assert_almost_equal(scale.qn_scale(self.exponential),
-                            scale._qn_naive(self.exponential), DECIMAL)
+        assert_almost_equal(
+            scale.qn_scale(self.normal), scale._qn_naive(self.normal), DECIMAL
+        )
+        assert_almost_equal(
+            scale.qn_scale(self.range), scale._qn_naive(self.range), DECIMAL
+        )
+        assert_almost_equal(
+            scale.qn_scale(self.exponential),
+            scale._qn_naive(self.exponential),
+            DECIMAL,
+        )
 
     def test_qn_robustbase(self):
         # from R's robustbase with finite.corr = FALSE
         assert_almost_equal(scale.qn_scale(self.range), 13.3148, DECIMAL)
-        assert_almost_equal(scale.qn_scale(self.stackloss),
-                            np.array([8.87656, 8.87656, 2.21914, 4.43828]),
-                            DECIMAL)
+        assert_almost_equal(
+            scale.qn_scale(self.stackloss),
+            np.array([8.87656, 8.87656, 2.21914, 4.43828]),
+            DECIMAL,
+        )
         # sunspot.year from datasets in R only goes up to 289
-        assert_almost_equal(scale.qn_scale(self.sunspot[0:289]), 33.50901,
-                            DECIMAL)
+        assert_almost_equal(
+            scale.qn_scale(self.sunspot[0:289]), 33.50901, DECIMAL
+        )
 
     def test_qn_empty(self):
         empty = np.empty(0)
@@ -255,3 +293,18 @@ def test_axis2(self):
     def test_axisneg1(self):
         m, s = self.h(self.X, axis=-1)
         assert_equal(m.shape, (40, 10))
+
+
+def test_mad_axis_none():
+    # GH 7027
+    a = np.array([[0, 1, 2], [2, 3, 2]])
+
+    def m(x):
+        return np.median(x)
+
+    direct = mad(a=a, axis=None)
+    custom = mad(a=a, axis=None, center=m)
+    axis0 = mad(a=a.ravel(), axis=0)
+
+    np.testing.assert_allclose(direct, custom)
+    np.testing.assert_allclose(direct, axis0)