Repository: statsmodels/statsmodels

Issue:
robust.mad is not being computed correctly or is non-standard definition; it returns the median
Hello,

The statsmodel median absolute deviation function robust.mad in not correct in my opinion. It returns the median. The function stand_mad returns the mad. Maybe this is intentional.

Here is some example code with its output.

import numpy as np
import statsmodels.robust.scale as sm
from scipy.stats import norm as Gaussian

mu=100.0
sigma=15.0
x = np.random.normal(mu, sigma, 10000)

print 'mu, sigma ', mu, sigma

print 'statsmodel MAD: ', sm.mad(x)
print 'statsmodel standardized MAD: ', sm.stand_mad(x)
# x = np.asarray(x)

c = Gaussian.ppf(3.0/4.0)

print 'MAD to rms ratio: ', c, 1.0/c

m = np.median(x)
mymad = np.median(np.fabs(m - x) / c)

print 'my MAD: ', mymad
print 'median(x), median(x)/c: ', np.median(x), np.median(x)/c

mu, sigma  100.0 15.0
statsmodel MAD:  148.312355816
statsmodel standardized MAD:  15.1501297196
MAD to rms ratio:  0.674489750196 1.48260221851
my MAD:  15.1501297196
median(x), median(x)/c:  100.035163825 148.312355816

mu, sigma are the test gaussian data with mean 100 and sigma=15.0

mad returns  the median
stand_mad returns the sigma

I propose that stand_mad is deprecated and mad is replaced with the code 
in stand_mad



Discussion:
My reading:

mad is used when `mu` is assumed to be zero, The main current use is in RLM applied to the residuals, when the mean is automatically zero *), `mad` is currently the default for RLM

std_mad is the usual standalone definition, when we don't have an assumption on `mu`.

*) actually, after checking, I'm not sure because it's applied to `resid` not the weighted residuals `wresid`.

I don't know if there are better names, Skipper implemented this from Huber's book.

These functions were leftover from Jonathan's work IIRC. I'll have to check on the proposal vs. what we need for RLM, but I'm sympathetic to the confusion.  

2 bug reports means we need to do something here

I had a statsmodels.stats.robust_descriptive in work (before I got distracted with release problems), that focuses on robust measures for skew and kurtosis, but could/should also get robust standard deviation.

(barely related: a collection of measures comparing two arrays http://statsmodels.sourceforge.net/devel/tools.html#measure-for-fit-performance-eval-measures
I don't remember why it ended up in tools
) 

A PR to change this to standard (non-confusing) usage would be very welcome. 

Looking at the code, we already have `stand_mad`

Subtracting the median is the only difference between `mad` and `stand_mad`, AFAICS.

Yes. `stand_mad` is what you should use if want traditional MAD on an existing univariate random variable. Our `mad` is used internally on the residuals in RLM for a consistent estimator for the std error of the regression IIRC (and is also what's used by default in rlm in MASS in R and likely in Huber's book for RLM).

Probably we just combine both via a keyword argument of mad (with a deprecation warning). E.g., like in R,

mad(x, center=np.median)

and we change RLM internally to mad(x, 0) or whatever. 

(where's np.nanmedian?)

We need to check if the call to RLM doesn't get messier with an extra keyword.

Improving the docstring of `mad` to point to `stand_mad` and to mention the difference might be enough to direct users to the function that they want. ?

The keyword wouldn't go to RLM. The user doesn't need to control this. I was just thinking internally to change the call to `mad(resid, center=0)`. And change `mad` to take a variable, a center, and a correction number in addition to just the variable and correction keyword now. Then we just get rid of stand_mad.

Yes, looks fine. 
I just checked: RLM uses a string `'mad'` or `'stand_mad'` which is independent of the call signature for the mad function. 

Yeah, I'm just removing the ability to use stand_mad.


Resolving patch:
diff --git a/statsmodels/robust/robust_linear_model.py b/statsmodels/robust/robust_linear_model.py
--- a/statsmodels/robust/robust_linear_model.py
+++ b/statsmodels/robust/robust_linear_model.py
@@ -187,9 +187,12 @@ def _estimate_scale(self, resid):
         """
         if isinstance(self.scale_est, str):
             if self.scale_est.lower() == 'mad':
-                return scale.mad(resid)
+                return scale.mad(resid, center=0)
             if self.scale_est.lower() == 'stand_mad':
-                return scale.stand_mad(resid)
+                return scale.mad(resid)
+            else:
+                raise ValueError("Option %s for scale_est not understood" %
+                                 self.scale_est)
         elif isinstance(self.scale_est, scale.HuberScale):
             return self.scale_est(self.df_resid, self.nobs, resid)
         else:
@@ -222,15 +225,13 @@ def fit(self, maxiter=50, tol=1e-8, scale_est='mad', init=None, cov='H1',
         maxiter : int
             The maximum number of iterations to try. Default is 50.
         scale_est : string or HuberScale()
-            'mad', 'stand_mad', or HuberScale()
+            'mad' or HuberScale()
             Indicates the estimate to use for scaling the weights in the IRLS.
             The default is 'mad' (median absolute deviation.  Other options are
-            use 'stand_mad' for the median absolute deviation standardized
-            around the median and 'HuberScale' for Huber's proposal 2.
-            Huber's proposal 2 has optional keyword arguments d, tol, and
-            maxiter for specifying the tuning constant, the convergence
-            tolerance, and the maximum number of iterations.
-            See models.robust.scale for more information.
+            'HuberScale' for Huber's proposal 2. Huber's proposal 2 has
+            optional keyword arguments d, tol, and maxiter for specifying the
+            tuning constant, the convergence tolerance, and the maximum number
+            of iterations. See statsmodels.robust.scale for more information.
         tol : float
             The convergence tolerance of the estimate.  Default is 1e-8.
         update_scale : Bool
@@ -252,6 +253,12 @@ def fit(self, maxiter=50, tol=1e-8, scale_est='mad', init=None, cov='H1',
             raise ValueError("Convergence argument %s not understood" \
                 % conv)
         self.scale_est = scale_est
+        if (isinstance(scale_est,
+                       basestring) and scale_est.lower() == "stand_mad"):
+            from warnings import warn
+            warn("stand_mad is deprecated and will be removed in 0.7.0",
+                 FutureWarning)
+
         wls_results = lm.WLS(self.endog, self.exog).fit()
         if not init:
             self.scale = self._estimate_scale(wls_results.resid)
@@ -591,9 +598,6 @@ class RLMResultsWrapper(lm.RegressionResultsWrapper):
 #    model_ols = models.regression.OLS(endog, exog)
 #    results_ols = model_ols.fit()
 
-#    model_huber = RLM(endog, exog, M=norms.HuberT(t=2.))
-#    results_huber = model_huber.fit(scale_est="stand_mad", update_scale=False)
-
 #    model_ramsaysE = RLM(endog, exog, M=norms.RamsayE())
 #    results_ramsaysE = model_ramsaysE.fit(update_scale=False)
 
diff --git a/statsmodels/robust/scale.py b/statsmodels/robust/scale.py
--- a/statsmodels/robust/scale.py
+++ b/statsmodels/robust/scale.py
@@ -14,7 +14,8 @@
 import norms
 from statsmodels.tools import tools
 
-def mad(a, c=Gaussian.ppf(3/4.), axis=0):  # c \approx .6745
+def mad(a, c=Gaussian.ppf(3/4.), axis=0, center=np.median):
+    # c \approx .6745
     """
     The Median Absolute Deviation along given axis of an array
 
@@ -26,40 +27,27 @@ def mad(a, c=Gaussian.ppf(3/4.), axis=0):  # c \approx .6745
         The normalization constant.  Defined as scipy.stats.norm.ppf(3/4.),
         which is approximately .6745.
     axis : int, optional
-        The defaul is 0.
+        The defaul is 0. Can also be None.
+    center : callable or float
+        If a callable is provided, such as the default `np.median` then it
+        is expected to be called center(a). The axis argument will be applied
+        via np.apply_over_axes. Otherwise, provide a float.
 
     Returns
     -------
     mad : float
-        `mad` = median(abs(`a`))/`c`
+        `mad` = median(abs(`a` - center))/`c`
     """
     a = np.asarray(a)
-    return np.median((np.fabs(a))/c, axis=axis)
+    if callable(center):
+        center = np.apply_over_axes(center, a, axis)
+    return np.median((np.fabs(a-center))/c, axis=axis)
 
 def stand_mad(a, c=Gaussian.ppf(3/4.), axis=0):
-    """
-    The standardized Median Absolute Deviation along given axis of an array.
-
-    Parameters
-    ----------
-    a : array-like
-        Input array.
-    c : float, optional
-        The normalization constant.  Defined as scipy.stats.norm.ppf(3/4.),
-        which is approximately .6745.
-    axis : int, optional
-        The defaul is 0.
-
-    Returns
-    -------
-    mad : float
-        `mad` = median(abs(`a`-median(`a`))/`c`
-    """
-
-    a = np.asarray(a)
-    d = np.median(a, axis = axis)
-    d = tools.unsqueeze(d, axis, a.shape)
-    return np.median(np.fabs(a - d)/c, axis = axis)
+    from warnings import warn
+    warn("stand_mad is deprecated and will be removed in 0.7.0. Use mad "
+         "instead.", FutureWarning)
+    return mad(a, c=c, axis=axis)
 
 class Huber(object):
     """
@@ -138,7 +126,7 @@ def __call__(self, a, mu=None, initscale=None, axis=0):
             est_mu = False
 
         if initscale is None:
-            scale = stand_mad(a, axis=axis)
+            scale = mad(a, axis=axis)
         else:
             scale = initscale
         scale = tools.unsqueeze(scale, axis, a.shape)
@@ -235,7 +223,7 @@ def __call__(self, df_resid, nobs, resid):
         h = (df_resid)/nobs*(self.d**2 + (1-self.d**2)*\
                     Gaussian.cdf(self.d)-.5 - self.d/(np.sqrt(2*np.pi))*\
                     np.exp(-.5*self.d**2))
-        s = stand_mad(resid)
+        s = mad(resid)
         subset = lambda x: np.less(np.fabs(resid/x),self.d)
         chi = lambda s: subset(s)*(resid/s)**2/2+(1-subset(s))*(self.d**2/2)
         scalehist = [np.inf,s]


Test patch:
diff --git a/statsmodels/robust/tests/test_scale.py b/statsmodels/robust/tests/test_scale.py
--- a/statsmodels/robust/tests/test_scale.py
+++ b/statsmodels/robust/tests/test_scale.py
@@ -25,8 +25,8 @@ def test_mean(self):
     def test_median(self):
         assert_almost_equal(np.median(self.chem), 3.385, DECIMAL)
 
-    def test_stand_mad(self):
-        assert_almost_equal(scale.stand_mad(self.chem), 0.52632, DECIMAL)
+    def test_mad(self):
+        assert_almost_equal(scale.mad(self.chem), 0.52632, DECIMAL)
 
     def test_huber_scale(self):
         assert_almost_equal(scale.huber(self.chem)[0], 3.20549, DECIMAL)
@@ -51,12 +51,12 @@ def __init__(self):
         np.random.seed(54321)
         self.X = standard_normal((40,10))
 
-    def test_stand_mad(self):
-        m = scale.stand_mad(self.X)
+    def test_mad(self):
+        m = scale.mad(self.X)
         assert_equal(m.shape, (10,))
 
-    def test_mad(self):
-        n = scale.mad(self.X)
+    def test_mad_center(self):
+        n = scale.mad(self.X, center=0)
         assert_equal(n.shape, (10,))
 
 class TestMadAxes():
@@ -65,19 +65,19 @@ def __init__(self):
         self.X = standard_normal((40,10,30))
 
     def test_axis0(self):
-        m = scale.stand_mad(self.X, axis=0)
+        m = scale.mad(self.X, axis=0)
         assert_equal(m.shape, (10,30))
 
     def test_axis1(self):
-        m = scale.stand_mad(self.X, axis=1)
+        m = scale.mad(self.X, axis=1)
         assert_equal(m.shape, (40,30))
 
     def test_axis2(self):
-        m = scale.stand_mad(self.X, axis=2)
+        m = scale.mad(self.X, axis=2)
         assert_equal(m.shape, (40,10))
 
     def test_axisneg1(self):
-        m = scale.stand_mad(self.X, axis=-1)
+        m = scale.mad(self.X, axis=-1)
         assert_equal(m.shape, (40,10))
 
 class TestHuber():