Repository: scipy/scipy

Issue:
scipy.stats.mstats.winsorize documentation error
I'm using SciPy version 0.10.1 with GIT revision: '772893ff951c61f15a710b3b773ac12e69254fa4'.

In the mstats module, the winsorize function looks to have a bad specification for the upper percentile limit.

The docstring says, "The (limits[0])th lowest values are set to the (limits[0])th percentile, and the (limits[1])th highest values are set to the (limits[1])th percentile."

In this mode of usage, if you wanted to winsorize at the 5% and 95% levels, you would call the function with limits=(0.05, 0.05). But based on the docstring, this makes it sound as if the limits[1]th highest values (so top 5%) will get the score from the (limits[1])th (so 5%) percentile. But actually you get the (1-limits[1])th percentile (the 95% score).

I hit the error because the documentation makes it sound very much like the way to use the function is with limits=(0.05, 0.95) (I would argue this is the most clear way to accept args for winsorization). But if you do this on an array of, for example, 10 elements, you'll see that the output is a constant array.



Discussion:
Could you add a short self-contained example please?

(didn't look yet.)
winsorize should be consistent with tmean and the other trimmed functions

(0.05, 0.05) is more consistent with symmetric trimming or winsorizing. (stats only has symmetric trimming IIRC)

Sure, that's fine. But then the docstring is incorrect.

Here's an example showing the difference. Keep in mind, I am fine with the limits=(0.05, 0.05) version being considered correct. I just mean that, in that case, it is not the (limits[1])th percentile that the upper (limits[1])th most points get clipped to. It is actually the (1-limits[1])th percentile that they get clipped to.

```
In [104]: some_data
Out[104]:
array([ 1.69131681,  0.33446892,  0.68600033,  0.51787011,  1.04507263,
        0.15676183,  0.58246934,  0.21323679, -0.00690474,  1.9993031 ])

In [105]: import scipy

In [106]: scipy.stats.mst
scipy.stats.mstats         scipy.stats.mstats_basic   scipy.stats.mstats_extras

In [106]: scipy.stats.mstats.winsorize(some_data, limits=(0.05, 0.95))
Out[106]:
masked_array(data = [-0.00690474 -0.00690474 -0.00690474 -0.00690474 -0.00690474 -0.00690474
 -0.00690474 -0.00690474 -0.00690474 -0.00690474],
             mask = False,
       fill_value = 1e+20)


In [107]: scipy.stats.mstats.winsorize(some_data, limits=(0.05, 0.05))
Out[107]:
masked_array(data = [ 1.69131681  0.33446892  0.68600033  0.51787011  1.04507263  0.15676183
  0.58246934  0.21323679 -0.00690474  1.9993031 ],
             mask = False,
       fill_value = 1e+20)
```

Good, then it's a bug in the docstring that needs to be fixed.

Thanks for reporting, PR's to improve documentation (and/or code) are welcome.

I just ran into this same issue and this post helped to clear up the confusion.  Thanks.


Resolving patch:
diff --git a/scipy/stats/mstats_basic.py b/scipy/stats/mstats_basic.py
--- a/scipy/stats/mstats_basic.py
+++ b/scipy/stats/mstats_basic.py
@@ -1416,12 +1416,12 @@ def tsem(a, limits=None, inclusive=(True,True)):
 tsem.__doc__ = stats.tsem.__doc__
 
 
-def winsorize(a, limits=None, inclusive=(True,True), inplace=False, axis=None):
-    """
-    Returns a Winsorized version of the input array.
+def winsorize(a, limits=None, inclusive=(True, True), inplace=False,
+              axis=None):
+    """Returns a Winsorized version of the input array.
 
     The (limits[0])th lowest values are set to the (limits[0])th percentile,
-    and the (limits[1])th highest values are set to the (limits[1])th
+    and the (limits[1])th highest values are set to the (1 - limits[1])th
     percentile.
     Masked values are skipped.
 
@@ -1447,29 +1447,34 @@ def winsorize(a, limits=None, inclusive=(True,True), inplace=False, axis=None):
         Axis along which to trim. If None, the whole array is trimmed, but its
         shape is maintained.
 
+    Notes
+    -----
+    This function is applied to reduce the effect of possibly spurious outliers
+    by limiting the extreme values.
+
     """
     def _winsorize1D(a, low_limit, up_limit, low_include, up_include):
         n = a.count()
         idx = a.argsort()
         if low_limit:
             if low_include:
-                lowidx = int(low_limit*n)
+                lowidx = int(low_limit * n)
             else:
-                lowidx = np.round(low_limit*n)
+                lowidx = np.round(low_limit * n)
             a[idx[:lowidx]] = a[idx[lowidx]]
         if up_limit is not None:
             if up_include:
-                upidx = n - int(n*up_limit)
+                upidx = n - int(n * up_limit)
             else:
-                upidx = n - np.round(n*up_limit)
-            a[idx[upidx:]] = a[idx[upidx-1]]
+                upidx = n - np.round(n * up_limit)
+            a[idx[upidx:]] = a[idx[upidx - 1]]
         return a
     # We gonna modify a: better make a copy
     a = ma.array(a, copy=np.logical_not(inplace))
     #
     if limits is None:
         return a
-    if (not isinstance(limits,tuple)) and isinstance(limits,float):
+    if (not isinstance(limits, tuple)) and isinstance(limits, float):
         limits = (limits, limits)
     # Check the limits
     (lolim, uplim) = limits
@@ -1485,9 +1490,10 @@ def _winsorize1D(a, low_limit, up_limit, low_include, up_include):
     #
     if axis is None:
         shp = a.shape
-        return _winsorize1D(a.ravel(),lolim,uplim,loinc,upinc).reshape(shp)
+        return _winsorize1D(a.ravel(), lolim, uplim, loinc, upinc).reshape(shp)
     else:
-        return ma.apply_along_axis(_winsorize1D, axis,a,lolim,uplim,loinc,upinc)
+        return ma.apply_along_axis(_winsorize1D, axis, a, lolim, uplim, loinc,
+                                   upinc)
 
 
 #####--------------------------------------------------------------------------


Test patch: