Repository: statsmodels/statsmodels

Issue:
'medcouple' function in stats.stattools may calculate a wrong answer?
The 'medcouple' function at stats.stattools.

I refered to this paper to calculate the medcouple value:

> [1] Brys, G, M Hubert, and A Struyf 2004.A Robust Measure of Skewness. Journal of Computational and Graphical Statistics 13(4): 996–1017.

And this 'medcouple' function at stats.stattools used this paper:

> [2] M. Huberta and E. Vandervierenb, "An adjusted boxplot for skewed distributions" Computational Statistics & Data Analysis, vol. 52, pp.5186-5201, August 2008

But I seemingly haven't seen the paper[2] had redefined the 'medcouple' 'kernel' function from the paper[1]. So according to my test which refered paper[1], I think the medcouple may calculated a wrong answer.

Examples as follow:
1. The example, I used your Python 'medcouple' function:
```
import statsmodels
print('version =',statsmodels.version.version)
from statsmodels.stats.stattools import medcouple
data = np.array([0,1,2,2,3],dtype=float)
print('MC =',medcouple(data))
```
the result are:
```
version = 0.9.0
MC = 0.0
```
note: I used Python 3.6.2, NumPy 1.12.1, and statsmodels 0.9.0.

2. And the example of Matlab 'medcouple' function:
```
data = [0,1,2,2,3];
MC= medcouple(data')
```
the result are:
```
MC =

   -0.1667
```
note: I use Matlab R2016a, and the Matlab 'medcouple' function download at the this website:
[https://ww2.mathworks.cn/matlabcentral/fileexchange/28501-tests-to-identify-outliers-in-data-series?focused=5160176&tab=function](url)

3. And the example of R 'mc' function:
```
library("robustbase")
mc(c(0,1,2,2,3))
```
the result are:
```
[1] -0.1666667
```
note: I use R 3.5.1.
You can see the Matlab and R got same answer. But the Python answer is different.
In fact, I checked the value of 'spread / standardization' in 'stattools.py' at line 418, it is a matrix ( or two dimension array). For simplicity, I named it as 'matrix A'. And I named the matrix pass to the 'np.median()' at line 418 as 'matrix B'. In your implement, the 'matrix A == matrix B'
If you pass the 'data' ( used at above example ) to the 'medcouple' function, The matrix B will be:
```
-1      -1   0   0 
-1      -1   0   0
-0.333   0   1   1
```
But according to the paper[1], the matrix B should be:
```
-1      -1  -1   0 
-1      -1   0   1
-0.333   0   1   1
```
I found this difference caused by the 'medcouple' 'kernel' function return value when 'xi==Q2==xj'( this meaning could find on paper[2] or paper[1]).
If for all 'xi==Q2==xj', the 'kernel' function return value 0. The result is your code's result.
But in paper[1], the 'kernel' function return value should based on Eq(2.3) in paper[1]. Did the paper[2] change the definition?
So, it means the function 'medcouple' doesn't handle 'kernel' function value when 'xi==Q2==xj'. You forgot handle this situation when you got matrix A.
Did I say this problem clear? And I haven't saw others notice this problem.
Am I misunderstand or used in a wrong way?
Thank you for reading this problem and apologize to you about my poor English.
My email: ZJ.Cosmos@gmail.com , if you want to contact to me.
Thank you.


Discussion:
As far as I know it is a faithful implementation of the 2008 paper (I implemented ths function). If it is a correct implementation of the 2008 paper, then it is hard to say it is wrong, even if it differs from the 2004 paper.
In the 2008 paper they just refer to the 2004 paper, so this appears to be a bug.
> As far as I know it is a faithful implementation of the 2008 paper (I implemented ths function). If it is a correct implementation of the 2008 paper, then it is hard to say it is wrong, even if it differs from the 2004 paper.

@bashtage  Thank you for your reply.
Sorry, It seems that I didn't read the 2008 paper carefully. 
Please look at the sentence I marked in the  follow screenshot ( captuered from  2008 paper ):
 
![image](https://user-images.githubusercontent.com/9165501/48879379-5c645200-ee46-11e8-9427-3ea7d4642066.png)

Did the ‘occurs with zero probability’ means the 'h(xi,xj)' got 0 for all 'xi==Q2==xj'? If you said yes, I can sure that I misunderstood this sentence. 

But obviously, I have a another confusion.
Why does other implementation ( at least on Matlab and R ) used 2004 paper, but you used 2008 paper? Have some reason?
Other implementation are not always implement the 'medcouple' function before 2008 year, so why they don't chose the 2008 paper to implement?

I also noticed  the 2008 paper' author said that they have implemented the adjusted boxplot in S-PLUS and Matlab at paragraph 2 on page 5201 in their 2008 paper. I marked it in this screenshot :
![image](https://user-images.githubusercontent.com/9165501/48878196-62a3ff80-ee41-11e8-8ddc-fc3bfd2d5d54.png)

I also downloaded the LIBRA toolbox mentioned in the 2008 paper, and tested the 'mc' function in toolbox on my computer. 
Here is the my test:
```
data =[0 1 2 2 3];
mc(data')
```
the result:
```
ans =

   -0.1667
```

Obviously, this toolbox also implemented the 'medcouple' function based on 2004 paper. But the author said they made this implementation. 
So I also don't know why the author implement it based on the 2004 paper, instead of implement it based on their paper. 
LOL.
Thank you for reading and apologize to you about my poor English.
(I looked at this a bit earlier today)

The second article mentions
"For the special case x_i = Q_2 = x_j (which occurs with zero probability) a specific definition applies, see Brys et al. (2004) for the details"

So the 2008 paper doesn't change the ties at median treatment from the 2004 article.

So equ (2.3) in the first article is not implemented in statsmodels.
AFAICS, 2.3 sets an all but k tied pairs at median to +1 or -1 at "arbitrary" indices.
k is number of ties at median.
However, it looks like those +1 and -1 are added in support of the fast algorithm but are not essential.
What looks important is that the 2004 article adds only k zeros, while our algorithm adds zeros for all pairs that are tied at median, AFAICS

my guess is that all R related functions use the fast algorithm (which is in R robustbase) from the 2004 paper


What I didn't figure out when I briefly looked at the sections in the 2004 article is whether this treatment of ties at the medium have a theoretical justification or whether it's kind of arbitrary. Theory based on continuous random variables will not have ties.

A related analogy, some nonparametric hypothesis tests in scipy.stats have several methods how to treat ties or zeros at the threshold, but in that case each treatment is based on a different article.

So, I'm not sure whether we should treat this as a bug and fix it to match R and matlab, and the fast algorithm implementation, or whether there are options that different treatments for ties at median could be justified.

e.g. statsmodels gives more weight to the median tied observations and is less influenced by larger values, while R in the following example has medcouple much closer to the standard Pearson skew.
```
>>> st.medcouple([1] * 10 + [5] * 3)
array(0.0)
```

R robust base
```
> mc(c(1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 5, 5, 5))
[1] 1
```

```
>>> from scipy import stats
>>> stats.skew([1] * 10 + [5] * 3)
1.2780193008453875
```
e.g.
R mc doesn't treat one extra value as an 'outlier'

```
> mc(c(1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 5))
[1] 1
```

using small noise, the continuous random variable medcouple in the previous example would be around 0.52 (mean) or 0.6 (median medcouple)

```
>>> np.mean([st.medcouple(([1] * 10 + [5] * 3) + 1e-15*np.random.randn(13)) for _ in range(10000)])
0.52487

>>> np.median([st.medcouple(([1] * 10 + [5] * 3) + 1e-15*np.random.randn(13)) for _ in range(10000)])
0.6
```

and for 1 "outlier"
```
>>> np.median([st.medcouple(([1] * 10 + [5] * 1) + 1e-15*np.random.randn(11)) for _ in range(10000)])
0.0

>>> np.mean([st.medcouple(([1] * 10 + [5] * 1) + 1e-15*np.random.randn(11)) for _ in range(10000)])
0.0937
```
> So equ (2.3) in the first article is not implemented in statsmodels.
AFAICS, 2.3 sets an all but k tied pairs at median to +1 or -1 at "arbitrary" indices.
k is number of ties at median.
However, it looks like those +1 and -1 are added in support of the fast algorithm but are not essential.
What looks important is that the 2004 article adds only k zeros, while our algorithm adds zeros for all pairs that are tied at median, AFAICS

These are not arbitrary unfortunately, since the always has an even number of elements and so having them affects how the median is computed due to the tie.  It is an easy fix -- I'll submit a PR later.
> So equ (2.3) in the first article is not implemented in statsmodels.
> AFAICS, 2.3 sets an all but k tied pairs at median to +1 or -1 at "arbitrary" indices.
> k is number of ties at median.
> However, it looks like those +1 and -1 are added in support of the fast algorithm but are not essential.
> What looks important is that the 2004 article adds only k zeros, while our algorithm adds zeros for all pairs that are tied at median, AFAICS

@josef-pkt Thank you for your reply.
I also think the +1 and -1 added in Equ2.3 is not arbitrary. 
Here is my understanding of the Equ2.3 (I writed it on Word Application):
![image](https://user-images.githubusercontent.com/9165501/48892381-e9280380-ee78-11e8-9e8d-8b3a816c75ea.png)

![image](https://user-images.githubusercontent.com/9165501/48892476-22f90a00-ee79-11e8-9991-03ecd3b588e6.png)
Did I misunderstand your meaning?
In fact, I read the Matlab implementation source code which downloaded from the website mentioned above. In code 'medcouple.m', they also got the matrix A (mentioned above) first, then use Equ2.3 to got matrix B. So, it hardly to say the Equ2.3 are use for 'fast' algorithm because of the extra steps. This is my thought.

The sentence '**The first and third equations in (2.3) are somewhat superfluous but are added
to avoid undefined kernels and to simplify the implementation of the algorithm described in
Section 4.**' on page 4 in 2004 paper.
I can't understand this sentence exactly. Did author means the +1 or -1 can be replaced by 0?
And I am sorry about that I am not understand the examples of 'continuous random variable'. I am a college student, and have not enough knowledge to understand it. 
I think I should read more relevant paper.
So, I am not sure if it is a bug based on your opinion , you can talk with other teammate or developer to confirm.
Thank you for reading and apologize to you about my poor English.
The important cases occur when the number of ties with the median is even.
This creates an asymmetric matrix in the kernel and is where the bug is
apparent. This is why 1, 2, 2, 3, 4 has the issue, but I think 1,2,2, 2,3
would not.

On Thu, Nov 22, 2018, 09:47 ZJ <notifications@github.com wrote:

> So equ (2.3) in the first article is not implemented in statsmodels.
> AFAICS, 2.3 sets an all but k tied pairs at median to +1 or -1 at
> "arbitrary" indices.
> k is number of ties at median.
> However, it looks like those +1 and -1 are added in support of the fast
> algorithm but are not essential.
> What looks important is that the 2004 article adds only k zeros, while our
> algorithm adds zeros for all pairs that are tied at median, AFAICS
>
> @josef-pkt <https://github.com/josef-pkt> Thank you for your reply.
> I also think the +1 and -1 added in Equ2.3 is not arbitrary.
> Here is my understanding of the Equ2.3 (I writed it on Word Application):
> [image: image]
> <https://user-images.githubusercontent.com/9165501/48892381-e9280380-ee78-11e8-9e8d-8b3a816c75ea.png>
>
> [image: image]
> <https://user-images.githubusercontent.com/9165501/48892476-22f90a00-ee79-11e8-9991-03ecd3b588e6.png>
> Did I misunderstand your meaning?
> In fact, I read the Matlab implementation source code which downloaded
> from the website mentioned above. In code 'medcouple.m', they also got the
> matrix A (mentioned above) first, then use Equ2.3 to got matrix B. So, it
> hardly to say the Equ2.3 are use for 'fast' algorithm because of the extra
> steps. This is my thought.
>
> The sentence '
>
> *The first and third equations in (2.3) are somewhat superfluous but are
> added to avoid undefined kernels and to simplify the implementation of the
> algorithm described in Section 4.*' on page 4 in 2004 paper.
> I can't understand this sentence exactly. Did author means the +1 or -1
> can be replaced by 0?
> And I am sorry about that I am not understand the examples of 'continuous
> random variable'. I am a college student, and have not enough knowledge to
> understand it.
> I think I should read more relevant paper.
> So, I am not sure if it is a bug based on your opinion , you can talk with
> other teammate or developer to confirm.
> Thank you for reading and apologize to you about my poor English.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/statsmodels/statsmodels/issues/5395#issuecomment-440972488>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AFU5Rd7ph8on5bH5Bo1O6x0V8p1I_33Fks5uxnK_gaJpZM4YtTxs>
> .
>

The block with ties is always square and so it is nearly trivial to fix the
algo by replacing this block by a block that has a 0 anti diagonal, - 1
above the anti diagonal and 1 below.

On Thu, Nov 22, 2018, 09:51 Kevin Sheppard <kevin.k.sheppard@gmail.com>
wrote:

> The important cases occur when the number of ties with the median is even.
> This creates an asymmetric matrix in the kernel and is where the bug is
> apparent. This is why 1, 2, 2, 3, 4 has the issue, but I think 1,2,2, 2,3
> would not.
>
> On Thu, Nov 22, 2018, 09:47 ZJ <notifications@github.com wrote:
>
>> So equ (2.3) in the first article is not implemented in statsmodels.
>> AFAICS, 2.3 sets an all but k tied pairs at median to +1 or -1 at
>> "arbitrary" indices.
>> k is number of ties at median.
>> However, it looks like those +1 and -1 are added in support of the fast
>> algorithm but are not essential.
>> What looks important is that the 2004 article adds only k zeros, while
>> our algorithm adds zeros for all pairs that are tied at median, AFAICS
>>
>> @josef-pkt <https://github.com/josef-pkt> Thank you for your reply.
>> I also think the +1 and -1 added in Equ2.3 is not arbitrary.
>> Here is my understanding of the Equ2.3 (I writed it on Word Application):
>> [image: image]
>> <https://user-images.githubusercontent.com/9165501/48892381-e9280380-ee78-11e8-9e8d-8b3a816c75ea.png>
>>
>> [image: image]
>> <https://user-images.githubusercontent.com/9165501/48892476-22f90a00-ee79-11e8-9991-03ecd3b588e6.png>
>> Did I misunderstand your meaning?
>> In fact, I read the Matlab implementation source code which downloaded
>> from the website mentioned above. In code 'medcouple.m', they also got the
>> matrix A (mentioned above) first, then use Equ2.3 to got matrix B. So, it
>> hardly to say the Equ2.3 are use for 'fast' algorithm because of the extra
>> steps. This is my thought.
>>
>> The sentence '
>>
>> *The first and third equations in (2.3) are somewhat superfluous but are
>> added to avoid undefined kernels and to simplify the implementation of the
>> algorithm described in Section 4.*' on page 4 in 2004 paper.
>> I can't understand this sentence exactly. Did author means the +1 or -1
>> can be replaced by 0?
>> And I am sorry about that I am not understand the examples of 'continuous
>> random variable'. I am a college student, and have not enough knowledge to
>> understand it.
>> I think I should read more relevant paper.
>> So, I am not sure if it is a bug based on your opinion , you can talk
>> with other teammate or developer to confirm.
>> Thank you for reading and apologize to you about my poor English.
>>
>> —
>> You are receiving this because you were mentioned.
>> Reply to this email directly, view it on GitHub
>> <https://github.com/statsmodels/statsmodels/issues/5395#issuecomment-440972488>,
>> or mute the thread
>> <https://github.com/notifications/unsubscribe-auth/AFU5Rd7ph8on5bH5Bo1O6x0V8p1I_33Fks5uxnK_gaJpZM4YtTxs>
>> .
>>
>


Resolving patch:
diff --git a/statsmodels/stats/stattools.py b/statsmodels/stats/stattools.py
--- a/statsmodels/stats/stattools.py
+++ b/statsmodels/stats/stattools.py
@@ -415,7 +415,20 @@ def _medcouple_1d(y):
     is_zero = np.logical_and(lower == 0.0, upper == 0.0)
     standardization[is_zero] = np.inf
     spread = upper + lower
-    return np.median(spread / standardization)
+    h = spread / standardization
+    # GH5395
+    num_ties = np.sum(lower == 0.0)
+    if num_ties:
+        # Replacements has -1 above the anti-diagonal, 0 on the anti-diagonal,
+        # and 1 below the anti-diagonal
+        replacements = np.ones((num_ties, num_ties)) - np.eye(num_ties)
+        replacements -= 2 * np.triu(replacements)
+        # Convert diagonal to anti-diagonal
+        replacements = np.fliplr(replacements)
+        # Always replace upper right block
+        h[:num_ties, -num_ties:] = replacements
+
+    return np.median(h)
 
 
 def medcouple(y, axis=0):


Test patch:
diff --git a/statsmodels/stats/tests/test_statstools.py b/statsmodels/stats/tests/test_statstools.py
--- a/statsmodels/stats/tests/test_statstools.py
+++ b/statsmodels/stats/tests/test_statstools.py
@@ -208,12 +208,16 @@ def test_medcouple_symmetry(self, reset_randomstate):
         mcn = medcouple(-x)
         assert_almost_equal(mcp + mcn, 0)
 
+    def test_medcouple_ties(self, reset_randomstate):
+        x = np.array([1, 2, 2, 3, 4])
+        mc = medcouple(x)
+        assert_almost_equal(mc, 1.0 / 6.0)
+
     def test_durbin_watson(self, reset_randomstate):
         x = np.random.standard_normal(100)
         dw = sum(np.diff(x)**2.0) / np.dot(x, x)
         assert_almost_equal(dw, durbin_watson(x))
 
-
     def test_durbin_watson_2d(self, reset_randomstate):
         shape = (1, 10)
         x = np.random.standard_normal(100)