Repository: scipy/scipy

Issue:
Kolmogorov Smirnov 2 samples returning max distance location [Enhancement]
<!-- 
Thank you for taking the time to report a SciPy issue.
Please describe the issue in detail, and for bug reports
fill in the fields below. You can delete the sections that 
don't apply to your issue. You can view the final output
by clicking the preview button above.
-->

2 sample Kolmogorov Smirnov test (KS test) is about finding the maximal vertical distance between two empirical cumulative distributions; a large max distance is unlikely under the null assumption that the two empirical distributions are sampled from the same theoretical curve. This is implemented in `scipy.stats.ks_twosamp`.

The function calculates but does not return the location of the cumulative at which the max distance happens, but that would be extremely useful for a number of machine learning algorithms, e.g. to get an initial guess for a classifier between the two kinds of data.

I propose to add an optional argument to the function to return the location of the max distance between the distributions in terms of indices of the data and also value of the quantitative variable.

### Reproducing code example:
<!-- 
If you place your code between the triple backticks below, 
it will be marked as a code block automatically 
-->
Here a rough initial implementation for example purposes. The final few lines are the only ones that differ from the current implementation.
```python
def max_distance_cumulative(data1, data2):
    """
    Computes the maximal vertical distance between cumulative distributions
    (this is the statistic for KS tests). Code mostly copied from
    scipy.stats.ks_twosamp

    Parameters
    ----------
    data1 : array_like
        First data set
    data2 : array_like
        Second data set
    Returns
    -------
    d : float
        Max distance, i.e. value of the Kolmogorov Smirnov test. Sign is + if
        the cumulative of data1 < the one of data2 at that location, else -.
    x : float
        Value of x where maximal distance d is reached.
    """
    from numpy import ma

    (data1, data2) = (ma.asarray(data1), ma.asarray(data2))
    (n1, n2) = (data1.count(), data2.count())
    mix = ma.concatenate((data1.compressed(), data2.compressed()))
    mixsort = mix.argsort(kind='mergesort')
    csum = np.where(mixsort < n1, 1./n1, -1./n2).cumsum()

    # Check for ties
    if len(np.unique(mix)) < (n1+n2):
        ind = np.r_[np.diff(mix[mixsort]).nonzero()[0], -1]
        csum = csum[ind]
        mixsort = mixsort[ind]

    csumabs = ma.abs(csum)
    i = csumabs.argmax()

    d = csum[i]
    # mixsort[i] contains the index of mix with the max distance
    x = mix[mixsort[i]]

    return (d, x)

```

I'm happy to make a PR if there is interest for this.



Discussion:
I thought this suggestion is very useful to know the index at maxima.
If this idea put it in `scipy.stats.mstats.ks_twosamp()`, the return variable will be increased by 3(ks value, p-value, index at maxima).

If you do not mind, I would like to take this issue.
While I see the point that this information can be helpful in certain cases, I would not add it:

- it does not belong to the KS-test, i.e. most users don't need this Information when running the test (i meant in most cases when users just want to test whether the distributions are the same)
- the location is often not unique
- it is neither hard nor expensive to get all the locations (in the end, it is just argmax of cdf1 - cdf2) outside of the KS-test

Thank you Chris for your comment that revived the issue. While I respect your point of view, I don't think those points invalidate the issue:

 - it does belong to the test just like the statistics and p value, this location is used to calculate the test statistics directly. Moreover, although many users won't need it, there are certainly specific applications such as model building that are quite major

- non uniqueness is ok, we can report one just like argmax does

 - it is expensive in terms of expertise and boilerplate, since one has to take the scipy function, figure out what's going on (docs are not great there I'm afraid), and back engineer it. I've got a master in physics and a PhD in bioinformatics and it took me decent effort to retrace the steps, I don't think this is a trivial issue.

In my opinion this issue is a typical case where we could just expand the usefulness of the codebase with minimal effort. These situations are all too rare and we should just grab the opportunity when we can.
There is also the problem of backwards-compatability if a new element is added.

This is the code in `ks_2samp` to get the distance between the cdfs:
```
data1 = np.sort(data1)
data2 = np.sort(data2)
n1 = data1.shape[0]
n2 = data2.shape[0]
data_all = np.concatenate([data1, data2])
cdf1 = np.searchsorted(data1, data_all, side='right') / n1
cdf2 = np.searchsorted(data2, data_all, side='right') / n2
d = np.max(np.absolute(cdf1 - cdf2))
```

Let's see if there are further comments from other developers
gh-9753 changes lots of the 2-sample KS code, but only the last line in @chrisb83's snippet above is different.  Even so that PR could compute the argmin, argmax (for D+, D-) without much extra effort.

What would this mean in the 1-sample case? [Context: I've been thinking about extending `kstest` to cover `ks_2samp`, so that one could just call `kstest(data0, data1,...)` where`data0`, `data1` can be one of `(rvs0, cdf), (cdf0, cdf), (rvs0, rvs1)`. It already supports the first two.]



@iosonofabio Thanks for your code. It is exactly what I was looking for. However, I have a question about the code. I am feeding two CDF as numpy to the code, but I keep getting an error of " 'numpy.ndarray' object has no attribute 'count'". I have tried data1['Values'].count() as well. However, I still give the same error. I wonder if you or anyone else who had the same issue can guide me on how to get rid of this error.


Thank you so much.
This issue has been quiet for a while but I was exactly looking for this kind of functionality in `ks_2samp`. I am a modeller and knowing the location of the maximum is important in the context of my work (as @iosonofabio also suggested above). I will use the code above and would certainly appreciate this PR. 
I'm still happy to make a PR for this if there is hope. And since we are on the topic, it would seem reasonable to add and option to `ks_2samp` that returns the KS statistic with a sign instead of the absolute value. Would be SO USEFUL in many occasions.
given that there is interest in this functionality: instead of changing ks_2samp, what about adding a new function, e.g. ks_dist that returns the distance and the location?
Sure, I could do that, in fact I have the function already...

On Fri, Jul 10, 2020, at 06:33, Christoph Baumgarten wrote:
> 
> 
> given that there is interest in this functionality: instead of changing 
> ks_2samp, what about adding a new function, e.g. ks_dist that returns 
> the distance and the location?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub 
> <https://github.com/scipy/scipy/issues/9389#issuecomment-656337235>, or 
> unsubscribe 
> <https://github.com/notifications/unsubscribe-auth/AAJFEADVHYHWA6GSOJADXN3R2YSQ5ANCNFSM4F4T4CVA>.
>

With the introduction of `make_tuple_bunch`, I don't think there are backward compatibility concerns. (I haven't thought about whether this is a good idea or not; just thought I'd mention it.)
Thanks... Meanwhile I had a baby and probably not so much time for this anymore, would be nice if someone picked it up though
It's really better if someone else submits the PR because then I can review and merge.
After reading more carefully, @chrisb83's suggestion about making it a separate function makes more sense to me. There have been a lot of requests for statistical "distances" (in the loose sense, perhaps not metrics), and I count +4 supporters above (@iosonofabio, @caglorithm @HomaiRS @akahard2dj) for this, so maybe adding it would set things rolling for adding other distances (e.g. gh-7089).

A few suggestions to the contributor who wants to pursue this:
* Consider whether this should (also?) go in `special` as one of these [raw statistical functions](https://docs.scipy.org/doc/scipy/reference/special.html#raw-statistical-functions).
* We try to name things "distance" only if it satisfies the axioms of a metric, so think about the name. I haven't assessed whether this is a metric. (What is this called in other software?)
* Before starting, propose the idea on the [mailing list](https://mail.python.org/mailman3/lists/scipy-dev.python.org/), including the name(s) you like, and explaining why this is needed and why it is a "fundamental" thing within SciPy's scope. References to similar functionality in other software is a good idea. Invite people to this issue to discuss.
* If response to the email is positive, start on code. 
* To the extent possible, follow the example of [similar functions](https://docs.scipy.org/doc/scipy/reference/stats.html#statistical-distances).
* If it makes sense to share the code that calculates the KS-statistic between the two functions, please do so.
* Implement an `axis` argument manually only if you can actually vectorize the code. Return a NaN if something about the input is invalid (e.g. zero-length). Don't worry about `nan_policy`, etc. This is a reduction operation, so you can use a [decorator ](https://github.com/scipy/scipy/issues/14651) to add some of these bells and whistles.

... sounds like this will never see the light of day with this kind of entry barrier, but I'll keep fingers crossed they prove me wrong. It would be a pity, since it's literally two lines of code. Good luck!
My comment is only my opinion about a possible path forward. This issue stalled a while ago, so I'm only trying to help, not add barriers. If someone wants to try to submit a two-line PR to get this into the existing function, that's another possibility.
@tirthasheshpatel thanks for your help making wilcoxon` return the z-statistic. Do you think we should add this information to the return of the KS test or have a separate function?
> Do you think we should add this information to the return of the KS test or have a separate function?

I also think a separate distance function would be nice (since we have some already and this would fit in).

Resolving patch:
diff --git a/scipy/stats/_stats_py.py b/scipy/stats/_stats_py.py
--- a/scipy/stats/_stats_py.py
+++ b/scipy/stats/_stats_py.py
@@ -7549,40 +7549,55 @@ def chisquare(f_obs, f_exp=None, ddof=0, axis=0):
                             lambda_="pearson")
 
 
-KstestResult = namedtuple('KstestResult', ('statistic', 'pvalue'))
+KstestResult = _make_tuple_bunch('KstestResult', ['statistic', 'pvalue'],
+                                 ['statistic_location', 'statistic_sign'])
 
 
-def _compute_dplus(cdfvals):
+def _compute_dplus(cdfvals, x):
     """Computes D+ as used in the Kolmogorov-Smirnov test.
 
     Parameters
     ----------
     cdfvals : array_like
         Sorted array of CDF values between 0 and 1
+    x: array_like
+        Sorted array of the stochastic variable itself
 
     Returns
     -------
-      Maximum distance of the CDF values below Uniform(0, 1)
-"""
+    res: Pair with the following elements:
+        - The maximum distance of the CDF values below Uniform(0, 1).
+        - The location at which the maximum is reached.
+
+    """
     n = len(cdfvals)
-    return (np.arange(1.0, n + 1) / n - cdfvals).max()
+    dplus = (np.arange(1.0, n + 1) / n - cdfvals)
+    amax = dplus.argmax()
+    loc_max = x[amax]
+    return (dplus[amax], loc_max)
 
 
-def _compute_dminus(cdfvals):
+def _compute_dminus(cdfvals, x):
     """Computes D- as used in the Kolmogorov-Smirnov test.
 
     Parameters
     ----------
     cdfvals : array_like
         Sorted array of CDF values between 0 and 1
+    x: array_like
+        Sorted array of the stochastic variable itself
 
     Returns
     -------
-      Maximum distance of the CDF values above Uniform(0, 1)
-
+    res: Pair with the following elements:
+        - Maximum distance of the CDF values above Uniform(0, 1)
+        - The location at which the maximum is reached.
     """
     n = len(cdfvals)
-    return (cdfvals - np.arange(0.0, n)/n).max()
+    dminus = (cdfvals - np.arange(0.0, n)/n)
+    amax = dminus.argmax()
+    loc_max = x[amax]
+    return (dminus[amax], loc_max)
 
 
 @_rename_parameter("mode", "method")
@@ -7617,11 +7632,24 @@ def ks_1samp(x, cdf, args=(), alternative='two-sided', method='auto'):
 
     Returns
     -------
-    statistic : float
-        KS test statistic, either D, D+ or D- (depending on the value
-        of 'alternative')
-    pvalue : float
-        One-tailed or two-tailed p-value.
+    res: KstestResult
+        An object containing attributes:
+
+        statistic : float
+            KS test statistic, either D+, D-, or D (the maximum of the two)
+        pvalue : float
+            One-tailed or two-tailed p-value.
+        statistic_location : float
+            Value of `x` corresponding with the KS statistic; i.e., the
+            distance between the empirical distribution function and the
+            hypothesized cumulative distribution function is measured at this
+            observation.
+        statistic_sign : int
+            +1 if the KS statistic is the maximum positive difference between
+            the empirical distribution function and the hypothesized cumulative
+            distribution function (D+); -1 if the KS statistic is the maximum
+            negative difference (D-).
+
 
     See Also
     --------
@@ -7706,17 +7734,29 @@ def ks_1samp(x, cdf, args=(), alternative='two-sided', method='auto'):
     cdfvals = cdf(x, *args)
 
     if alternative == 'greater':
-        Dplus = _compute_dplus(cdfvals)
-        return KstestResult(Dplus, distributions.ksone.sf(Dplus, N))
+        Dplus, d_location = _compute_dplus(cdfvals, x)
+        return KstestResult(Dplus, distributions.ksone.sf(Dplus, N),
+                            statistic_location=d_location,
+                            statistic_sign=1)
 
     if alternative == 'less':
-        Dminus = _compute_dminus(cdfvals)
-        return KstestResult(Dminus, distributions.ksone.sf(Dminus, N))
+        Dminus, d_location = _compute_dminus(cdfvals, x)
+        return KstestResult(Dminus, distributions.ksone.sf(Dminus, N),
+                            statistic_location=d_location,
+                            statistic_sign=-1)
 
     # alternative == 'two-sided':
-    Dplus = _compute_dplus(cdfvals)
-    Dminus = _compute_dminus(cdfvals)
-    D = np.max([Dplus, Dminus])
+    Dplus, dplus_location = _compute_dplus(cdfvals, x)
+    Dminus, dminus_location = _compute_dminus(cdfvals, x)
+    if Dplus > Dminus:
+        D = Dplus
+        d_location = dplus_location
+        d_sign = 1
+    else:
+        D = Dminus
+        d_location = dminus_location
+        d_sign = -1
+
     if mode == 'auto':  # Always select exact
         mode = 'exact'
     if mode == 'exact':
@@ -7727,7 +7767,9 @@ def ks_1samp(x, cdf, args=(), alternative='two-sided', method='auto'):
         # mode == 'approx'
         prob = 2 * distributions.ksone.sf(D, N)
     prob = np.clip(prob, 0, 1)
-    return KstestResult(D, prob)
+    return KstestResult(D, prob,
+                        statistic_location=d_location,
+                        statistic_sign=d_sign)
 
 
 Ks_2sampResult = KstestResult
@@ -7899,8 +7941,8 @@ def ks_2samp(data1, data2, alternative='two-sided', method='auto'):
     Performs the two-sample Kolmogorov-Smirnov test for goodness of fit.
 
     This test compares the underlying continuous distributions F(x) and G(x)
-    of two independent samples.  See Notes for a description
-    of the available null and alternative hypotheses.
+    of two independent samples.  See Notes for a description of the available
+    null and alternative hypotheses.
 
     Parameters
     ----------
@@ -7920,10 +7962,21 @@ def ks_2samp(data1, data2, alternative='two-sided', method='auto'):
 
     Returns
     -------
-    statistic : float
-        KS statistic.
-    pvalue : float
-        One-tailed or two-tailed p-value.
+    res: KstestResult
+        An object containing attributes:
+
+        statistic : float
+            KS test statistic.
+        pvalue : float
+            One-tailed or two-tailed p-value.
+        statistic_location : float
+            Value from `data1` or `data2` corresponding with the KS statistic;
+            i.e., the distance between the empirical distribution functions is
+            measured at this observation.
+        statistic_sign : int
+            +1 if the empirical distribution function of `data1` exceeds
+            the empirical distribution function of `data2` at
+            `statistic_location`, otherwise -1.
 
     See Also
     --------
@@ -8054,11 +8107,25 @@ def ks_2samp(data1, data2, alternative='two-sided', method='auto'):
     cdf1 = np.searchsorted(data1, data_all, side='right') / n1
     cdf2 = np.searchsorted(data2, data_all, side='right') / n2
     cddiffs = cdf1 - cdf2
+
+    # Identify the location of the statistic
+    argminS = np.argmin(cddiffs)
+    argmaxS = np.argmax(cddiffs)
+    loc_minS = data_all[argminS]
+    loc_maxS = data_all[argmaxS]
+
     # Ensure sign of minS is not negative.
-    minS = np.clip(-np.min(cddiffs), 0, 1)
-    maxS = np.max(cddiffs)
-    alt2Dvalue = {'less': minS, 'greater': maxS, 'two-sided': max(minS, maxS)}
-    d = alt2Dvalue[alternative]
+    minS = np.clip(-cddiffs[argminS], 0, 1)
+    maxS = cddiffs[argmaxS]
+
+    if alternative == 'less' or (alternative == 'two-sided' and minS > maxS):
+        d = minS
+        d_location = loc_minS
+        d_sign = -1
+    else:
+        d = maxS
+        d_location = loc_maxS
+        d_sign = 1
     g = gcd(n1, n2)
     n1g = n1 // g
     n2g = n2 // g
@@ -8098,7 +8165,8 @@ def ks_2samp(data1, data2, alternative='two-sided', method='auto'):
             prob = np.exp(expt)
 
     prob = np.clip(prob, 0, 1)
-    return KstestResult(d, prob)
+    return KstestResult(d, prob, statistic_location=d_location,
+                        statistic_sign=d_sign)
 
 
 def _parse_kstest_args(data1, data2, args, N):
@@ -8177,14 +8245,37 @@ def kstest(rvs, cdf, args=(), N=20, alternative='two-sided', method='auto'):
 
     Returns
     -------
-    statistic : float
-        KS test statistic, either D, D+ or D-.
-    pvalue : float
-        One-tailed or two-tailed p-value.
+    res: KstestResult
+        An object containing attributes:
+
+        statistic : float
+            KS test statistic, either D+, D-, or D (the maximum of the two)
+        pvalue : float
+            One-tailed or two-tailed p-value.
+        statistic_location : float
+            In a one-sample test, this is the value of `rvs`
+            corresponding with the KS statistic; i.e., the distance between
+            the empirical distribution function and the hypothesized cumulative
+            distribution function is measured at this observation.
+
+            In a two-sample test, this is the value from `rvs` or `cdf`
+            corresponding with the KS statistic; i.e., the distance between
+            the empirical distribution functions is measured at this
+            observation.
+        statistic_sign : int
+            In a one-sample test, this is +1 if the KS statistic is the
+            maximum positive difference between the empirical distribution
+            function and the hypothesized cumulative distribution function
+            (D+); it is -1 if the KS statistic is the maximum negative
+            difference (D-).
+
+            In a two-sample test, this is +1 if the empirical distribution
+            function of `rvs` exceeds the empirical distribution
+            function of `cdf` at `statistic_location`, otherwise -1.
 
     See Also
     --------
-    ks_2samp
+    ks_1samp, ks_2samp
 
     Notes
     -----


Test patch:
diff --git a/scipy/stats/tests/test_mstats_basic.py b/scipy/stats/tests/test_mstats_basic.py
--- a/scipy/stats/tests/test_mstats_basic.py
+++ b/scipy/stats/tests/test_mstats_basic.py
@@ -1874,13 +1874,6 @@ def test_kstest_2samp(self):
                         res3 = stats.kstest(xm, y, alternative=alternative, mode=mode)
                         assert_equal(np.asarray(res1), np.asarray(res3))
 
-    def test_nametuples_agree(self):
-        result = stats.kstest([1, 2], [3, 4])
-        assert_(isinstance(result, stats._stats_py.KstestResult))
-        result2 = stats._stats_py.Ks_2sampResult(result.statistic, result.pvalue)
-        assert_(isinstance(result2, stats._stats_py.Ks_2sampResult))
-        assert_equal(result, result2)
-
 
 class TestBrunnerMunzel:
     # Data from (Lumley, 1996)
diff --git a/scipy/stats/tests/test_stats.py b/scipy/stats/tests/test_stats.py
--- a/scipy/stats/tests/test_stats.py
+++ b/scipy/stats/tests/test_stats.py
@@ -3958,6 +3958,24 @@ def test_ks1samp_allpaths(self):
         ])
         FuncData(kolmogn, dataset, (0, 1, 2), 3).check(dtypes=[int, float, bool])
 
+    @pytest.mark.parametrize("ksfunc", [stats.kstest, stats.ks_1samp])
+    @pytest.mark.parametrize("alternative, x6val, ref_location, ref_sign",
+                             [('greater', 6, 6, +1),
+                              ('less', 7, 7, -1),
+                              ('two-sided', 6, 6, +1),
+                              ('two-sided', 7, 7, -1)])
+    def test_location_sign(self, ksfunc, alternative,
+                           x6val, ref_location, ref_sign):
+        # Test that location and sign corresponding with statistic are as
+        # expected. (Test is designed to be easy to predict.)
+        x = np.arange(10) + 0.5
+        x[6] = x6val
+        cdf = stats.uniform(scale=10).cdf
+        res = ksfunc(x, cdf, alternative=alternative)
+        assert_allclose(res.statistic, 0.1, rtol=1e-15)
+        assert res.statistic_location == ref_location
+        assert res.statistic_sign == ref_sign
+
     # missing: no test that uses *args
 
 
@@ -4208,6 +4226,23 @@ def test_warnings_gh_14019(self):
             res = stats.ks_2samp(data1, data2, alternative='less')
             assert_allclose(res.pvalue, 0, atol=1e-14)
 
+    @pytest.mark.parametrize("ksfunc", [stats.kstest, stats.ks_2samp])
+    @pytest.mark.parametrize("alternative, x6val, ref_location, ref_sign",
+                             [('greater', 5.9, 5.9, +1),
+                              ('less', 6.1, 6.0, -1),
+                              ('two-sided', 5.9, 5.9, +1),
+                              ('two-sided', 6.1, 6.0, -1)])
+    def test_location_sign(self, ksfunc, alternative,
+                           x6val, ref_location, ref_sign):
+        # Test that location and sign corresponding with statistic are as
+        # expected. (Test is designed to be easy to predict.)
+        x = np.arange(10, dtype=np.float64)
+        y = x.copy()
+        x[6] = x6val
+        res = stats.ks_2samp(x, y, alternative=alternative)
+        assert res.statistic == 0.1
+        assert res.statistic_location == ref_location
+        assert res.statistic_sign == ref_sign
 
 def test_ttest_rel():
     # regression test