Repository: scipy/scipy

Issue:
`_cdf_cvm` inaccurate for small `n`
The documentation of `_cdf_cvm` notes:

> The function is not expected to be accurate for large values of x, say
    x > 2, when the cdf is very close to 1 and it might return values > 1
    in that case, e.g. _cdf_cvm(2.0, 12) = 1.0000027556716846.

However, in addition to that, the function is inaccurate for small values of `n`, in particular 2. Below approx. 0.042 the value drops to 0, and above approx. 0.666 it jumps to 1. To a smaller extent, similar things happen for other small `n`.

I don't know whether this can be fixed, but it should be documented.

#### Reproducing code example:
```
>>> _cdf_cvm([0.041, 0.042, 0.666, 0.667], n=2)
array([0.      , 0.031081, 0.993724, 1.      ])
```

#### Scipy/Numpy/Python version information:
```
1.6.3 1.20.2 sys.version_info(major=3, minor=8, micro=8, releaselevel='final', serial=0)
```

#### Plots
![bokeh_plot](https://user-images.githubusercontent.com/1738353/125690393-7c89a381-4713-45e9-b394-57e4fcb233c5.png)
![bokeh_plot](https://user-images.githubusercontent.com/1738353/125690535-144db187-ce68-473b-8839-90084fe13ef1.png)


Discussion:
#### Additional remark

I used these small n because I was checking the correctness of the implementation, and the only tabulation I could find, [Scott (2000)](http://dx.doi.org/10.1080/03610920008832479), uses these – probably because for larger n, the distribution quickly approaches n = ∞.

Here is the table:
![Scott2000TableI](https://user-images.githubusercontent.com/1738353/125690961-1f0421b4-d423-486d-81ad-e4b4380af3c9.png)

Comparing these tabulated values with `_cdf_cvm`:

```python
x = [0.11888, 0.34730, 0.46136, 0.58016, 0.74346, 1.16786]
for n in [2, 3, 4, 5, 6, None]:
    print(_cdf_cvm(x, n=n))
```
```
[0.462506 0.906943 0.960456 0.985229 1.       1.      ]
[0.475004 0.904628 0.956971 0.981798 0.995221 1.      ]
[0.481254 0.90347  0.955228 0.980082 0.993916 1.000132]
[0.485003 0.902776 0.954182 0.979053 0.993133 0.999905]
[0.487503 0.902312 0.953485 0.978366 0.992611 0.999754]
[0.500002 0.899997 0.95     0.974935 0.99     0.999   ]
```

For n = ∞ the agreement is good – for smaller n, not so much. Of course it is possible that `_cdf_cvm` is correct and Scott (2000) is not. However, the author notes that the values for n = 2 are based on an analytical calculation, and that for both 2 and 3 his values agree with those of an earlier paper.
Thank you for finding this. FYI, this is a private method so you should not use it as there is no guarantee that it will be present/break in any future release.
I am aware. I was thinking of asking you guys to expose this as a public function. I need the CvM CDF in a context outside of a CvM test.
Hi,

if you check the second reference in the documenation, you will find some more details on the approximation:

Csorgo, S. and Faraway, J. (1996). The Exact and Asymptotic Distribution of Cramér-von Mises Statistics. Journal of the Royal Statistical Society, pp. 221-234. => you can find it on the website https://people.bath.ac.uk/jjf23/papers/index.html

> Below approx. 0.042 the value drops to 0, and above approx. 0.666 it jumps to 1

That is correct, the support of the exact distribution is contained in [1/(12*n), n/3], see Eq 1.1 in the article

You can take a look at Table 1 in the article, it contains a comparison of the exact values and the approximation. And indeed, for n=2, the approximation is not great but that is not a problem with the implementation but rather the way the approximation works (error is about 1/n**2, see Eq 1.8)

So if you need more accurate values for small n, there are other approaches you could use, they are briefly mentioned in the article, too. I hope it helps...


@chrisb83 thanks!

The explanation makes sense, but it should be documented. Especially since the documentation already mentions inaccuracies, that suggests that these are the *only* inaccuracies.
> The explanation makes sense, but it should be documented. Especially since the documentation already mentions inaccuracies, that suggests that these are the _only_ inaccuracies.

Would you like to do a PR to improve the documentation of the function? As for making it public, I am not able to judge as I don't know how this can be used. But since the function is here, I am not against the idea. Then it would mean adding more documentation, input validation (can call a private method which does not have the validation part if that would impact speed) and tests.

What do you think @chrisb83?
@allefeld can you tell us more about applications of the distribution beyond the test?
@tupui, I can make a PR, might take a bit.

@mdhaber  my application is an implementation of the method of [Patra & Sen (2016)](http://users.stat.ufl.edu/~rohitpatra/PapersandDraft/FinalVerNPMixModelJRSSB_high_cont.pdf). Its purpose is to estimate the mixing proportion in a semi-parametric mixture model, useful e.g. for prevalence estimation as well as in the context of FDR correction.
I am not against adding it as a public method. There are a few points that we should take into account:

- It might be confusing if we implement an approximation of the cdf and not the exact CDF. In the CvM test, we clearly say that the p-value is computed using the approximation derived in the article (in the R package `goftest`, the same implementation is used for the CDF: https://search.r-project.org/CRAN/refmans/goftest/html/pCvM.html). At least for small values of n, a user probably expects higher accuracy whereas for the CvM test, it might not be as important (one should still update the documentation). 
- in stats, the CDFs are usually part of the distribution, there are no standalone CDFs => the right place might be `scipy.special`, this already contains functions like `kolmogorov`. but the integration requires a bit more work in that case

Resolving patch:
diff --git a/scipy/stats/_hypotests.py b/scipy/stats/_hypotests.py
--- a/scipy/stats/_hypotests.py
+++ b/scipy/stats/_hypotests.py
@@ -156,7 +156,7 @@ def __repr__(self):
 
 def _psi1_mod(x):
     """
-    psi1 is defined in equation 1.10 in Csorgo, S. and Faraway, J. (1996).
+    psi1 is defined in equation 1.10 in Csörgő, S. and Faraway, J. (1996).
     This implements a modified version by excluding the term V(x) / 12
     (here: _cdf_cvm_inf(x) / 12) to avoid evaluating _cdf_cvm_inf(x)
     twice in _cdf_cvm.
@@ -208,7 +208,7 @@ def _cdf_cvm_inf(x):
     """
     Calculate the cdf of the Cramér-von Mises statistic (infinite sample size).
 
-    See equation 1.2 in Csorgo, S. and Faraway, J. (1996).
+    See equation 1.2 in Csörgő, S. and Faraway, J. (1996).
 
     Implementation based on MAPLE code of Julian Faraway and R code of the
     function pCvM in the package goftest (v1.1.1), permission granted
@@ -245,12 +245,16 @@ def _cdf_cvm(x, n=None):
     Calculate the cdf of the Cramér-von Mises statistic for a finite sample
     size n. If N is None, use the asymptotic cdf (n=inf).
 
-    See equation 1.8 in Csorgo, S. and Faraway, J. (1996) for finite samples,
+    See equation 1.8 in Csörgő, S. and Faraway, J. (1996) for finite samples,
     1.2 for the asymptotic cdf.
 
     The function is not expected to be accurate for large values of x, say
     x > 2, when the cdf is very close to 1 and it might return values > 1
-    in that case, e.g. _cdf_cvm(2.0, 12) = 1.0000027556716846.
+    in that case, e.g. _cdf_cvm(2.0, 12) = 1.0000027556716846. Moreover, it
+    is not accurate for small values of n, especially close to the bounds of
+    the distribution's domain, [1/(12*n), n/3], where the value jumps to 0
+    and 1, respectively. These are limitations of the approximation by Csörgő
+    and Faraway (1996) implemented in this function.
     """
     x = np.asarray(x)
     if n is None:
@@ -317,7 +321,7 @@ def cramervonmises(rvs, cdf, args=()):
     ----------
     .. [1] Cramér-von Mises criterion, Wikipedia,
            https://en.wikipedia.org/wiki/Cram%C3%A9r%E2%80%93von_Mises_criterion
-    .. [2] Csorgo, S. and Faraway, J. (1996). The Exact and Asymptotic
+    .. [2] Csörgő, S. and Faraway, J. (1996). The Exact and Asymptotic
            Distribution of Cramér-von Mises Statistics. Journal of the
            Royal Statistical Society, pp. 221-234.
 
diff --git a/tools/unicode-check.py b/tools/unicode-check.py
--- a/tools/unicode-check.py
+++ b/tools/unicode-check.py
@@ -10,7 +10,8 @@
 # The set of Unicode code points greater than 127 that we
 # allow in the source code.
 box_drawing_chars = set(chr(cp) for cp in range(0x2500, 0x2580))
-allowed = (set(['®', 'é', 'ê', 'ó', 'ö', 'ü', 'λ', 'π', 'ω', '∫', '≠', '≥']) |
+allowed = (set(['®', 'é', 'ê', 'ó', 'ö', 'ő', 'ü',
+                'λ', 'π', 'ω', '∫', '≠', '≥']) |
            box_drawing_chars)
 
 


Test patch: