Repository: scipy/scipy

Issue:
ENH: stats.ttest_ind is inconsistent with R. It does not allow testing against a single datapoint
### Is your feature request related to a problem? Please describe.

It is frequently useful to perform a two-sample t-test for the case when one of the samples has only one datapoint, and variances are equal. Mathematically, only the total variance is required for the test, so variance of 0 can be assumed for the degenerate dataset.

### Describe the solution you'd like.

I would like for the command

ttest_ind(np.random.normal(0,1,100), [1], equal_var=True)

to produce a test statistic and a p-value, not (NAN, NAN) as it does at the moment. Can be made consistent with R, where one can do the following without problem

t.test(rnorm(100),1,var.equal = TRUE)

### Describe alternatives you've considered.

_No response_

### Additional context (e.g. screenshots, GIFs)

_No response_


Discussion:
Often the formula for the t statistic is expressed in terms of the individual sample variances. Can the formula be reduced to show that the NaN you get is just an artifact of the way the formula is expressed? Or is it an assumption, as you wrote, that the sample variance is zero?

If it is an assumption, does the test still produce uniformly distributed p-values under the null hypothesis, or is there a modification to the null distribution?

What I'm after is whether this a correct thing to do, an approximation, or just something that gives non-NaN results.

(At first glance it looks like it's correct that there is an `(n-1)/(n-1)` cancellation - not an assumption - but I can't really look until later.)
I am sorry. The solution was recommended to me by stats SE community, to them it makes common sense. I am trying to find the proof for this case online, but so far I have not succeeded. Here is the discussion on SE, feel free to join in

https://stats.stackexchange.com/questions/606893

Looking at the implementation of the ttest_ind in scipy, I can certainly tell that it attempts to calculate the variance of both arrays in all cases, which should result in a NAN if the array only has one datapoint. Thus, one could extend the functionality of the ttest_ind by allowing exactly one of the arrays to have length 1. This can be done by putting a bunch of IF statements: throwing an exception of both have length 1, otherwise setting the corresponding variance to 0 manually if only one of the arrays have length 1.

So the edit to the function is simple. It still remains to be seen if this is actually correct. I am hoping if somebody with more statistics knowledge could comment
No problem; I'll investigate later today. It looks like it's correct, not an assumption, at least for equal variances.
Yup, I can't find anything wrong with this. The formulas I'm finding present the pooled variance as a weighted average of the sample variances, but symbolically, the `(n-1)` weights cancel with the `(n-1)`s in the denominators of the unbiased sample variances, and we're left with the sum of squared differences of the sample means. This is zero for the sample with only one observation.

After making this fix, as a sanity check:
```python3
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
rng = np.random.default_rng(246834602926842)

x = rng.normal(size=(10000, 2))
y = rng.normal(size=(10000, 1))
res = stats.ttest_ind(x, y, equal_var=True, axis=-1)
plt.hist(res.pvalue)
plt.xlabel('p-value')
plt.ylabel('frequency')
plt.title('p-values produced by ttest_ind under the null hypothesis')
```
![image](https://user-images.githubusercontent.com/6570539/222023862-ecf9b4a1-496d-4fcf-9a62-5076b87025f6.png)

Looks quite uniform to me.  Also `stats.ttest_ind([2, 3, 5], [1.5], equal_var=True)` gives the same result as `t.test(c(2, 3, 5), c(1.5), equal.var=TRUE)` in R.

There are a few ways to get the same numerical result, including setting the sample variance to zero (although this _looks_ a bit suspicious, so it would deserve an explanatory comment). **PR welcome**, but please keep in mind that we only want to change the `equal_var=True` case. I haven't looked at the unequal variance math lately, but I doubt it works the same way, and R produces a `not enough 'y' observations` error in this case. 
@mdhaber I suppose that would help support the issue we've seen on Dunnett's test with the test case having a single obs in the last group.
Sure. If you can open a PR, we can probably merge it today.
👍 for visibility also putting this here. I propose to add the following in `_equal_var_ttest_denom`.

```python
    if n1 == 1:
        v1 = 0
    if n2 == 1:
        v2 = 0
```

Then the only thing is that there is the `RuntimeWarning`. Shall I just filter it out in the root function?

Yes, that and a comment would be good. I would also be OK with one-liners in this case, e.g. `v1 = 0 if n1 == 1 else v1`.

Preferably, the "Precision loss occurred in moment calculation due to " "catastrophic cancellation." would not be emitted by `_moment`  at all if there is only one observation.

The others due to division by zero in `var *= np.divide(n, n-ddof)` can safely be silenced with `np.errstate`, I think.

Resolving patch:
diff --git a/scipy/stats/_stats_py.py b/scipy/stats/_stats_py.py
--- a/scipy/stats/_stats_py.py
+++ b/scipy/stats/_stats_py.py
@@ -1131,7 +1131,8 @@ def _moment(a, moment, axis, *, mean=None):
                               keepdims=True) / np.abs(mean)
         with np.errstate(invalid='ignore'):
             precision_loss = np.any(rel_diff < eps)
-        if precision_loss:
+        n = a.shape[axis] if axis is not None else a.size
+        if precision_loss and n > 1:
             message = ("Precision loss occurred in moment calculation due to "
                        "catastrophic cancellation. This occurs when the data "
                        "are nearly identical. Results may be unreliable.")
@@ -6963,6 +6964,14 @@ def _unequal_var_ttest_denom(v1, n1, v2, n2):
 
 
 def _equal_var_ttest_denom(v1, n1, v2, n2):
+    # If there is a single observation in one sample, this formula for pooled
+    # variance breaks down because the variance of that sample is undefined.
+    # The pooled variance is still defined, though, because the (n-1) in the
+    # numerator should cancel with the (n-1) in the denominator, leaving only
+    # the sum of squared differences from the mean: zero.
+    v1 = np.where(n1 == 1, 0, v1)[()]
+    v2 = np.where(n2 == 1, 0, v2)[()]
+
     df = n1 + n2 - 2.0
     svar = ((n1 - 1) * v1 + (n2 - 1) * v2) / df
     denom = np.sqrt(svar * (1.0 / n1 + 1.0 / n2))
@@ -7423,8 +7432,13 @@ def ttest_ind(a, b, axis=0, equal_var=True, nan_policy='propagate',
         n2 = b.shape[axis]
 
         if trim == 0:
+            if equal_var:
+                old_errstate = np.geterr()
+                np.seterr(divide='ignore', invalid='ignore')
             v1 = _var(a, axis, ddof=1)
             v2 = _var(b, axis, ddof=1)
+            if equal_var:
+                np.seterr(**old_errstate)
             m1 = np.mean(a, axis)
             m2 = np.mean(b, axis)
         else:
@@ -7436,6 +7450,7 @@ def ttest_ind(a, b, axis=0, equal_var=True, nan_policy='propagate',
         else:
             df, denom = _unequal_var_ttest_denom(v1, n1, v2, n2)
         res = _ttest_ind_from_stats(m1, m2, denom, df, alternative)
+
     return Ttest_indResult(*res)
 
 


Test patch:
diff --git a/scipy/stats/tests/test_stats.py b/scipy/stats/tests/test_stats.py
--- a/scipy/stats/tests/test_stats.py
+++ b/scipy/stats/tests/test_stats.py
@@ -3064,9 +3064,8 @@ def test_moment_empty_moment(self):
 
     def test_skewness(self):
         # Scalar test case
-        with pytest.warns(RuntimeWarning, match="Precision loss occurred"):
-            y = stats.skew(self.scalar_testcase)
-            assert np.isnan(y)
+        y = stats.skew(self.scalar_testcase)
+        assert np.isnan(y)
         # sum((testmathworks-mean(testmathworks,axis=0))**3,axis=0) /
         #     ((sqrt(var(testmathworks)*4/5))**3)/5
         y = stats.skew(self.testmathworks)
@@ -3113,9 +3112,8 @@ def test_skew_constant_value(self):
 
     def test_kurtosis(self):
         # Scalar test case
-        with pytest.warns(RuntimeWarning, match="Precision loss occurred"):
-            y = stats.kurtosis(self.scalar_testcase)
-            assert np.isnan(y)
+        y = stats.kurtosis(self.scalar_testcase)
+        assert np.isnan(y)
         #   sum((testcase-mean(testcase,axis=0))**4,axis=0)/((sqrt(var(testcase)*3/4))**4)/4
         #   sum((test2-mean(testmathworks,axis=0))**4,axis=0)/((sqrt(var(testmathworks)*4/5))**4)/5
         #   Set flags for axis = 0 and
@@ -3201,8 +3199,8 @@ class TestStudentTest:
     P1_1_g = 1 - (P1_1 / 2)
 
     def test_onesample(self):
-        with suppress_warnings() as sup, np.errstate(invalid="ignore"), \
-                pytest.warns(RuntimeWarning, match="Precision loss occurred"):
+        with suppress_warnings() as sup, \
+                np.errstate(invalid="ignore", divide="ignore"):
             sup.filter(RuntimeWarning, "Degrees of freedom <= 0 for slice")
             t, p = stats.ttest_1samp(4., 3.)
         assert_(np.isnan(t))
@@ -4230,8 +4228,8 @@ def test_ttest_rel():
     assert_array_almost_equal([t,p],tpr)
 
     # test scalars
-    with suppress_warnings() as sup, np.errstate(invalid="ignore"), \
-            pytest.warns(RuntimeWarning, match="Precision loss occurred"):
+    with suppress_warnings() as sup, \
+            np.errstate(invalid="ignore", divide="ignore"):
         sup.filter(RuntimeWarning, "Degrees of freedom <= 0 for slice")
         t, p = stats.ttest_rel(4., 3.)
     assert_(np.isnan(t))
@@ -4457,8 +4455,7 @@ def test_ttest_ind():
                               [t, p])
 
     # test scalars
-    with suppress_warnings() as sup, np.errstate(invalid="ignore"), \
-            pytest.warns(RuntimeWarning, match="Precision loss occurred"):
+    with suppress_warnings() as sup, np.errstate(invalid="ignore"):
         sup.filter(RuntimeWarning, "Degrees of freedom <= 0 for slice")
         t, p = stats.ttest_ind(4., 3.)
     assert_(np.isnan(t))
@@ -5221,6 +5218,28 @@ def test_ttest_ind_from_stats_inputs_zero():
     assert_equal(result, [np.nan, np.nan])
 
 
+def test_ttest_single_observation():
+    # test that p-values are uniformly distributed under the null hypothesis
+    rng = np.random.default_rng(246834602926842)
+    x = rng.normal(size=(10000, 2))
+    y = rng.normal(size=(10000, 1))
+    q = rng.uniform(size=100)
+
+    res = stats.ttest_ind(x, y, equal_var=True, axis=-1)
+    assert stats.ks_1samp(res.pvalue, stats.uniform().cdf).pvalue > 0.1
+    assert_allclose(np.percentile(res.pvalue, q*100), q, atol=1e-2)
+
+    res = stats.ttest_ind(y, x, equal_var=True, axis=-1)
+    assert stats.ks_1samp(res.pvalue, stats.uniform().cdf).pvalue > 0.1
+    assert_allclose(np.percentile(res.pvalue, q*100), q, atol=1e-2)
+
+    # reference values from R:
+    # options(digits=16)
+    # t.test(c(2, 3, 5), c(1.5), var.equal=TRUE)
+    res = stats.ttest_ind([2, 3, 5], [1.5], equal_var=True)
+    assert_allclose(res, (1.0394023007754, 0.407779907736), rtol=1e-10)
+
+
 def test_ttest_1samp_new():
     n1, n2, n3 = (10,15,20)
     rvn1 = stats.norm.rvs(loc=5,scale=10,size=(n1,n2,n3))
@@ -5318,8 +5337,8 @@ def test_ttest_1samp_popmean_array():
 
 class TestDescribe:
     def test_describe_scalar(self):
-        with suppress_warnings() as sup, np.errstate(invalid="ignore"), \
-             pytest.warns(RuntimeWarning, match="Precision loss occurred"):
+        with suppress_warnings() as sup, \
+              np.errstate(invalid="ignore", divide="ignore"):
             sup.filter(RuntimeWarning, "Degrees of freedom <= 0 for slice")
             n, mm, m, v, sk, kurt = stats.describe(4.)
         assert_equal(n, 1)
@@ -5416,10 +5435,9 @@ def test_describe_empty(self):
 
 
 def test_normalitytests():
-    with pytest.warns(RuntimeWarning, match="Precision loss occurred"):
-        assert_raises(ValueError, stats.skewtest, 4.)
-        assert_raises(ValueError, stats.kurtosistest, 4.)
-        assert_raises(ValueError, stats.normaltest, 4.)
+    assert_raises(ValueError, stats.skewtest, 4.)
+    assert_raises(ValueError, stats.kurtosistest, 4.)
+    assert_raises(ValueError, stats.normaltest, 4.)
 
     # numbers verified with R: dagoTest in package fBasics
     st_normal, st_skew, st_kurt = (3.92371918, 1.98078826, -0.01403734)