Repository: statsmodels/statsmodels

Issue:
statsmodels.stats.descriptivestats.describe returning wrong standard error of the mean?
https://github.com/statsmodels/statsmodels/blob/300b6fba90c65c8e94b4f83e04f7ae1b0ceeac2e/statsmodels/stats/descriptivestats.py#L382-L387

On line L387 to calculate the standard error, shouldn't it be ``std_err.loc[count > 0] /= count.loc[count > 0] ** 0.5``? It seems it's missing a square root to me?

Right now in version 0.13.2, I have `descriptivestats` telling me a different standard error of the mean compared to `DescrStatsW`:

```python
>> print(statsmodels.stats.descriptivestats.describe([0, 1, 2, 3], ["nobs", "std", "std_err"]))
                0
nobs     4.000000
std_err  0.322749 # This seems wrong
std      1.290994

>> statsmodels.stats.weightstats.DescrStatsW([0, 1, 2, 3]).std_mean
0.6454972243679029 # This seems correct

>> scipy.stats.sem([0, 1, 2, 3])
0.6454972243679028
```

statsmodels.stats.descriptivestats.describe returning wrong standard error of the mean?
https://github.com/statsmodels/statsmodels/blob/300b6fba90c65c8e94b4f83e04f7ae1b0ceeac2e/statsmodels/stats/descriptivestats.py#L382-L387

On line L387 to calculate the standard error, shouldn't it be ``std_err.loc[count > 0] /= count.loc[count > 0] ** 0.5``? It seems it's missing a square root to me?

Right now in version 0.13.2, I have `descriptivestats` telling me a different standard error of the mean compared to `DescrStatsW`:

```python
>> print(statsmodels.stats.descriptivestats.describe([0, 1, 2, 3], ["nobs", "std", "std_err"]))
                0
nobs     4.000000
std_err  0.322749 # This seems wrong
std      1.290994

>> statsmodels.stats.weightstats.DescrStatsW([0, 1, 2, 3]).std_mean
0.6454972243679029 # This seems correct

>> scipy.stats.sem([0, 1, 2, 3])
0.6454972243679028
```



Discussion:
Yes looks like a bug
thanks for reporting

AFAICS, The numbers reported by `Describtion` don't have unit tests.




I can help and add in the square root as a pull request fairly trivially but I'm not sure where to put unit tests - in fact I couldn't even find where existing unit tests are.
PRs are welcome.  Even better would be to add some tests.
Yes looks like a bug
thanks for reporting

AFAICS, The numbers reported by `Describtion` don't have unit tests.




I can help and add in the square root as a pull request fairly trivially but I'm not sure where to put unit tests - in fact I couldn't even find where existing unit tests are.
PRs are welcome.  Even better would be to add some tests.

Resolving patch:
diff --git a/statsmodels/stats/descriptivestats.py b/statsmodels/stats/descriptivestats.py
--- a/statsmodels/stats/descriptivestats.py
+++ b/statsmodels/stats/descriptivestats.py
@@ -384,7 +384,7 @@ def numeric(self) -> pd.DataFrame:
         mean = df.mean()
         mad = (df - mean).abs().mean()
         std_err = std.copy()
-        std_err.loc[count > 0] /= count.loc[count > 0]
+        std_err.loc[count > 0] /= count.loc[count > 0] ** 0.5
         if self._use_t:
             q = stats.t(count - 1).ppf(1.0 - self._alpha / 2)
         else:


Test patch:
diff --git a/statsmodels/stats/tests/test_descriptivestats.py b/statsmodels/stats/tests/test_descriptivestats.py
--- a/statsmodels/stats/tests/test_descriptivestats.py
+++ b/statsmodels/stats/tests/test_descriptivestats.py
@@ -1,6 +1,7 @@
 import numpy as np
 from numpy.testing import assert_almost_equal, assert_equal
 import pandas as pd
+import scipy.stats
 import pytest
 
 from statsmodels.iolib.table import SimpleTable
@@ -169,5 +170,15 @@ def test_extension_types(df):
     np.testing.assert_allclose(res.frame.c, res.frame.d)
 
 
+def test_std_err(df):
+    """
+    Test the standard error of the mean matches result from scipy.stats.sem
+    """
+    np.testing.assert_allclose(
+        Description(df["a"]).frame.loc["std_err"],
+        scipy.stats.sem(df["a"])
+    )
+
+
 def test_describe(df):
     pd.testing.assert_frame_equal(describe(df), Description(df).frame)