Repository: scipy/scipy
Pull request: BUG: fix refguide-check namedtuple handling
Handle scipy.stats namedtyple *Result type classes, e.g. `ConfidenceInterval(low=0.9950085825848624, high=0.9971212407917498)` which can show up in both expected and actual doctest output.  Here "handle" means convert these into tuples of numeric values and compare those.

Previously, refuide-check only handled the cases where the expected output was given as a tuple. Otherwise, it defaulted to comparing strings.
With this change, fiddling with least-significant digits should not be necessary anymore.

#### Reference issue
<!--Example: Closes gh-WXYZ.-->

closes gh-14282



Problem statement:
DOC, CI: stats skewtest refguide failure
This is showing up in the CI of unrelated PRs like gh-14281 and appears to be related to small floating point precision difference (a bit outside my expertise, but pasting the raw log contents below):

```
021-06-22T22:42:06.0038137Z ===========
2021-06-22T22:42:06.0038731Z scipy.stats
2021-06-22T22:42:06.0039292Z ===========
2021-06-22T22:42:06.0039732Z 
2021-06-22T22:42:06.0040287Z scipy.stats.skewtest
2021-06-22T22:42:06.0044399Z --------------------
2021-06-22T22:42:06.0044659Z 
2021-06-22T22:42:06.0045354Z File "build/testenv/lib/python3.8/site-packages/scipy/stats/stats.py", line 1408, in skewtest
2021-06-22T22:42:06.0045792Z Failed example:
2021-06-22T22:42:06.0046136Z     skewtest([2, 8, 0, 4, 1, 9, 9, 0])
2021-06-22T22:42:06.0046476Z Expected:
2021-06-22T22:42:06.0046843Z     SkewtestResult(statistic=0.44626385374196975, pvalue=0.6554066631275459)
2021-06-22T22:42:06.0047232Z Got:
2021-06-22T22:42:06.0047587Z     SkewtestResult(statistic=0.4462638537419698, pvalue=0.6554066631275459)
2021-06-22T22:42:06.0047859Z 
2021-06-22T22:42:06.0047991Z 
2021-06-22T22:42:06.0048585Z File "build/testenv/lib/python3.8/site-packages/scipy/stats/stats.py", line 1410, in skewtest
2021-06-22T22:42:06.0049033Z Failed example:
2021-06-22T22:42:06.0049358Z     skewtest([1, 2, 3, 4, 5, 6, 7, 8000])
2021-06-22T22:42:06.0049693Z Expected:
2021-06-22T22:42:06.0050074Z     SkewtestResult(statistic=3.571773510360407, pvalue=0.0003545719905823133)
2021-06-22T22:42:06.0050439Z Got:
2021-06-22T22:42:06.0050811Z     SkewtestResult(statistic=3.5717735103604067, pvalue=0.00035457199058231423)
2021-06-22T22:42:06.0051078Z 
2021-06-22T22:42:06.0051208Z 
2021-06-22T22:42:06.0051811Z File "build/testenv/lib/python3.8/site-packages/scipy/stats/stats.py", line 1412, in skewtest
2021-06-22T22:42:06.0052251Z Failed example:
2021-06-22T22:42:06.0052613Z     skewtest([100, 100, 100, 100, 100, 100, 100, 101])
2021-06-22T22:42:06.0053231Z Expected:
2021-06-22T22:42:06.0053599Z     SkewtestResult(statistic=3.5717766638478072, pvalue=0.000354567720281634)
2021-06-22T22:42:06.0053977Z Got:
2021-06-22T22:42:06.0054335Z     SkewtestResult(statistic=3.5717766638478063, pvalue=0.00035456772028163574)
2021-06-22T22:42:06.0054615Z 
2021-06-22T22:42:06.0054745Z 
2021-06-22T22:42:06.0055035Z ERROR: refguide or doctests have errors
```

Hints from issue discussion:
There's an issue in refguide-check, I'll take a look. 
Locally a different one fails for me, but it's numerical noise anyway and hopefully gh-14283 makes it more robust by delegating these cases to `np.allclose`.

Commit message: BUG: fix refguide-check namedtuple handling

Handle scipy.stats namedtyple *Result type classes, e.g.
`ConfidenceInterval(low=0.9950085825848624, high=0.9971212407917498)`
which can show up in both expected and actual doctest output.

Here "handle" means convert these into tuples of numeric values
and compare those.

Patch to review:
diff --git a/tools/refguide_check.py b/tools/refguide_check.py
index 6c3e2db5e8f0..219da4fa4615 100755
--- a/tools/refguide_check.py
+++ b/tools/refguide_check.py
@@ -490,6 +490,22 @@ def check_rest(module, names, dots=True):
       'Inf': np.inf,}
 
 
+def try_convert_namedtuple(got):
+    # suppose that "got" is smth like MoodResult(statistic=10, pvalue=0.1).
+    # Then convert it to the tuple (10, 0.1), so that can later compare tuples.
+    num = sum(symb == '=' for symb in got)
+    if num == 0:
+        # not a nameduple, bail out
+        return got
+    regex = (r'[\w\d_]+\(' +
+             ', '.join([r'[\w\d_]+=(.+)']*num) +
+             r'\)')
+    grp = re.findall(regex, got.replace('\n', ' '))
+    # fold it back to a tuple
+    got_again = '(' + ', '.join(grp[0]) + ')'
+    return got_again
+
+
 class DTRunner(doctest.DocTestRunner):
     DIVIDER = "\n"
 
@@ -595,6 +611,14 @@ def check_output(self, want, got, optionflags):
                 s_got = ", ".join(s_got[1:-1].split())
                 return self.check_output(s_want, s_got, optionflags)
 
+            if "=" not in want and "=" not in got:
+                # if we're here, want and got cannot be eval-ed (hence cannot
+                # be converted to numpy objects), they are not namedtuples
+                # (those must have at least one '=' sign).
+                # Thus they should have compared equal with vanilla doctest.
+                # Since they did not, it's an error.
+                return False
+
             if not self.parse_namedtuples:
                 return False
             # suppose that "want"  is a tuple, and "got" is smth like
@@ -602,16 +626,10 @@ def check_output(self, want, got, optionflags):
             # Then convert the latter to the tuple (10, 0.1),
             # and then compare the tuples.
             try:
-                num = len(a_want)
-                regex = (r'[\w\d_]+\(' +
-                         ', '.join([r'[\w\d_]+=(.+)']*num) +
-                         r'\)')
-                grp = re.findall(regex, got.replace('\n', ' '))
-                if len(grp) > 1:  # no more than one for now
-                    return False
-                # fold it back to a tuple
-                got_again = '(' + ', '.join(grp[0]) + ')'
-                return self.check_output(want, got_again, optionflags)
+
+                got_again = try_convert_namedtuple(got)
+                want_again = try_convert_namedtuple(want)
+                return self.check_output(want_again, got_again, optionflags)
             except Exception:
                 return False
 


Code review comments:
diff_hunk: @@ -595,23 +611,25 @@ def check_output(self, want, got, optionflags):
                 s_got = ", ".join(s_got[1:-1].split())
                 return self.check_output(s_want, s_got, optionflags)
 
+            if "=" not in want and "=" not in got:
+                # if we're here, want and got cannot be eval-ed (hence cannot
+                # be converted to numpy objects), they are not namedtuples
+                # (those must have at least one '=' sign).
+                # Thus they should have compared equal with vanilla doctest.
+                # Since they did not, it's an error.
+                return False
+
             if not self.parse_namedtuples:
                 return False
             # suppose that "want"  is a tuple, and "got" is smth like
             # MoodResult(statistic=10, pvalue=0.1).
             # Then convert the latter to the tuple (10, 0.1),
             # and then compare the tuples.
             try:
-                num = len(a_want)
-                regex = (r'[\w\d_]+\(' +
-                         ', '.join([r'[\w\d_]+=(.+)']*num) +
-                         r'\)')
-                grp = re.findall(regex, got.replace('\n', ' '))
-                if len(grp) > 1:  # no more than one for now
-                    return False
-                # fold it back to a tuple
-                got_again = '(' + ', '.join(grp[0]) + ')'
-                return self.check_output(want, got_again, optionflags)
+
+                got_again = try_convert_namedtuple(got)
+                want_again = try_convert_namedtuple(want)
+                return self.check_output(want_again, got_again, optionflags)
path: tools/refguide_check.py
text: @user1:
Can we move this part to an `else` clause of `try/except`? Ideally we'd only have code in the `try/except` that we expect to fail for a good reason. I assume the "good reason" here is that it couldn't be converted to namedtuple, rather than `check_output` throwing some error.

@author:
Readability is in the eye of the reviewer, so done --- moved the return line out of the except part. Is that what you meant Eric?

diff_hunk: @@ -490,6 +490,22 @@ def check_rest(module, names, dots=True):
       'Inf': np.inf,}
 
 
+def try_convert_namedtuple(got):
+    # suppose that "got" is smth like MoodResult(statistic=10, pvalue=0.1).
+    # Then convert it to the tuple (10, 0.1), so that can later compare tuples.
+    num = sum(symb == '=' for symb in got)
path: tools/refguide_check.py
text: @user1:
```suggestion
    num = got.count('=')
```

@author:
Done, thanks!