CI: Improve error message format in validate_docstrings.py
State: closed

Enhance the error message generated by executing `scripts/validate_docstrings.py --format=actions --errors=EX03 method-name`. 

Additionally, there appears to be a discrepancy, as including the ``--error=EX03`` flag still results in the display of another error. The cause of this inconsistency is unclear, and it might be related to a specific configuration.

(This observation and suggestion were brought to our attention by @asishm https://github.com/pandas-dev/pandas/issues/56804#issuecomment-1885402362)

After 
<img width="631" alt="image" src="https://github.com/pandas-dev/pandas/assets/77875500/ee8e3125-617b-4b5d-8203-57e892741d23">

Before 
<img width="615" alt="image" src="https://github.com/pandas-dev/pandas/assets/77875500/7ba9b8f8-ea93-48b4-8c7d-6e9fa5128355">

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Labels:
name: Docs
name: CI
description: Continuous Integration

Comments:
> Additionally, there appears to be a discrepancy, as including the `--error=EX03` flag still results in the display of another error. The cause of this inconsistency is unclear, and it might be related to a specific configuration.

This is because when a method name is passed, it goes through the `print_validate_one_results(func_name)` path which only takes the method name and ignores the other arguments.

https://github.com/pandas-dev/pandas/blob/5955ca6645e45d23c978076ab8e556cb91ef124c/scripts/validate_docstrings.py#L444-L458

this patch should address using the error argument.

```diff
diff --git a/scripts/validate_docstrings.py b/scripts/validate_docstrings.py
index 76d64d27b2..0e9ccd79a1 100755
--- a/scripts/validate_docstrings.py
+++ b/scripts/validate_docstrings.py
@@ -411,7 +411,7 @@ def print_validate_all_results(
     return exit_status
 
 
-def print_validate_one_results(func_name: str) -> None:
+def print_validate_one_results(func_name: str, errors: list[str] | None) -> None:
     def header(title, width=80, char="#") -> str:
         full_line = char * width
         side_len = (width - len(title) - 2) // 2
@@ -426,13 +426,25 @@ def print_validate_one_results(func_name: str) -> None:
     sys.stderr.write(f"{result['docstring']}\n")
 
     sys.stderr.write(header("Validation"))
+
     if result["errors"]:
-        sys.stderr.write(f'{len(result["errors"])} Errors found for `{func_name}`:\n')
+        buff = []
+        err_count = 0
+
         for err_code, err_desc in result["errors"]:
+            if errors and err_code not in errors:
+                continue
+            err_count += 1
             if err_code == "EX02":  # Failing examples are printed at the end
-                sys.stderr.write("\tExamples do not pass tests\n")
+                buff.append("\tExamples do not pass tests\n")
                 continue
-            sys.stderr.write(f"\t{err_desc}\n")
+            buff.append(f"\t{err_desc}\n")
+        if not err_count:
+            sys.stderr.write(f'Docstring for "{func_name}" correct. :)\n')
+        else:
+            sys.stderr.write(f'{err_count} Errors found for `{func_name}`:\n')
+            for msg in buff:
+                sys.stderr.write(msg)
     else:
         sys.stderr.write(f'Docstring for "{func_name}" correct. :)\n')
 
@@ -454,7 +466,7 @@ def main(func_name, prefix, errors, output_format, ignore_deprecated, ignore_fun
             ignore_functions,
         )
     else:
-        print_validate_one_results(func_name)
+        print_validate_one_results(func_name, errors)
         return 0
 
```
in addition, it looks like the CI check for validating EX03 issues returns success even if errors exist (that are not part of the ignored methods)

looking at the logs of this build - https://github.com/pandas-dev/pandas/actions/runs/7484980320/job/20372693999?pr=56827#step:9:258 there is an error reported for EX03, however the CI is still green
> This seems reasonable. I think it can be helpful to still keep one tab at the beginning, so the list of errors is still indented. What do you think?

Can't agree any more!

<img width="673" alt="image" src="https://github.com/pandas-dev/pandas/assets/77875500/7c0ad0b6-40c2-4dc9-a877-89798e1c6fcb">

The CI failure is strange, but it doesn't seem to be related to this pr's modification.