DOC: Adding script to validate docstrings, and generate list of all functions/methods with state
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Adding a new script to check whether a docstring follows the defined standards or not. Also, without arguments, the script returns a report of all the functions and methods in the API, with information about its docstring.

CC @jorisvandenbossche 

Labels:
name: Docs

Comments:
Cool! 

Does the running of the doctests already work? (I don't seem to get output of that)
> Does the running of the doctests already work? (I don't seem to get output of that)

Ah, I see it is in the code, but not yet enable to check
Yes, that's right, still not using it, as I need to review why in some cases the expected output doesn't match, even if it's the same, because of something with indentation. But that function already runs them.
Some quick feedback:

- Not all methods seems to be found, eg Series.mean is missing from the output
- It might be good to sort the methods/output (although of course when opening in a spreadsheet, it can easily be done afterwards)
- I think this can be a really useful overview to start from to make a todo list for the sprint
> because of something with indentation

presumably that would be fixed by setting the `NORMALIZE_WHITESPACE` flag?

FYI, I had to make the file executable with `chmod +x` so that that `./scripts/validate_docstrings.py` worked.
Specifically, we could add `doctest_optionflags= NORMALIZE_WHITESPACE` to our `setup.cfg`. I don't think that interferes with regular `pytest` flags when you aren't doing a doctest.
I'll take a look at the tests now. Regarding the missing methods, I was filtering (supposed to be) duplicates, to avoid having twice the stuff implemented in `generic.py`, but it seems functions using the same wrapper as all the stats were also filtered.

I added a column to detect potential code with the same docstring instead. And I made some changes to try to also have the information for properties and some other code not implemented as a "regular" Python function or method.
You can see the output here if you want to take a quick look before running it locally:
https://docs.google.com/spreadsheets/d/1XK25QrVJLUwVIScvRcmysiRe0QRMpCPz78kHsCeOSw4/edit?usp=sharing
Didn't know about the `NORMALIZE_WHITESPACE` flag, thanks for it @TomAugspurger. So far I added it to the script.

@jorisvandenbossche while I agree that having the input sorted would be nicer, I think it's simpler (and it doesn't need to load anything into memory) as it is now. The order is not random, is the one specified in `api.rst`, so it should make some sense. Also, sorting it is as simple as `./validate_docstrings.py | tail -n+2 | sort`. Unless you really think that sorting results is much better, I'll leave it like it is.
@jorisvandenbossche thanks for the review, useful comments. I think I addressed all them.

I think we'll want to keep adding validations as we find that people is making errors, but when you think this is in a good enough state, may be we can get it merged, so people working on the docstrings can start using it.
Great updates! Also good idea to print the docstring. Certainly for ones that are not composited of different parts / shared docstrings etc, it is good to see how the textual one is in totality. I added a commit to always try to dedent it.

> I think we'll want to keep adding validations as we find that people is making errors, but when you think this is in a good enough state, may be we can get it merged, so people working on the docstrings can start using it.

Indeed! Merging now.

Thanks a lot for this!