CI: Adding script to validate consistent and correct capitalization among headings in documentation (#26941)
State: closed

Closes #26941

This script should be running correctly to validate capitalization for various rst files.  Many keywords will need to be added into the script as more and more files are tested on this script. Currently testing on the following rst files through code_checks.sh:

doc/source/development/contributing.rst
doc/source/index.rst doc/source/ecosystem.rst

contributing.rst and ecosystem.rst should produce an exit code of 1, indicating there are incorrect capitalization in their headings.

Labels:
name: Docs
name: CI
description: Continuous Integration

Comments:
Hello @tonywu1999! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-03-07 15:09:27 UTC
This is a lot of code; is there no third party package to do something like this? 
@WillAyd 

I've looked around when trying to solve this problem and I don't think there's a third party package that does everything that we need.  Ultimately, the more complicated parts of the script involve the combination of finding headers in an RST file & determining if a string follows our capitalization convention while ensuring keywords like pandas are properly capitalized too.

I will keep looking for third party packages online to hopefully simplify the lines of this script.  In the meantime, my goal right now is to check if the script is working as expected.
I'm seeing some problems in the CI/Checks that are occurring outside of the errors I purposely created myself.  I'm not exactly sure if my script is causing these errors or if these are errors independent of my script.

1. Linting
Check import format using isort
##[error]Skipped 2 files
Check import format using isort DONE
##[error]Process completed with exit code 1

2. Publish benchmarks artifacts
##[error]Path does not exist /home/runner/work/pandas/pandas/asv_bench/benchmarks.log
##[error]Exit code 1 returned from process: file name '/home/runner/runners/2.163.1/bin/Runner.PluginHost', arguments 'action "GitHub.Runner.Plugins.Artifact.PublishArtifact, Runner.Plugins"'.

Furthermore, I'm having trouble suppressing stderr output that is produced from an imported helper method.  When I attempt to suppress the stderr output on my local machine using a context manager (_class suppress_stdout_stderr_ on line 35), the suppressing works.  However, only when I push my code onto GitHub does the suppressing fail to work on CI/Checks.  This causes more clutter than I would want to appear on CI/Checks.  
I'm in my phone and can't access github actions, but I think both errors are false positives that fail in all PRs, but don't make the CI fail. Is it possible that your script is not exiting with a zero code, or that another error is happening? 
Ok, so to reiterate what you said, I do not need to worry about the 'linting' and 'publish benchmark artifacts' checks.  

Also, I set an exit code of 1 in validate_rst_title_capitalization.py if there are titles/headings that have improper capitalization.  Currently, the script is checking to validate a file that has headings with improper capitalization, so it is returning an exit code of 1 at the moment.  Is this the reason why the context manager is able to suppress stderr output on my local machine but not on GitHub?  I'm very confused on what is going on with regards to suppressing output.




Just ignore those errors then.

I don't think what I said is related to the output not being suppressed in github, not sure about that. 
@tonywu1999 thank you for the code.

You'll need to run "black" on the new created file.

```bash
$ black  path/to/your/file
```

(Don't panic, but it's going to change a lot of lines).

> I'm in my phone and can't access github actions

```Options -> Request Desktop site``` :)

cc @datapythonista 
@datapythonista 

Because the suppressing wasn't working, I changed my code so that rather than using a third party module to find headings in an RST file, I replaced the module with my own code from scratch.  Now, I'm fairly confident the script is outputting on CI/Checks exactly what I want it to output.  

Where should I go from here?  Also if a file contains headings that are not properly capitalized, should I return an system exit code other than zero?
I edited the script so that now there is a function (correct_title_capitalization) that let's the user know what the corrected capitalization title should be.

Example output:
ci/../doc/source/development/contributing.rst:824:Heading capitalization formatted incorrectly. Please correctly capitalize "Type Hints" to "Type hints" 
I edited the script so the main function has been simplified.  I got rid of the bad_title_dictionary as it is not needed anymore.  
@datapythonista 

Are there any additional steps I should take to get the pull request approved?  I'm not sure where to go from here.
@datapythonista

Thank you for your input!  I made the changes that you requested 3 days ago.  
Thank you for your guidance @datapythonista
@pandas-dev/pandas-core CI is red to show how the errors look like, if someone can review we can fix the errors and merge. 
I'm +/- 0 on this. I don't know that the errors highlighted really are all errors, nor that its worth taking this on in pandas. 

But don't oppose strongly if others feel this is moving in the right direction
@tonywu1999 can you add to the description the issue you're addressing here please? @WillAyd we've got some discussion about this, and one of the main issues is that even in the navigation we have inconsistencies, like `pandas code style` and `Pandas Maintenance` next to each other.

The agreement was to not capitalize more than the first letter, and fix al titles.
@datapythonista when you say description, do you mean in the first comment block of my code?  Also do you want me to simply add "GH 26941" to the description?
I mean in this github issue, not in the code, sorry
@datapythonista I changed the description of this issue.  Let me know if you need anything else!
That is the title. ;)

I updated the description myself. This gives context to reviewers who otherwise think there has been no previous discussion. And also, this will make github close the issue once this is merged. 
@datapythonista Haha my bad.  Thank you for your help.
@tonywu1999 looks like some (maybe new?) docstrings dont pass this check.  can you update
@tonywu1999 can you fix the titles you left to see how the errors look like? I think we can get this merged once it's green.
Also, please merge master into your branch, so we make sure we're validating the latest version of the docs. Thanks!
Hi @datapythonista 

I fixed the titles just now and committed the changes.  There should be title changes in doc/source/development/contributing.rst
@datapythonista 
I'm not sure what you meant by that.  I merged with master and CI/Checks, which my code affects, is running fine.  Are you referring to the two X's for pandas-dev.pandas and pandas-dev.pandas(Linux py36_locale) ?
Yes, something failed in that build. Probably something unrelated to this pr, but we need to have it green to merge this. I'm in my phone and can't look at the logs right now, but if you can have a look and see what's the problem, that would be great. 
@datapythonista 

I looked at the error logs. However, I'm not really sure what's causing the errors.