DOC: Improving code quality of doc/make.py, PEP-8 and refactoring (#19631)
State: closed

- [X] closes #19631
closes #15591 
closes #10340 
- [ ] tests added / passed
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Summary of the PR:
- doc/make.py in lint and respects PEP-8
- Making the structure of the code more organized
- Added command line arguments for the number of sphinx jobs (#15591), for the PYTHONPATH (#10340)
- Should work in both Python 2 and 3
- Tracebacks are returned if a command fails (before just the output from the command was shown, but no traceback on where the exception happened).


Labels:
name: Docs

Comments:
Updated the PR, I left `html`, `latex`, `clean` and `zip_html`.

Just let me know if something else should be kept.
latex_forced is used as well I think
Yep latex_forced is user too.

https://github.com/pandas-dev/pandas/wiki/Release-Checklist should have everything other than the symlinks, which I’ve done manually.

________________________________
From: Joris Van den Bossche <notifications@github.com>
Sent: Sunday, February 11, 2018 4:08:19 PM
To: pandas-dev/pandas
Cc: Tom Augspurger; Mention
Subject: Re: [pandas-dev/pandas] DOC: Improving code quality of doc/make.py, PEP-8 and refactoring (#19631) (#19634)


latex_forced is used as well I think

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/pandas-dev/pandas/pull/19634#issuecomment-364793935>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABQHIsfvrD2eKVCJCu2MfzdcSrMbbGkFks5tT2TTgaJpZM4SBAvW>.

Updated the PR with `latex_forced`.

@TomAugspurger, we've got rid of `make.py all`, which was implemented as an alias of `make.py html`. But may be we can get it back to execute all the steps for the release: `./make.py clean; ./make.py; ./make.py zip_html;  ./make.py latex_forced`.

Just let me know if you think that's a good idea.
thanks @datapythonista 

so this will prob need to have a followup with some instructions on how to use it? or is completely obvious from the --help? maybe a README.md?
@jreback a README never hurts, and we could surely have some more info, but I think the usage is quite covered in:
* The docstring at the beginning of the script
* The docstrings of each command
* The README.rst file in `doc/`
* The Contributing to documentation guide:https://pandas.pydata.org/pandas-docs/stable/contributing.html#id42
* And the release checklist: https://github.com/pandas-dev/pandas/wiki/Release-Checklist
* The help command

I'm working in implementing a way to build a single document from the api (and pep8fying `doc/source/conf.py` which I forgot in this PR). I'll review the documentation about `doc/make.py` once I finish that, and add documentation for anything not explained.
> I'm working in implementing a way to build a single document from the api (

What do you already have? Because I updated yesterday my script I shared, and it is basically working (was planning to do a PR to add it this morning, but did not yet come to it)
@jorisvandenbossche I'll share what I've got in a branch later today. But my approach was to create a temporary directory, copy the files to have a minimal sphinx project there, run sphinx on it, and then copy the single api page it generates to the real project.

I see 2 advantages in this approach, first I found it tricky to build a single page from the api from sphinx, and second I prefer to avoid deleting all the .rst files to build the project. Doing a `git checkout` on them later is not a big deal if you know about it, but newcomers may find it confusing.
you know —single already exists yes?
@jreback thanks for double checking, but yes, I do. Actually I want to reuse it for simplicity of the interface. But now you can do something like `./make.py html --single contributing.rst` but not something like `./make.py hml --single pandas.DataFrame.head` which is what @jorisvandenbossche and myself were discussing, and which will be very convenient for the sprint.
I was planning to remove this "OK to delete those files" as well, as it is indeed very annoying (and I think not needed, if you specify to sphinx that those files can be ignored, that's the approach I was taking)

I will put up my approach as well as a PR, and then you can test it and we can see which approach is the easiest for the end user.
@jorisvandenbossche if you plan to remove the file deletion, which I think it's implemented in sphinx `conf.py`, can you please first review and merge #19839, so there are no conflicts with it later? It's a simple PR to make the file PEP-8 compatible.

If you can build a single api doc without deleting the rst, that's a surely much better approach than what I did. I didn't see how to do that. There are still couple of things missing, but this is my approach: https://github.com/datapythonista/pandas/commit/5b4cc3ba4c48c497f93d31d8d4bfec16db4e046e
@datapythonista my attempt is here: https://github.com/pandas-dev/pandas/pull/19840. I used `exclude_patterns` to not have to delete the files. This now seems to work smoothly (try eg `python make.py --docstring DataFrame.join`), apart from a problem I have with `subprocess.check_call` (will comment on the PR). 
I also still can be smarter (now hardcoded 'method', but could in principle detect this from the passed method/function name, instead of requiring the user to specify this).

#19839 will not give too much conflict, I only deleted some lines from conf.py