[make docs] please help make the validation process easier
Author: stas00
Pull request: no
Labels: name: Feature request

Writing serious documents in .rst is such a pain because the sphinx builder is terrible at times. 

If all goes well I can incrementally run `make docs` and it rebuilds just the modified page which is relatively quick, while it still re-runs highlighting on all pages (not needed for the doc I'm working on)

But if a single error happens it rebuilds everything from scratch which takes forever and chances are that it fails again are very high. since half the time I have no idea what the error is. It's good if it even tells me the line number but sometimes it doesn't even give any context - what a horrible tool. So I have to do a lot of guessing and a lot of waiting and by the end of it I don't really want to finish the doc I was very inspired to write.

There must be a better way to isolate just the page I'm working on. I don't care for cross references, I just want to be able to quickly validate that my page will "compile" and not error.

For example how do I hack `make docs` to not do die on:
```
Warning, treated as error:
```
This is extremely painful, as after each error it rebuilds everything which takes forever.

I think what would ease the process in this particular situation is to leave warnings as warnings only make them errors when I commit, and obviously on CI. So something that:

* doesn't treat warnings as errors
* doesn't rebuild everything if something failed in the previous run
* doesn't re-run highlighting on all pages
* ideally a way to work on just one page of my choice - surely it could detect the only modified file - but if it's too much to ask I would be happy to manually supply it

Something like:
```
utils/checksingledoc.py file.rst
```
I don't know sphinx, so it's very hard for me to know what to propose. Perhaps it could build a project made of a single file (or several files) on the fly and that it could solve the problem? Of course, it will need to ignore cross-references as those won't be available and probably other features that I didn't think of.

Or perhaps there is an existing 3rd party program that lints .rst in the same way shpinx does and could be configured to do things that will make the doc writing easier?

Thank you!

@sgugger 

Comments:
I am not aware of anything that could make life easier on this as I would have implemented it/documented it if I knew of it. Your solution of creating a project with just two files does not work, as it would then very likely be impossible to import the file in question and sphinx needs to do that.

I'm not aware of any software that lints properly the .rst. Happy to add any new functionality the doc styler that could help here (as this one runs fast and can be run on a given .py/.rst file) though it's medium priority in terms of development of the project (we do want users to be able to build the documentation smoothly and easily but there are other things more important).

Thank you for feedback, @sgugger 

-------------------

So to disable `Warning, treated as error:` I need to drop `-W` in:

```
cd docs && make html SPHINXOPTS="-W"
```

and then need to figure out how to skip highlighting as it re-works all files on every run.

------------------

I started looking at finding a single page linter that supports sphinx's custom parser. I will post my findings here:
- https://pypi.org/project/restructuredtext-lint/ - says partially supports sphinx
- https://pypi.org/project/doc8/ - supports sphinx, but may have its own demands



Also did you know sphinx has parallel processing with `-j`?

I added `-a -E`, which forces a full rebuild, just for the test so that we are comparing the same things.

```
time make html SPHINXOPTS="-a -E"
real    1m15.265s
user    1m15.114s
sys     0m1.790s
```

```
time make html SPHINXOPTS="-a -E -j 6"
real    0m39.555s
user    1m31.551s
sys     0m7.608s
```

this is almost twice as fast! 

It seems that on my setup `-j 5` is just as fast but less heat get generated (41 sec).

It has `-j auto` - to use all cpu cores, but it's a bad idea, since it won't get any faster with 12 or more workers. Any number of workers beyond 5 on my setup provides a tiny speedup.

Do you think it'd be a good idea to add say `SPHINXOPTS="-j 4"` as the default?




I filed a bug report https://github.com/sphinx-doc/sphinx/issues/8681 since if that `re-highlighting of all modules` stage gets fixed to not re-run on all modules when only 1 files is modified and I drop `-W` temporarily - then the rebuild should be almost instantaneous for a single modified file and thus we won't need to look for an outside linter.


Last time I tried ot use multiprocessing I didn't get any speed up, but it might have been because I was trying the auto option. We can certainly try with 4 cores to begin with.

Excellent!

In general `auto` is almost never a good option w/o knowing the user's setup. That's why I never use `make test`, which runs `pytest -n auto` - I have 12 cpu cores and it can't possibly run 12 workers on 2 gpus - the outcome is really bad.

> I filed a bug report [sphinx-doc/sphinx#8681](https://github.com/sphinx-doc/sphinx/issues/8681) since if that `re-highlighting of all modules` stage gets fixed to not re-run on all modules when only 1 files is modified and I drop `-W` temporarily - then the rebuild should be almost instantaneous for a single modified file and thus we won't need to look for an outside linter.

sphinx dev has fixed this issue in master, so now `make docs` for one modified file is blazingly fast - ~5sec.

Most of the overhead is loading tf+pt I think.
```
time python -c "import torch, tensorflow"
2021-01-18 09:54:49.558444: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0

real    0m1.810s
user    0m2.375s
sys     0m1.219s
```

I personally am pretty happy with this outcome, so closing this ticket.

Oh, that's very nice!