DOC: Add sphinx spelling extension
State: closed

This is my first time contributing to a project this big so I hope everything is okay
I am still working on fixing bugs and polishing a few things but here is what I have done so far:

- fixed typos
- updated a wordlist txt file with words to be ignored by the spellchecker
-  added the spellcheck method to the DocBuilder class (should this be named spelling?)

I've re-used the _sphix_build method to run the spelling command, to do this I had to update the _sphix_build method to include 'spelling' as a kind.  I'd like to know if this is the best approach of If I should just replicate the code.

I have added a few configuring options in order to get better results when using the spellcheck, the first one was to use the wordlist text file, the second was to ignore known PyPI packages names and finally to show suggestions of a misspelt word. 

At the moment the spellcheck will run if we type the command: `python make.py spellcheck`  and the spellcheck method is called, I haven't yet figured out how to fail the build when there are exceptions like the Issue 21079 suggested.

----
- [x] closes #21079 
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Labels:
name: Docs

Comments:
Do you have a rough measure of how long this takes to run on the docs?
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21109?src=pr&el=h1) Report
> Merging [#21109](https://codecov.io/gh/pandas-dev/pandas/pull/21109?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/4274b840e64374a39a0285c2174968588753ec35?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/21109/graphs/tree.svg?width=650&height=150&src=pr&token=eZ4WkYLtcO)](https://codecov.io/gh/pandas-dev/pandas/pull/21109?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #21109   +/-   ##
=======================================
  Coverage   91.85%   91.85%           
=======================================
  Files         153      153           
  Lines       49546    49546           
=======================================
  Hits        45509    45509           
  Misses       4037     4037
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.25% <100%> (ø)` | :arrow_up: |
| #single | `41.87% <33.33%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/21109?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/arrays/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/21109/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2NhdGVnb3JpY2FsLnB5) | `95.69% <ø> (ø)` | :arrow_up: |
| [pandas/core/groupby/groupby.py](https://codecov.io/gh/pandas-dev/pandas/pull/21109/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9ncm91cGJ5LnB5) | `92.66% <ø> (ø)` | :arrow_up: |
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/21109/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `96.12% <ø> (ø)` | :arrow_up: |
| [pandas/core/reshape/merge.py](https://codecov.io/gh/pandas-dev/pandas/pull/21109/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS9tZXJnZS5weQ==) | `94.25% <ø> (ø)` | :arrow_up: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/21109/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.22% <ø> (ø)` | :arrow_up: |
| [pandas/io/json/table\_schema.py](https://codecov.io/gh/pandas-dev/pandas/pull/21109/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2pzb24vdGFibGVfc2NoZW1hLnB5) | `98.29% <ø> (ø)` | :arrow_up: |
| [pandas/core/strings.py](https://codecov.io/gh/pandas-dev/pandas/pull/21109/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3RyaW5ncy5weQ==) | `98.62% <ø> (ø)` | :arrow_up: |
| [pandas/core/tools/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/21109/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvdG9vbHMvZGF0ZXRpbWVzLnB5) | `84.98% <ø> (ø)` | :arrow_up: |
| [pandas/core/arrays/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/21109/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2Jhc2UucHk=) | `83.95% <ø> (ø)` | :arrow_up: |
| [pandas/core/panel.py](https://codecov.io/gh/pandas-dev/pandas/pull/21109/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcGFuZWwucHk=) | `97.87% <ø> (ø)` | :arrow_up: |
| ... and [15 more](https://codecov.io/gh/pandas-dev/pandas/pull/21109/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21109?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21109?src=pr&el=footer). Last update [4274b84...d929098](https://codecov.io/gh/pandas-dev/pandas/pull/21109?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

I apologise for taking so long to reply but this took me longer than I expected. I've added all the words to the wordlist (this file is now huge) and corrected some stuff, I will need your help in regards to some changes that I have done:

- changed `indicies` to `indices` since indices is already in the plural form (I only changed comments and docstrings, there are some variables using this indicies but I didn't want to touch them unless you agree)
- changed `performant` to `efficient` this term was used plenty of times and I'm not sure if changing it is desirable (if not I'll revert the changes)

The spelling extension is funny as it divides words by special characters (anything like `-` or `'`) so I had to add some words that don't exist in the dictionary but make sense when attached to others. (hope this makes sense?)

I've also given it some thought and I think we could fail the build by raising an exception if there are any words in this output.txt file, the idea is to:
- open the file
- readlines
- pass if empty
Not sure if this is a smart way to approach the problem though. 

Finally, the time it takes to spellcheck. If every single file needs to be generated then it will take about 2-3minutes to generate and read the files and about 1 minute to spellcheck and output misspelt words.

If the files are already generated then it runs in 1 minute. This is with the `spelling_show_suggestions` option set to False because this option seems to increase the time it takes to spellcheck to about 3x worse.

Thanks for the feedback @WillAyd some of these changes where made yesterday and after working today on it I got a better understanding of a few terms. I also forgot to double check the previous commit, my apologies for it.

I will revert the changes from efficient back to performant. I'll try to do all the changes before submitting another commit so I'll wait a bit 😃👍 
Hello @FabioRosado! Thanks for updating the PR.



































































Cheers ! There are no PEP8 issues in this Pull Request. :beers: 

##### Comment last updated on June 02, 2018 at 13:56 Hours UTC
I got some time to work on this PR and here is the updated version:
- Reverted changes from `efficient` back to `performant`
- Revert `lex-sorted` back to `lexsorted`
- Fixed lint issues
- Added new section on documentation to inform how to run the spell check and how to add words to the file

As for the MultiIndex I changed two terms but as I searched for multi-index it seems that this term appears a fair amount of time so I'm not sure if I should change ALL of them from multi-index to MultiIndex or just leave it as it is.

On the documentation about the spell check should I explain better about the output.txt file and how to find the word that is being marked as spell check?

Finally, should I raise an exception if any errors were found after running the spellcheck command? I think the best way to do this would be opening the output.txt file, check if it's empty and if not raise an exception. Not sure if this i the best way to approach this problem though 😃 
> Finally, should I raise an exception if any errors were found after running the spellcheck command?

Yes, ideally. Do you know how long the spellchecker takes to run over our docs? We're close to our time limit on Travis for the doc build.
Okay I’ll add that bit of code when i get home later on. The spell checker takes 1 minute to run but if the docs weren’t generated yet they are generated first so it takes up to 6 minutes to run 
The code will not raise an exception if there are misspelt words.  I was unsure which exception to raise so I went with SyntaxError with a message that perhaps will be enough to let people know why the exception was raised 😃 

--EDIT--
Sorry for the number of commits but this will be my last one until you want me to change something. I've added the entry to whatsnew hopefully everything looks good. Thanks for the help and patience with me 😄 👍 
I am going on holidays for the next 4-5 days so I won't be able to work on these issues. But when I'm back home on Saturday I will update the PR with all the requested changes 👍 
Ops I thought I had reverted all the terms but I missed those two sorry for that. 
They are now changed and I added the makefile rule hope all is okay now, thanks for the help 😄 👍  

We already have
https://github.com/pandas-dev/pandas/blob/master/scripts/announce.py for
getting a list of contributors.

We could refactor that to update the wordlist of contributors when it's rn.

On Mon, Jun 4, 2018 at 12:51 AM, Fábio Rosado <notifications@github.com>
wrote:

> *@FabioRosado* commented on this pull request.
> ------------------------------
>
> In doc/source/names_wordlist.txt
> <https://github.com/pandas-dev/pandas/pull/21109#discussion_r192631878>:
>
> > +Skolasinski
> +Rinoc
> +Gieseke
> +Safia
> +Abdalla
> +Saumitra
> +Shahapure
> +Pölsterl
> +Rubbert
> +Sinhrks
> +Siu
> +Kwan
> +Seabold
> +Carrucciu
> +Hoyer
> +Pascoe
>
> The spelling extension checks every single file for text/docstrings so the
> names of the contributors needed to be whitelisted in order to be ignored
> by the spellchecker.
>
> How would you advise on generating this? The reason why I'm asking is that
> the spelling extension uses path.join on anything that you pass as a
> wordlist (one or two files) so we only get an exception when trying to use
> let's say a list of names.
>
> I suppose we could run the command and write it to file on every run?
> Perhaps using something like a tempfile to get all those names and use that
> file on every run?
>
> Also, I have encountered a few issues when decoding and splitting every
> name - the spelling extension makes some weird word divisions at times,
> when I try to decode and split the list Yan Facai name will be split like
> this: ['颜发才（Yan', 'Facai）'] since the wordlist expects one line per word,
> then this name will always be marked as wrong due to how the split divided
> the name. I suppose using regex could help with this matter but not sure.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/21109#discussion_r192631878>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIgtosWu7GK3xKWP3HG9wrMuwvp7pks5t5MrcgaJpZM4UDwfG>
> .
>

I see. But the names need to be added to a wordlist like the file named `spelling_wordlist.txt`. Since the extension captures all text within the project every term will have to be whitelisted (unless its a grammatical error) which means that the name of every contributor needs to be included in a file somewhere. 

It makes sense that perhaps the contributor names should be in a separate file like WillAyd suggested. Also, the extension needs very specific ways to whitelist terms - one term per line. As I pointed out, the problem with scrappy contributors name is that the names will need to be split into both newlines + spaces so every name is a whitelisted word.

Due to the fact that the extension divides words by special characters, it could potentially cause trouble with names such as the one from  Yan Facai (example: 颜发才（Yan Facai） ) so the name needs to be added with both the special characters and normal characters - you can see this on the wordlist.

So I'm unsure what to do now in regards to jreback suggestions. If anyone else could give a hand in order for me to solve this issue and the PR merged I'll be happy to fix/modify anything 😃 👍 
> It makes sense that perhaps the contributor names should be in a separate file

Agreed. I would

1. Modify `scripts/announce.py`  to take an optional flag that disables the `+` at the end of the name of new contributors
2. Run `scripts/announce.py v0.4.0..HEAD` to get a list of all the contributors
3. Update `scripts/announce.py` to update the list of contributors (will have to do some de-duplication)
4. Update the docbuild to run scripts/announce.py` on each build.

I've been meaning to change the docs around so that 4 is done anyone. I'll hopefully get to it by the end of the week.

@TomAugspurger that would be nice but I'm of the opinion that should be a separate PR, if only because that will introduce some code changes that exceed what we would normally allow in a DOC change. OK to merge this and have that as a follow up?
I'm fine with doing it as a followup, as long as the doc build is currently
passing. I image this will be hard to keep up to date with master.

On Mon, Jun 4, 2018 at 1:50 PM, William Ayd <notifications@github.com>
wrote:

> @TomAugspurger <https://github.com/TomAugspurger> that would be nice but
> I'm of the opinion that should be a separate PR, if only because that will
> introduce some code changes that exceed what we would normally allow in a
> DOC change. OK to merge this and have that as a follow up?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/21109#issuecomment-394459554>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIl2AXfcDek417vmU2BdfcvVi5xLbks5t5YFzgaJpZM4UDwfG>
> .
>

i am ok with only merging the actual spelling changes

the machinery needs to be done in a maintenance aware manner (no hard coding)

so if this were split up then ok
What hard coding are you referring to, the contributors list?

It'll have to always be "hard coded" in the sense that it's a static text
file. IIUC, in the meantime if we have a new contributor name, the doc
build will fail, and they'll need to manually add their name to the
contributors list?

On Mon, Jun 4, 2018 at 2:20 PM, Jeff Reback <notifications@github.com>
wrote:

> i am ok with only merging the actual spelling changes
>
> the machinery needs to be done in a maintenance aware manner (no hard
> coding)
>
> so if this were split up then ok
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/21109#issuecomment-394468072>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIiIFSDkyEJsegXWS7wUCC9S8qE8Mks5t5Yh2gaJpZM4UDwfG>
> .
>

yes my point is that it needs to be auto generated 
otherwise this will become a maintenance burden

if u want to make this a manual action that is ok too

but it should not be part of he build (is it? my question was never answered)
It's not currently part of the doc build on Travis, so for now it's the ability to run spellcheck + fixing a bunch of typos.

Opened https://github.com/pandas-dev/pandas/issues/21354 for the followup.
Thanks @FabioRosado! If you're interested in the followup changes to the build processes feel free to claim it on #21354.
@h-vetinari this is merged so your comments are probably going to be lost.
Feel free to make a new PR with the fixes.

On Thu, Jun 7, 2018 at 9:35 AM, h-vetinari <notifications@github.com> wrote:

> *@h-vetinari* commented on this pull request.
> ------------------------------
>
> In doc/source/release.rst
> <https://github.com/pandas-dev/pandas/pull/21109#discussion_r193769837>:
>
> > @@ -4115,7 +4115,7 @@ Bug Fixes
>    datetime64 when calling DataFrame.apply. (:issue:`2374`)
>  - Raise exception when calling to_panel on non uniquely-indexed frame (:issue:`2441`)
>  - Improved detection of console encoding on IPython zmq frontends (:issue:`2458`)
> -- Preserve time zone when .append-ing two time series (:issue:`2260`)
> +- Preserve time zone when .appending two time series (:issue:`2260`)
>
> The hyphen was surely intentional...
>
> —
> You are receiving this because you modified the open/close state.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/21109#pullrequestreview-126809631>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIo53cmeI07hXfZMFz0BtJEBk2_39ks5t6TpDgaJpZM4UDwfG>
> .
>

Even though this PR was already merged I'm happy to fix these issues. 
In regards to MultiIndex vs Multi-index it's a bit confusing because both terms seem to be in use (for examples in the `io.rst` the term Multi-index is used on a title but on advanced MultiIndex is used) which one you think is right @TomAugspurger ?

Also apologies for replying here :)
MultiIndex is what we want