DOC: Updating operators docstrings
State: closed

Checklist for the pandas documentation sprint (ignore this if you are doing
an unrelated PR):

- [X] PR title is "DOC: update the <your-function-or-method> docstring"
- [ ] The validation script passes: `scripts/validate_docstrings.py <your-function-or-method>`
- [X] The PEP8 style check passes: `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] The html version looks good: `python doc/make.py --single <your-function-or-method>`
- [X] It has been proofread on language by another sprint participant

Please include the output of the validation script below between the "```" ticks:

```
################################################################################
####################### Docstring (pandas.DataFrame.gt)  #######################
################################################################################

Wrapper for flexible comparison methods gt

Examples
--------

>>> df1 = pd.DataFrame({'num1': range(1,6),
...                     'num2': range(2,11,2),
...                     'num3': range(1,20,4)})
>>> df1
   num1  num2  num3
0     1     2     1
1     2     4     5
2     3     6     9
3     4     8    13
4     5    10    17
>>> df2 = pd.DataFrame({'num1': range(6,11),
...                     'num2': range(1,10,2),
...                     'num3': range(1,20,4)})
>>> df2
   num1  num2  num3
0     6     1     1
1     7     3     5
2     8     5     9
3     9     7    13
4    10     9    17
>>> df1.gt(df2)
    num1  num2   num3
0  False  True  False
1  False  True  False
2  False  True  False
3  False  True  False
4  False  True  False

################################################################################
################################## Validation ##################################
################################################################################

Errors found:
	Closing quotes should be placed in the line after the last text in the docstring (do not close the quotes in the same line as the text, or leave a blank line between the last text and the quotes)
	Use only one blank line to separate sections or paragraphs
	Summary does not end with dot
	No extended summary found
	Errors in parameters section
		Parameters {'other', 'level', 'axis'} not documented
	No returns section found
	See Also section not found

################################################################################
####################### Docstring (pandas.DataFrame.ge)  #######################
################################################################################

Wrapper for flexible comparison methods ge

Examples
--------

>>> df1 = pd.DataFrame({'num1': range(1,6),
...                     'num2': range(2,11,2),
...                     'num3': range(1,20,4)})
>>> df1
   num1  num2  num3
0     1     2     1
1     2     4     5
2     3     6     9
3     4     8    13
4     5    10    17
>>> df2 = pd.DataFrame({'num1': range(6,11),
...                     'num2': range(1,10,2),
...                     'num3': range(1,20,4)})
>>> df2
   num1  num2  num3
0     6     1     1
1     7     3     5
2     8     5     9
3     9     7    13
4    10     9    17
>>> df1.ge(df2)
    num1  num2   num3
0  False  True   True
1  False  True   True
2  False  True   True
3  False  True   True
4  False  True   True

################################################################################
################################## Validation ##################################
################################################################################

Errors found:
	Closing quotes should be placed in the line after the last text in the docstring (do not close the quotes in the same line as the text, or leave a blank line between the last text and the quotes)
	Use only one blank line to separate sections or paragraphs
	Summary does not end with dot
	No extended summary found
	Errors in parameters section
		Parameters {'other', 'axis', 'level'} not documented
	No returns section found
	See Also section not found



```
If the validation script still gives errors, but you think there is a good reason to deviate in this case (and there are certainly such cases), please state this explicitly.

This PR follows up on previous attempt: https://github.com/pandas-dev/pandas/pull/20291. Please see discussion.

Specifically, comparison methods in ops.py including `.gt()` and `.ge()` do not have standard docstring like other methods. They use a generalized wrapper summary line. This pull request simply adds examples by conditionally building docstring with examples. General changes include: adding global string variable, __flex_comp_doc_FRAME_ and `if` logic to the method, *_flex_comp_method_FRAME*. 


Labels:
name: Docs

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20415?src=pr&el=h1) Report
> Merging [#20415](https://codecov.io/gh/pandas-dev/pandas/pull/20415?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/b45eb265dd88648fb02ebb0fdcfb168364ab664e?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/20415/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/20415?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #20415      +/-   ##
==========================================
- Coverage   42.46%   42.46%   -0.01%     
==========================================
  Files         161      161              
  Lines       51557    51556       -1     
==========================================
- Hits        21892    21891       -1     
  Misses      29665    29665
```

| Flag | Coverage Δ | |
|---|---|---|
| #single | `42.46% <100%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/20415?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/ops.py](https://codecov.io/gh/pandas-dev/pandas/pull/20415/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvb3BzLnB5) | `55.2% <100%> (-0.06%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20415?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20415?src=pr&el=footer). Last update [b45eb26...6879e89](https://codecov.io/gh/pandas-dev/pandas/pull/20415?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

This looks close to ready.  Pls rebase.
Needs rebase.  Aside from that and two small requested edits, this is ready pending OK from @datapythonista .
Thanks for the contribution @ParfaitG.

Your examples look good, but if you have time it'd be cool to add some extra stuff to this PR (sorry we couldn't provide more clear information during the sprint).

I'd personally reuse a single docstring for `eq`, `ne`, `gt`, `gte`, `lt`, `lte`. I think we could provide a bit longer description, mainly for new users, but also explain how these methods are more flexible than the equivalent operators (`==`, `!=`, `>`...) as the axis and levels can be specified.

Then, it'd be good to have the `Parameters` section.

Also, the `See Also` section can be included to link to all these methods sharing the docstring.

For the examples, what you did looks good, but it could help the users if the dataframes are a bit smaller (something like 2 cols and 3 rows), and the example is a bit more meaningful (e.g. instead of columns `num1`... a value that users can understand and quickly compare between dataframes).

How does this sound?
Hello @ParfaitG! Thanks for updating the PR.

 - There are no PEP8 issues in the file [`pandas/core/ops.py`](https://github.com/ParfaitG/pandas/blob/722ae81e803b437127dda4999997111738e91019/pandas/core/ops.py) !



##### Comment last updated on September 27, 2018 at 23:44 Hours UTC
Understood @datapythonista ... please see updated PR with a more comprehensive revision on the comparison operators.
Thanks for the update @ParfaitG. It looks better, but as we discussed before, I don't think it is as useful to have the example of each comparison in each page, than to show well why the methods exists, how they are different from the operators, and how you can use them on different axis.

Yesterday I was working on a tutorial that among other things covers this. You can take a look at the `pandas DataFrame` section to get the idea of what I mean: https://github.com/datapythonista/pandas-tutorials/blob/master/02_Data_structures.ipynb

What I think we should do: Get rid of all the `_eq_example_FRAME`... variables, and move the example to the main docstring (just one method to start with). Same for `_comp_others`. If I'm not missing anything, the same exact content is being using in every docstring anyway (so, there is no value on keeping it in a separate variable).

Then, the example should show:
- Comparing with a scalar (`other` parameter) and the rest of parameters by default.
- Comparing with another `DataFrame`.
- Comparing with `Series`, show how the `axis` parameter changes the behavior.
- Comparing with a `MultiIndex`, show how the `level` parameter changes the behavior.
- Showing the cases where the operator is equivalent to the method

As there are many cases to show, you can keep using the different methods in each of them, so more or less all them are covered. Meaning that when you compare with a scalar you can use `eq`, when you compare with a `DataFrame` you can use `gt`, and so on.

Let me know how does this sound, and if there is anything I can help with.
@datapythonista ... Did you get my note above regarding the need for double curly braces? The entire docstring is being run with `str.format()` and we need to escape the actuals from format's placeholders. Please advise.
Oh, I see, didn't see that comment, sorry. As the number of variables is just two, and the number of curly brackets in the docstring is much larger, I think it'd be simpler to use `docstring % dict(var=...` than `docstring.format(...`. What do you think? Is there any advantage in `format`?
To stay consistent, most string formatting in `ops.py` uses `format`. And from my understanding, the modulo operator is an outmoded, less preferred way for string interpolation in Python. Nonetheless, I did run validation script and these methods passed. Next week I will report the validation result for a few for your reference.
@datapythonista  ... agreed. I removed the `if` logic but did not directly pass into `@Appender` due to the line length rules raise by the flake8 check: `git diff upstream/master -u -- "*.py" | flake8 --diff`.
@jreback...Forgive me, I am not aware of doc-tests but I did run the validation string script on all methods (`eq`, `ne`,  etc.). Re blank line between examples, will do in several hours.

@datapythonista ... Thanks for your help. Curious, should we adjust arithmetic operator doc strings in same manner as comparison operators with a different PR?
@ParfaitG, what @jreback is talking about is #22459. But we still don't have `pandas/core/ops.py` in the CI, so I'd leave adding this docstring to a separate PR where we can add some more files.

Regarding the arithmetic operator docstring, I didn't checked it, but we have more than 1,000 than need improvements, so I guess it'll be one of them, and it'd be really great if you can work on it.
> you requested some changes that I think have been addressed. Can you take a look before we merge?

@datapythonista I think all my concerns have been addressed.

@ParfaitG Thanks for pushing this through.  It took a little longer than expected, but you really nailed it.
@datapythonista, Please see update with arithmetic operators docstring revamped to resemble comparison operators. I removed the need of separate `df_examples` as previously set up.
@datapythonista, I have updated the DataFrame arithmetic operator docstring. And while I was at it, I updated Series arithmetic docstring. I also come to find out there was no Series comparison docstring. It just reuses the arithmetic docstring. 

So, I added a brand new docstring for Series comparison operators which also involved adding a new `def` method in *Series* section at bottom. The `ops.py` does compile and all `validation_string` methods with `doc_test` do pass. Please advise on this extensive update. I hope your head doesn't spin for the confusion!
Yes, but both are being called separately above. And docstrings are assigned differently. But you make a good point we can combine and have callers pass args for conditional returns.
@ParfaitG is it a big problem to address the new changes in a separate PR? The new changes are more substantial and will require some discussions and review, and having the previous changes which were ready to be merged, makes it more complex and time consuming.

I think you should be able to do a `git reset --soft <VERSION_WITH_PREVIOUS_CHANGES>`, and then create a new branch and commit the changes there.
@datapythonista, Absolutely makes sense. I will rollback in this PR and leave Series methods docstrings unchanged and propose an adjustment after merge.
@datapythonista, please note I have rolled back and updated the DataFrame arithmetic operator docstring. Let me know your thoughts and any next steps.
Also, @ParfaitG, I was taking a look at the CI, and seems like we've got tests that validate the operations docstrings, that's why the checks are failing. Can you run the tests, and update the ones that are failing please?
@datapythonista ... I may need some direction on adding "doctests" using that shell script. Do I run your `pytest` command and/or run shell script, `ci/doctests.sh`, then push committed changes to this branch? 

As for other item, before I ever commit changes, I ran `validate_string` on most operator methods; `flake8-diff` for whitespace and line length; and even `make_html` to visually see doc output. Any other checks to run? I can show `validate_string` results on 1-2 methods as I did in opening of this PR. However, CI test dump shows `add`  might have an issue. I will look into this method and ensure `validate_string` runs correct on ALL methods.
Sorry for the lack of context @ParfaitG. The validations you're doing are perfect, and I don't think your changes have any mistake.

What we would like is that every time someone sends a PR, validate whether all examples in the docstrings are correct. Correct in the way that the shown output is the actual result a user would get if they run the examples. For that, we've got the file `ci/doctests.sh`. While what we'd like is to validate every single docstring in pandas, so far we just added the files `core/series.py`, `core/frame.py` and `core/generic.py`. And even in those, we still have some broken examples, that at the moment we skip, and that we're slowly fixing.

What we think it would be good is to add `core/ops.py` to that validation, as you fix all (or most) of the docstrings in that file. So, not only your changes are correct, but any future change to thoses docstrings will necessarily have to pass the doctests.

So, what you can do to make this happen is to edit the file `ci/doctests.sh` and add another validation like the one I sent for `core/generic.py` but for `core/ops.py`. Remove to start with all the stuff in the `-k` parameter (including the parameter itself). Then run the file, and see if there is any broken docstring. If there is any method broken unrelated to the PR, you add its name to the `-k` option. If it's one of the ones you changes (it won't) you fix it. And with some luck, nothing will fail, and we'll start validating all the docstrings on that file since we merge your PR.

Unrelated to this, if you run the pandas "normal" tests, you'll see that some tests are broken. Seems like there are tests validating the content of the docstrings you updated. I assume we wanted to check that when we "inject" the docstrings, every method (`add`, `mul`...) gets its docstring, and not anonther. You'll need to update these tests. You can also see in the checks in this conversation, how they are in red. After fixing them they'll be in green, and we'll be ready to merge your changes.

Sorry this is getting a bit too complicated. Let me know if you need help. 
@datapythonista ... Please advise on why tests failed. I updated `ci/doctests` per your instruction, finding only one method is defined and failed for `ops.py`: `_gen_eval_kwargs`.  I did notice a few others failed for `frame.py`, `series.py`, and `generic.py` and added those in the skip `-k` argument. Let me know if my implementation was off. Also, per this [PR](https://github.com/pandas-dev/pandas/pull/22701) I had to  remove the `addopts = --strict-data-files` from my `setup.cfg` to get this to work or it kept raising _unrecognized argument_.

Also, this time, `validate_string` script failed on the arithmetic methods with something like *summary should be one line*, so I broke the first line up to accommodate, moving *With reverse version, `{reverse}`.* to next line. I think this relates to one of the test fails here. Btw - we do not cover all docstrings for `ops.py`. Recall my earlier attempt that I rolled back to update the `Series` versions of these same comparison and arithmetic operators. We only touch the `DataFrame` versions here.
The failure occurs because you are no longer substituting `equiv` so you aren't getting something like `dataframe + other` to show up in the add doctoring, which the test is explicitly checking for.
@ParfaitG I see that the doctest that fails in `ops.py` is just because of quotes:
```
Expected:
    {"reversed": True, "truediv": True}
Got:
    {'reversed': True, 'truediv': True}
```
Being such a small change, if you want to change it in this PR, instead of skipping the `_gen_eval_kwargs` test, that would also be good.
@ParfaitG can you please take care of the conflicts. Merging after that, and when it's all green. Sorry about the delay.
thanks for the update @ParfaitG 

@jbrockmendel I think you reviewed this before. Do you mind taking another look and merge it if you're happy?
@datapythonista a handful of questions checking that your comments have been resolved.  If those are all OK, then LGTM.
You're right, didn't see those `index = [...]`.

@ParfaitG I think you should be able to catch those with `flake8 --doctests pandas/core/ops.py`