CLN: @doc - base.py & indexing.py
State: closed

- [x] working on #31942 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Docs
name: Clean

Comments:
Replacing all docstring decorator at once will be too hard to accomplish. I would prefer to do it step-by-step. Clean a few / one file in a PR, and do things iteratively. 

Would you mind help me review the current changes? At the mean time, I will keep working on next part.
@HH-MWB can you post screenshots of the affected docstrings?
> @HH-MWB can you post screenshots of the affected docstrings?

@WillAyd  No problem. Should I use `help()` or `.__doc__`? 
You can build the doc strings and just post a screenshot of the HTML
> You can build the doc strings and just post a screenshot of the HTML

Get! Will do, thanks!
> thanks @HH-MWB lgtm. would rather [#31970 (comment)](https://github.com/pandas-dev/pandas/pull/31970#discussion_r379617289) was addressed here but could be a follow-up.
> 
> would it be feasible to inherit the docstring template from the super class instead of explicitly specifying it? It would address my concern for Categorical array.

@simonjayhawkins Sorry for the late reply. 

First, I agree with you that **inherit the docstring from the super class is very intuitive**, and also could saves us a lot of time. Then, when I dive deeper for this case, I found that the original docstring is different with it's super class. This difference confused me and I am trying to figure it out what cause this difference and how can we solve it.


### Inherit Docstring
I think it is a very good idea to inherit the docstring. Maybe this should not be a default behavior but at least we should be able to do inheritance easily. 

A potential quick solution for this is to create another decorator, for example, `doc_inhert`. I would expect it to bring the first non-empty docstring for it's super class. 

For example, `B.f` should have the docstring inhert from `A.f`:
```python
class A:
    def f(self):
        """func doc"""

class B(A):
    @doc_inhert
    def f(self):
        pass
```

Another example, `C.f` should have the docstring inhert from `A.f`, because `B.f` is empty:
```python
class A:
    def f(self):
        """func doc"""

class B(A):
    def f(self):
        pass

class C(B):
    @doc_inhert
    def f(self):
        pass
```

I am still looking to see if there are other better solutions. For example, one thing in my mind will be create decorator for the class, and do docstring inhert for all methods belong to that class.

Would you mind if I create a new issue about docstring inheritance and propose my idea there? I am not 100% sure, but I think we already have some issues about this. 

Open for suggestions and advices. What should I do next?

### Case Here

As what I mentioned above, the original docstring is different from it's super class's docstring. Should I keep it as the original one or switch to be same as the super class one?

Here is the original docstring:
```
Find indices where elements should be inserted to maintain order.

Find the indices into a sorted {klass} `self` such that, if the
corresponding elements in `value` were inserted before the indices,
the order of `self` would be preserved.

.. note::

    The {klass} *must* be monotonically sorted, otherwise
    wrong locations will likely be returned. Pandas does *not*
    check this for you.

Parameters
----------
value : array_like
    Values to insert into `self`.
side : {{'left', 'right'}}, optional
    If 'left', the index of the first suitable location found is given.
    If 'right', return the last such index.  If there is no suitable
    index, return either 0 or N (where N is the length of `self`).
sorter : 1-D array_like, optional
    Optional array of integer indices that sort `self` into ascending
    order. They are typically the result of ``np.argsort``.

Returns
-------
int or array of int
    A scalar or array of insertion points with the
    same shape as `value`.

    .. versionchanged:: 0.24.0
        If `value` is a scalar, an int is now always returned.
        Previously, scalar inputs returned an 1-item array for
        :class:`Series` and :class:`Categorical`.

See Also
--------
sort_values
numpy.searchsorted

Notes
-----
Binary search is used to find the required insertion points.

Examples
--------
>>> x = pd.Series([1, 2, 3])
>>> x
0    1
1    2
2    3
dtype: int64

>>> x.searchsorted(4)
3

>>> x.searchsorted([0, 4])
array([0, 3])

>>> x.searchsorted([1, 3], side='left')
array([0, 2])

>>> x.searchsorted([1, 3], side='right')
array([1, 3])

>>> x = pd.Categorical(['apple', 'bread', 'bread',
                        'cheese', 'milk'], ordered=True)
[apple, bread, bread, cheese, milk]
Categories (4, object): [apple < bread < cheese < milk]

>>> x.searchsorted('bread')
1

>>> x.searchsorted(['bread'], side='right')
array([3])

If the values are not monotonically sorted, wrong locations
may be returned:

>>> x = pd.Series([2, 1, 3])
>>> x.searchsorted(1)
0  # wrong result, correct would be 1
```

Here is the docstring from it's super class:
```
Find indices where elements should be inserted to maintain order.

.. versionadded:: 0.24.0

Find the indices into a sorted array `self` (a) such that, if the
corresponding elements in `value` were inserted before the indices,
the order of `self` would be preserved.

Assuming that `self` is sorted:

======  ================================
`side`  returned index `i` satisfies
======  ================================
left    ``self[i-1] < value <= self[i]``
right   ``self[i-1] <= value < self[i]``
======  ================================

Parameters
----------
value : array_like
    Values to insert into `self`.
side : {'left', 'right'}, optional
    If 'left', the index of the first suitable location found is given.
    If 'right', return the last such index.  If there is no suitable
    index, return either 0 or N (where N is the length of `self`).
sorter : 1-D array_like, optional
    Optional array of integer indices that sort array a into ascending
    order. They are typically the result of argsort.

Returns
-------
array of ints
    Array of insertion points with the same shape as `value`.

See Also
--------
numpy.searchsorted : Similar method from NumPy.
```
> You can build the doc strings and just post a screenshot of the HTML

Hi @WillAyd, I might mis-understand what you said. 

I create HTML via `python make.py html`, but because all files changed is under pandas.core and pandas.util, the html doc might not cover those things.
> The pandas.core, pandas.compat, and pandas.util top-level modules are PRIVATE. Stable functionality in such modules is not guaranteed.


> Would you mind if I create a new issue about docstring inheritance and propose my idea there? I am not 100% sure, but I think we already have some issues about this.
> 
> Open for suggestions and advices. What should I do next?

Thanks for the detail. It looks like we should definitely have a separate issue/PR for this discussion.
> I create HTML via `python make.py html`, but because all files changed is under pandas.core and pandas.util, the html doc might not cover those things.

am I correct in thinking the only published docstring affected by this PR is https://dev.pandas.io/docs/reference/api/pandas.Series.searchsorted.html? If so the screenshot of that should be OK.
> am I correct in thinking the only published docstring affected by this PR is https://dev.pandas.io/docs/reference/api/pandas.Series.searchsorted.html? If so the screenshot of that should be OK.

Thanks for your help @simonjayhawkins! Here is the screen shots. 

<img width="1920" alt="Screen Shot 2020-02-16 at 2 40 57 AM" src="https://user-images.githubusercontent.com/50187675/74600942-3e47ca80-5066-11ea-83f3-15ed28e6acc3.png">

<img width="1920" alt="Screen Shot 2020-02-16 at 2 42 47 AM" src="https://user-images.githubusercontent.com/50187675/74600944-40118e00-5066-11ea-92b4-307362d7cf85.png">


The screenshots are from locally generated documentation?
> The screenshots are from locally generated documentation?

Sorry for the confusion, but yes, it is locally generated doc.
Hello @HH-MWB! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:















There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-03-11 23:21:34 UTC
Hello @simonjayhawkins. It seems very hard to remove all `# type: ignore` because we might have to bind a variable to the function instead of the decorator. 

For use case of `@doc(f, ...)`, we should be able to get original templates from function `f`. If there are any other way might work, please let me know, and I will be more than happy to give it a try.

**However, we can reduce using `# type: ignore` from 6 times to 3 times.**

Here is my proposal by using class:
```python
class doc:
    """
    A decorator take docstring templates, concatenate them and perform string
    substitution on it.

    This decorator will add a variable "_docs" to the wrapped function to keep
    track the original docstring template for potential usage. If it should be
    consider as a template, it will be saved as a string. Otherwise, it will be
    saved as callable, and later user __doc__ and dedent to get docstring.

    Parameters
    ----------
    *args : str or callable
        The string / docstring / docstring template to be appended in order
        after default docstring under function.
    **kwags : str
        The string which would be used to format docstring template.
    """

    def __init__(self, *args: Union[str, Callable], **kwargs: str) -> None:
        self._docs: List[Union[str, Callable]] = []
        for arg in args:
            if hasattr(arg, "_docs"):
                self._docs.extend(arg._docs)  # type: ignore
            elif isinstance(arg, str) or arg.__doc__:
                self._docs.append(arg)

        self.parameters = kwargs

    def __call__(self, func: F) -> F:
        if func.__doc__:
            self._docs.insert(0, dedent(func.__doc__))

        func.__doc__ = "".join(
            [
                arg.format(**self.parameters)
                if isinstance(arg, str)
                else dedent(arg.__doc__)  # type: ignore
                for arg in self._docs
            ]
        )

        func._docs = self._docs  # type: ignore

        return func
```

We can use same idea to reduce `# type: ignore` from using function:
```python
def doc(*args: Union[str, Callable], **kwargs: str) -> Callable[[F], F]:
    """
    A decorator take docstring templates, concatenate them and perform string
    substitution on it.

    This decorator will add a variable "_docs" to the wrapped function to keep
    track the original docstring template for potential usage. If it should be
    consider as a template, it will be saved as a string. Otherwise, it will be
    saved as callable, and later user __doc__ and dedent to get docstring.

    Parameters
    ----------
    *args : str or callable
        The string / docstring / docstring template to be appended in order
        after default docstring under function.
    **kwags : str
        The string which would be used to format docstring template.
    """

    def decorator(func: F) -> F:
        @wraps(func)
        def wrapper(*args, **kwargs) -> Callable:
            return func(*args, **kwargs)

        # collecting docstring and docstring templates
        docs: List[Union[str, Callable]] = []
        if func.__doc__:
            docs.append(dedent(func.__doc__))

        for arg in args:
            if hasattr(arg, "_docs"):
                docs.extend(arg._docs)  # type: ignore
            elif isinstance(arg, str) or arg.__doc__:
                docs.append(arg)

        # formatting templates and concatenating docstring
        wrapper.__doc__ = "".join(
            [
                arg.format(**kwargs)
                if isinstance(arg, str)
                else dedent(arg.__doc__)  # type: ignore
                for arg in docs
            ]
        )

        # saving docstring templates list
        wrapper._docs = docs  # type: ignore

        return cast(F, wrapper)

    return decorator
```

Open for feed backs here. Which one is better? Any other improvement suggestions please?
> Open for feed backs here. Which one is better? Any other improvement suggestions please?

probably best to leave as a follow-up and have more detailed discussion and just keep the changes to the decorator in this PR limited to the changes needed for using a docstring from a method that is not decorated.
> > Open for feed backs here. Which one is better? Any other improvement suggestions please?
> 
> probably best to leave as a follow-up and have more detailed discussion and just keep the changes to the decorator in this PR limited to the changes needed for using a docstring from a method that is not decorated.

Sounds very reasonable. I will follow-up latter. 
> There is a trailing space and a mypy error, other than that looks good.

@datapythonista, @simonjayhawkins. Sorry to bother you. This [mypy error](https://github.com/pandas-dev/pandas/pull/31970/checks?check_run_id=492785846) makes me very confused.

In [this commit](https://github.com/pandas-dev/pandas/pull/31970/commits/7bf242e03bc10aeee4a7c6df515df23fbca23859), I removed `type: ignore` but everything works fine. However, now it shows the type error. I am confused why the original code can pass without error but the new code can't.

I know we can easily fix this by adding `type: ignore`, but I would like to know what case this problem and if there are a better way to solve that.
Great thanks @HH-MWB ! If you can do a follow up to fix the IndexOpsMixin import would be very much appreciated!
> Great thanks @HH-MWB ! If you can do a follow up to fix the IndexOpsMixin import would be very much appreciated!

Yes. I am thinking what is the best way of doing that. Definitely want to keep making contributing on this! Thanks.