ENH: Added DataFrame.compare and Series.compare (GH30429)
State: closed

Added `DataFrame.differences` and `Series.differences` methods. 

Have not yet added whatsnew entries. Will do so after review of API and behavior design.

A few design considerations open for discussion (among other things):
1. Index/column names: (self, other) vs (left, right) vs (old, new)
I used the (self, other) pair as I think they naturally match with the parameter names and the concept of comparing `self` with `other`. Although (left, right) or (old, new) may be slight more intuitive for users in some use cases, these new functions not necessarily compare two objects that are left and right (may be stacked on `axis=0`, or old and new.

1. Parameter name: axis
I feel axis may not be a very clear name, since we are stacking the output along this axis, rather than comparing objects along this axis. I considered using the name `stack_axis` or `orient` like `DataFrame.to_json` does.

1. I made the methods such that two NaNs are considered equal, hence not different. I think this should be the desired behavior since we are trying to compare two objects. However, under current implementation, a `None` will be considered equal to `np.nan` too. This is probably something we need to be careful for.

Let me know what you guys think. I'm ready to be scrutinized and criticized!

- [x] closes #30429
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Enhancement

Comments:
ping
ping
cc @TomAugspurger @jorisvandenbossche 
Hello @fujiaxiang! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:













There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-05-27 03:35:53 UTC
> What happens if the two objects have different labels? Does this align prior to checking the differences? Will something like
> 
> ```python
> a = pd.Series([np.nan, np.nan])
> b = pd.Series([])
> a.differences(b)
> ```
> 
> have no differences?

This implementation can only compare `DataFrame` or `Series` that are identically-labelled. It raises the following error when this is violated:
```python
ValueError: Can only compare identically-labeled Series objects
```

The part that raises this error is the first step 
```python
self == other
```
@jreback @TomAugspurger Anyone can help me with the Sphinx doc-string warning? I tried many many times but I couldn't figure out what is wrong. Stackoverflow doesn't seem to have an existing answer either.

I keep getting this message:
```
/home/runner/work/pandas/pandas/pandas/core/frame.py:docstring of pandas.DataFrame.differences:13: WARNING: Unexpected indentation.
/home/runner/work/pandas/pandas/pandas/core/frame.py:docstring of pandas.DataFrame.differences:14: WARNING: Block quote ends without a blank line; unexpected unindent.
/home/runner/work/pandas/pandas/pandas/core/series.py:docstring of pandas.Series.differences:13: WARNING: Unexpected indentation.
/home/runner/work/pandas/pandas/pandas/core/series.py:docstring of pandas.Series.differences:14: WARNING: Block quote ends without a blank line; unexpected unindent
```
then followed by 
```
build finished with problems, 4 warnings.
##[error]Process completed with exit code 1.
```
in the end.
@WillAyd thanks for the suggestion, let me try that
ping
ping
@WillAyd Your latest comment also appeared in the same thread as some of @jreback's earlier comments, so I replied there.
Basically I renamed the parameter `axis` to `align_axis` as I think your suggestion makes sense. I also added tests for ``MultiIndex" cases and the implementation works fine.
> what was the ratioale behind `align_axis`, we don't use this term anywhere else. what is wrong with `axis`?

My thought was that `axis` typically refers to how a function is applied against a frame. In this case things are applied element wise regardless and then `axis` or `align_axis` is used to determine how the construct the output
A bit late, but a general comment: I am fine with the *functionality* being added to pandas, but I have some reservations about the need to have it as a DataFrame *method* (alternative would be a function, either top-level or in one of the submodules, eg testing). 

We already have many methods, and the barrier to add a new one should be relatively high. In this specific case you also have the overlap with the existing `diff`, and the two might be easy to confuse.
Alternately we could get rid of the axis argument altogether with this. I think its ambiguous even along say the index if a user wants to see the diff between the frames "interwoven" or simply stacked one on top of another

Theres up to 4 ways to want the output aligned which one argument wouldn't cover, so I think could just pick one format and leave it to the user to reshape as desired
I'm open with any suggestion. Indeed it is important that we discuss about the details and make a careful decision.

Here is my personal preferences / opinion:
1. I think this feature is primarily used as a convenience method during data exploration / pipeline testing / debugging phase, rather than used in production. Users may want to compare two different sets of outputs to decide what to do next, then discard the results. This is somewhat similar to the method `describe`. Hence, handiness is probably more important than how "complete" the arguments are
1. Implementing the functionality as a method would be the most convenience (i.e handy) for users. Top-level function is still ok, since most users would already have done `import pandas as pd`.
1. Regarding the confusion with existing method `diff`, perhaps we can consider renaming the method to `compare`?
1. Regarding @WillAyd's mention of "4 ways to align the output", I would disagree with this. Simply stacking the two objects together defeats the original intent of this function, which is to compare two data objects. Also, we already have `pd.concat` to achieve that. Again, this goes back to the value of "handiness". I would value a parameter to allow me to easily control the orientation of output alignment when I simply want to have a convenient way to compare two data pieces during data exploration. I would not want to take other steps to reshape this output since I just want to take a look at the result before discarding it.

In summary, I think we should keep it as a method, but we can consider renaming it to "compare".
Let me know what you guys think.
guys, any thoughts?
generally lgtm ex @jorisvandenbossche comment https://github.com/pandas-dev/pandas/pull/30852#issuecomment-599128871
> In summary, I think we should keep it as a method, but we can consider renaming it to "compare".

+1 for compare
@simonjayhawkins I have renamed the method to `compare`. Can you review again?
@jreback can you review?
ping
ping
Do we have a test where the two objects are not aligned?

e.g.

```
df1 = pd.DataFrame(np.ones((3,3)))
df2 = pd.DataFrame(np.zeros((2, 1)))
```
> Do we have a test where the two objects are not aligned?
> 
> e.g.
> 
> ```
> df1 = pd.DataFrame(np.ones((3,3)))
> df2 = pd.DataFrame(np.zeros((2, 1)))
> ```

@mroeschke I have just added some. I'm expecting it to raise ``ValueError`` using `pytes.raises`.
question above and can you merge master.
> question above and can you merge master.

answered above
k thanks @fujiaxiang 

any other comments @pandas-dev/pandas-core  will merge in 24 hours otherwise.
thanks @fujiaxiang very nice!