PERF: fix assert_frame_equal can be very slow
State: closed

- [ ] closes #38183
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Add kwarg ``check_index`` into ``assert_series_equal`` to allow elimination of multiple index checking (for each column) in ``assert_frame_equal``.

Performance wise.

```
import pandas as pd
import numpy as np
from string import ascii_letters

idx = np.random.choice(np.array(list(ascii_letters)), size=10000).astype(object)
a = np.random.random_integers(0, 100, (10000, 1000))

%time pd.testing.assert_frame_equal(pd.DataFrame(a), pd.DataFrame(a.copy()))
Wall time: 319 ms

%time pd.testing.assert_index_equal(pd.Index(idx), pd.Index(idx.copy()))
Wall time: 993 µs

%time pd.testing.assert_frame_equal(pd.DataFrame(a, index=idx), pd.DataFrame(a, index=idx))
Wall time: 281 ms
```

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Looks like green.
hmm can you merge master (this looks like it's running some older code)
Did that.
But presumably we have the same irrelevant failures anyway.
> Did that.
> But presumably we have the same irrelevant failures anyway.

it was some other errors.

but in spite of this change i am not sure this actually fixes things. (tests times are about the same)
> tests times are about the same

i doubt we have any/many tests that are as poorly behaved as the example from #38183
> 
> 
> > Did that.
> > But presumably we have the same irrelevant failures anyway.
> 
> it was some other errors.
> 
> but in spite of this change i am not sure this actually fixes things. (tests times are about the same)

Execution of the following command improved.
```
%time pd.testing.assert_frame_equal(pd.DataFrame(a, index=idx), pd.DataFrame(a, index=idx))
```
from 50 seconds to 280 something milliseconds (#38183).
I can think of a test ensuring that ``assert_frame_equals`` calls ``assert_index_equal`` only once.
This is not a performance test, but for now we see that the problem with the performance observed in the original issue is due to multiple calls of the latter one.
What do you think of this approach?
@jreback @jbrockmendel 

In fact, ``assert_index_equal`` is called twice: first time for checking index equivalence and second - for columns.
Added this test.
> > > Did that.
> > > But presumably we have the same irrelevant failures anyway.
> > 
> > 
> > it was some other errors.
> > but in spite of this change i am not sure this actually fixes things. (tests times are about the same)
> 
> Execution of the following command improved.
> 
> ```
> %time pd.testing.assert_frame_equal(pd.DataFrame(a, index=idx), pd.DataFrame(a, index=idx))
> ```
> 
> from 50 seconds to 280 something milliseconds (#38183).

ok maybe something else is stil take time
its this build: https://travis-ci.org/github/pandas-dev/pandas/jobs/747040921 that is *really* really slow, used to be like 30min.
pre-commit checks are faling
We normally have a high bar for adding parameters to the public api. Since `assert_series_equal` is public, should we give this more consideration.

we could just have an internal `assert_series_equal` for now with the extra parameter, and dispatch from the public `assert_series_equal` with the api unchanged.
> We normally have a high bar for adding parameters to the public api. Since `assert_series_equal` is public, should we give this more consideration.
> 
> we could just have an internal `assert_series_equal` for now with the extra parameter, and dispatch from the public `assert_series_equal` with the api unchanged.

-1 on this (which was already suggested), this add even more complexity in an already complex world. we *already* have a `check_index` keyword so nbd here.
can you show how this is faster, specifically  are things decreasing in the duration? which builds are faster?
> 
> 
> can you show how this is faster, specifically are things decreasing in the duration? which builds are faster?

I compared ``iloc`` vs ``_ixs`` and did not notice much difference. So, I reverted back to ``iloc``.
thanks @ivanovmg