BrandonFors/pandas-github-issues huggingface dataset:
ENH: Fix output of assert_frame_equal if indexes differ and check_like=True
State: closed

- [x] closes #37478 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

As described in #37478 This PR fixes some very misleading output when ```assert_frame_equal``` is called with differing index values and ```check_like=True```

I have added a ```check_order``` parameter to the ```assert_index_equal``` function.  This does essentially the same thing as the ```check_like``` parmeter on ```assert_frame_equal``` (but for indexes).  I think ```check_order``` gives a much clearer idea of what the parameter does, but it is a break from the previous naming.  I am unsure whether I should have stuck with ```check_like``` for the new parameter.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
> 
> 
> Thanks @amilbourne for the PR! Looks good
> 
> I don't know what the pandas policy is on API changes for the testing module (I think it's private but I think it gets used outside our testing suite) so one thing you might need to get agreement on
> 
> Related to the above if we regard this as a user-facing change you will need to add a whatsnew note

I guess that ``assert_index_equal`` is indeed a part of public API, since the docs are available here https://github.com/pandas-dev/pandas/pull/37479#pullrequestreview-519321722
@ivanovmg, @arw2019  I agree that ```assert_index_equal``` is a part of public API, I just wasn't sure which changes needed a whatsnew note.  I will add one.  Thanks.
I made the suggested changes and added a unit test for the new `check_order` param.
The build is failing due to a failing test, but it doesn't seem to be anything to do with the changes I made, so I'm assuming that is an anomaly in the build (or a flakey test or something).

The failing test is:
test_slice_irregular_datetime_index_with_nan from pandas/tests/indexing/test_partial.py

This test is also failing for other unrelated PRs, so I don't think it was me :-)
@jreback I moved the parameter as asked and rebased.  The build is failing on
`pandas/tests/plotting/test_datetimelike.py::TestTSPlot::test_format_timedelta_ticks_wide`
But I don't think that is related to anything I changed.
thanks @amilbourne very nice
CI: 32 bit maybe_indices_to_slice
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: 32bit
description: 32-bit systems

Comments:
hmm you are backing this to fix the 32-bit tests? does this have a perf hit?

Not sure yet, still trying to get this passing
got some more troubleshooting to do, but in the interim, some greenish: #37365, #37402, #37449, #37451, #37456, #37470, #37472, #37475
and green but less trivial #37425, #37436, #37423, #37425, #37179, #37164, #34683 

The deluge is going to slow to a crawl since the next task on my list is a doozy: figuring out why the categorical validators all behave slightly differently.
alright, got the 32bit working
> can u run asvs

i can start them running overnight.  this should be totally neutral on 64bit since np.intp is an alias for np.int64
forgot to run asvs overnight, just started them now.
ran with `-b index`

```
BENCHMARKS NOT SIGNIFICANTLY CHANGED.
```
@jreback if this is ready to merge will backport, otherwise can just skip failing 32 bit test on MacPython.
thanks @jbrockmendel 
@meeseeksdev backport 1.1.x
Backport PR #35875 on branch 1.1.x (COMPAT: Ensure rolling indexers return intp during take operations)
State: closed

Backport PR #35875: COMPAT: Ensure rolling indexers return intp during take operations

Labels:
name: 32bit
description: 32-bit systems

Comments:
just `test_slice_irregular_datetime_index_with_nan` on 32bit outstanding which needs to be fixed on master and backported.
TST/REF: finish collecting sample tests
State: closed

Parametrizes a few tests from the frame.methods.test_sample file, splits up giant tests from the generic file.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks, just a note before you go all in on moving the generic tests. we ought to consider just splitting these back to series/dataframe. yes now they are essentiallly in 2 places, but we already have a ton like this. the original reason for generic was to also test panel, well that's over so something to consider.
BUG: Fix for #37454: allow reversed axis when plotting with TimedeltaIndex
State: closed

- [x] closes #37454
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
@jreback, looks like everything is working except some unrelated failures with FutureWarning.
@jreback, I think this is good to go
thanks @theavey very nice
BUG: Correct crosstab for categorical inputs
State: closed

Change catch types to reflect error changes

closes #37465

- [X] closes #37465
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry (N/A: Never released, caught in statsmodels pre-release testing)


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Retrigger CI
@bashtage if u can add a note to 1.1.4 can push this in

ping on greenish 
@jreback green and note added. 
sorry i misread the OP

@bashtage can u move the note to 1.2
@jreback I confirmed that this is not a bug in the 1.1.x branch.

I have left out a release note since this code has never been released.  This seems like the right thing to do, isn't it?

For posterity, if needed, the line is

`- Fixed regression in :meth:`crosstab` that failed when the input series are categorical (:issue:`37465`)`

so I can add to 1.20 if required.

Hello @bashtage! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-10-31 00:09:24 UTC
thanks @bashtage
BUG: Fix .hist and .plot.hist when passing existing figure (#37278)
State: closed

- [x] closes #37278
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Labels:

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@charlesdong1991 can you take a look
closing as stale, if you want to continue working, please ping.
Backport PR #35898 on branch 1.1.x: CI: docker 32-bit linux build
State: closed

Backport PR #35898

Labels:
name: 32bit
description: 32-bit systems

Comments:
32 bit failures 
```
=========================== short test summary info ============================
FAILED pandas/tests/indexing/test_partial.py::TestPartialSetting::test_slice_irregular_datetime_index_with_nan
FAILED pandas/tests/window/test_grouper.py::TestGrouperGrouping::test_groupby_rolling_string_index
FAILED pandas/tests/window/test_grouper.py::TestGrouperGrouping::test_groupby_rolling_no_sort
= 3 failed, 69851 passed, 4429 skipped, 14962 deselected, 1047 xfailed, 6 xpassed, 12 warnings in 758.55s (0:12:38) =
```

This matches https://dev.azure.com/pandas-dev/pandas-wheels/_build/results?buildId=46211&view=logs&j=bb1c2637-64c6-57bd-9ea6-93823b2df951&t=9c62e7ff-391e-517d-ede1-af63bc6a25da with `test_groupby_nat_exclude` and `test_info_int_columns` fixed here.

will merge this and address these failures separately

`test_groupby_rolling_string_index` and `test_groupby_rolling_no_sort` only occuring on 1.1.

`test_slice_irregular_datetime_index_with_nan` is failing on master and only appeared on https://github.com/MacPython/pandas-wheels/pull/106 after the last commit so is recent and not picked up #35898
Backport PR #37432 on branch 1.1.x
State: closed

(cherry picked from commit 52925335fe6d5f1ea6c79bb03da70654d9be9668)

Tried to merge it. Hopefully nothing was broken in the process. Run the indexing tests locally, but have to wait for ci

cc @simonjayhawkins 

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
Thanks @phofl
BUG: BusinessDay.apply_index with offset
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Frequency
description: DateOffsets

Comments:
is this a bug?
> is this a bug?

yes, will add whatsnew
updated+green
CI: Troubleshoot PY38 windows build
State: closed

Some of the error messages look like a OOM, so this is just trying cutting down the number of tests collected/run by ~40%

Labels:
name: CI
description: Continuous Integration
name: Window
description: rolling, ewma, expanding

Comments:
@simonjayhawkins thoughts on a better way to approach this.  the win38 build has passed twice in a row here, suggesting this might be on to something
In case this helps with troubleshooting: on my local Windows 10 machine on a py38 env, the test suite fails with a stack overflow exception. However, when I run tests folder-by-folder, the folder it had failed on passes. Looks like some kind of memory leak accumulating stuff on stack to me.
The windows py38 build has succeeded here 4ish times in a row, making me think the issue is just "the test suite is too big for that particular build"

Thoughts on more elegant ways to solve this cc @pandas-dev/pandas-core ?
so we are skipping high memory tests? ok sounds fine to me to merge (is there any pure debugging code here?)
> so we are skipping high memory tests? ok sounds fine to me to merge (is there any pure debugging code here?)

ATM this PR isnt just skipping high-memory tests, its also very hackily running a subset of tests
can u try skipping some files instead (until it reliably passes)

py.test --ignore=somedir

would be a more less fragile i think
> can u try skipping some files instead (until it reliably passes)

@AlexKirko can you give this a shot locally before I try it on the CI?
this looks reasonable, merging will create an issue to see if can troubleshoot at some point.
ENH: Improve numerical stability for Pearson corr() and cov()
State: closed

- [x] closes #37448
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Floating number issues when summing the same number often enough...

Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
But what if you're trying to get the correlation of series with very small floats? [1.0e-20, 2.0e-20, 3.0e-20] is perfectly correlated with itself, but has a variance smaller than 1e-15 ...
Hm yes, that a good point. Have to add tests for these cases and change my fix.
Welford does the trick. Also saves us one for loop. Running asvs
asv results

```
       before           after         ratio
     [d321be6e]       [11e8f250]
     <20816~1^2>       <37448~1> 
+      7.93±0.1ms      9.00±0.03ms     1.14  stat_ops.Correlation.time_corr_wide('pearson')
-     13.5±0.04ms       10.5±0.3ms     0.77  stat_ops.Correlation.time_corr_wide_nans('pearson')
```
Failure seems unrelated
cc @jreback 

  before           after         ratio
     [0647f02c]       [16598811]
     <29485^2>        <37448>   
+      8.77±0.1ms       10.4±0.3ms     1.18  stat_ops.Correlation.time_corr_wide('pearson')


asv results
CLN: Breakup agg
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Extracted dictionary parts to `agg_dict_like`, and changed from `isinstance(arg, dict)` to `is_dict_like(arg)`. Also renamed `aggregate_multiple_funcs` to `agg_list_like`.

Labels:
name: Clean
name: Apply
description: Apply, Aggregate, Transform, Map
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Docstrings aren't quite right here; obj can be other types. Hold off on merging.
Tests pass when adding the following line to aggregation.aggregate:

    assert isinstance(obj, (Series, DataFrame, SeriesGroupBy, DataFrameGroupBy, BaseWindow, Resampler))

This is a bit ghastly, going to just leave the description as generic unless this specificity is desired.
@jreback green-ish
PERF: faster dir() calls
State: closed

I experience slow tab-completion in iPython, so I've optimized `dir` calls in pandas, e.g.

```python
>>> n = 100_000
>>> ser = pd.Series(['a'] * n]
>>> %timeit dir(ser)
3.73 ms ± 34.7 µs per loop  # master
253 µs ± 4.3 µs per loop  # this PR
```

It does this by caching the output for the info axis, when the info axis may have string, and returning an empty set for info axes that cannot have strings (numeric indexes etc.)

The above didn't actually improve the subjective speed of tab completion for me, so the problem there probably is in Ipython, but the change in this PR can't hurt either.

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
thanks @topper-123
Upgrade pygrep, use in-built rst-directive-colons
State: closed

I got a PR in to pre-commit/pygrep-hooks to make the "incorrect-sphinx-directives" hook a built-in one (as it's not specific to pandas) :tada: So, let's use it?

They also have another useful hook (`rst-inline-touching-normal`), which caught a couple of errors which I've fixed below.

The reason why I've kept `rst-backticks` to just `rst` files (as opposed to `types: [text]` like the other two) is that in docstrings, the numpy guide instructs to use single backticks for variable (though if it was ever desirable to change that, then this hook would make it pretty easy to check for consistency)

Labels:
name: CI
description: Continuous Integration
name: Code Style
description: Code style, linting, code_checks

Comments:
great thanks @MarcoGorelli very nice!
REGR: fix rank algo for read-only data
State: closed

Closes #37290

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
hmm i think this is actually failing see the traivs CI: https://travis-ci.org/github/pandas-dev/pandas/jobs/739214922

```
>   ranked_mat[:, i] = rank_1d(mat[:, i])
E   TypeError: Argument 'in_arr' has incorrect type (expected numpy.ndarray, got pandas._libs.algos._memoryviewslice)
pandas/_libs/algos.pyx:347: TypeError
_ TestDataFrameCorr.test_corr_nullable_integer[spearman-other_column2-nullable_column1] _
[gw0] linux -- Python 3.7.9 /home/travis/miniconda3/envs/pandas-dev/bin/python
self = <pandas.tests.frame.methods.test_cov_corr.TestDataFrameCorr object at 0x7fcbc4f5a4d0>
nullable_column = <IntegerArray>
[1, 2, <NA>]
Length: 3, dtype: Int64
other_column = array([ 1.,  2., nan]), method = 'spearman'
    @td.skip_if_no_scipy
    @pytest.mark.parametrize(
        "nullable_column", [pd.array([1, 2, 3]), pd.array([1, 2, None])]
    )
    @pytest.mark.parametrize(
        "other_column",
        [pd.array([1, 2, 3]), np.array([1.0, 2.0, 3.0]), np.array([1.0, 2.0, np.nan])],
    )
    @pytest.mark.parametrize("method", ["pearson", "spearman", "kendall"])
    def test_corr_nullable_integer(self, nullable_column, other_column, method):
        # https://github.com/pandas-dev/pandas/issues/33803
        data = DataFrame({"a": nullable_column, "b": other_column})
>       result = data.corr(method=method)
pandas/tests/frame/methods/test_cov_corr.py:190: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pandas/core/frame.py:8300: in corr
    correl = libalgos.nancorr_spearman(mat, minp=min_periods)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>   ranked_mat[:, i] = rank_1d(mat[:, i])
E   TypeError: Argument 'in_arr' has incorrect type (expected numpy.ndarray, got pandas._libs.algos._memoryviewslice)
```
looks like you need to update the types in nancorr_spearman and rank_2d (which call rank_1d)
Thanks for the note, indeed needed to update some other places where rank_1d is called
thanks @jorisvandenbossche 
@meeseeksdev backport 1.1.x
REGR: fix groupby std() with nullable dtypes
State: closed

Fixes #37415


Labels:
name: Groupby
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
At some point we need to properly support nullable dtypes in the cython groupby operations (by passing+using the mask), but for now this ensure it at least works (although performance will not be ideal)
> (Did try to bisect to be sure of changes that caused regression but 1.0.0.rc raised `TypeError: cannot safely cast non-equivalent float64 to int32`)

rerun bisect with updated code sample for runner see https://github.com/pandas-dev/pandas/issues/37415#issuecomment-717182856, so fix seems reasonable
Ah, yes, sorry I should have mentioned, but I already narrowed it down to that PR. Thanks anyway!

Ci failure is unrelated I think
thanks @jorisvandenbossche 
@meeseeksdev backport 1.1.x
For follow-ups, I opened https://github.com/pandas-dev/pandas/issues/37493 for general "native" support for masked arrays in groupby algos, and https://github.com/pandas-dev/pandas/issues/37494 for the specific issue to use the correct dtype for the result.
Fix regression in iloc with boolean list
State: closed

- [x] closes #36741
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Run tests in ``indexes`` and ``indexing`` locally without failure.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
cc @simonjayhawkins @jreback 

Is this early enough to get into 1.1.4?
@phofl it appears the test got lost in the merge
Thanks very much. That was not that clever :)
code fix is ok, but this is going to be tricky to backport as this code as changed a bit.

cc @jbrockmendel merge if looks good to you.
this is fine, thanks @phofl 

not sure how hard the actual backport to 1.1.4 is
@meeseeksdev backport 1.1.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.1.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 52925335fe6d5f1ea6c79bb03da70654d9be9668
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #37432: Fix regression in iloc with boolean list'
```

4. Push to a named branch :

```
git push YOURFORK 1.1.x:auto-backport-of-pr-37432-on-1.1.x
```

5. Create a PR against branch 1.1.x, I would have named this PR:

> "Backport PR #37432 on branch 1.1.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
yep this needs manual. @phofl @simonjayhawkins
Failing test_missing_required_dependency in pandas-wheels
State: closed

- [ ] closes #33999
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

tested at https://github.com/simonjayhawkins/pandas-release/actions/runs/329803554 ( patched from Gist https://gist.github.com/simonjayhawkins/f26d3e26fd55159299b38204f7f3295c and skips for test_missing_required_dependencies removed for conda and pip tests)

following merge can also remove from https://github.com/MacPython/pandas-wheels/blob/5543626b261f5cb55cebfb300c156718c1796ce3/config.sh#L34

Labels:
name: CI
description: Continuous Integration

Comments:
@meeseeksdev backport 1.1.x
PERF: reverted change from commit 7d257c69 to solve issue  #37081
State: closed

reverted change from commit 7d257c697 to solve issue  #37081

- [x] closes #37081
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
Is it the check itself (e.g evaluating `self._mgr.any_extension_types`) that is causing the slowdown?

If this is reverted, everything else in the referenced commit can be reverted along with it.
> Is it the check itself (e.g evaluating `self._mgr.any_extension_types`) that is causing the slowdown?
> 
> If this is reverted, everything else in the referenced commit can be reverted along with it.

It's not the check, it's using `df._mgr.reduce` instead of `func(values)`, the added condition only make it visible by forcing the benchmark to use of df._mgr.reduce instead of func(values). If that's really the case for the other dtypes, then i guess we may have a deeper issue (only checked for numerical df which is the example used in the issue).

PS : Sorry for the late response had a rough week.
> flake8 is failing pre-commit also you likely have to change some tests to get this to pass

`git diff upstream/master -u -- "*.py" | flake8 --diff` returns no error (in local)

`./ci/code_checks.sh` seems to run fine (no errors in local).
see the error in the ci: https://github.com/pandas-dev/pandas/pull/37426/checks?check_run_id=1343971174
pls post the revised asv's as well
> > flake8 is failing pre-commit also you likely have to change some tests to get this to pass
> 
> `git diff upstream/master -u -- "*.py" | flake8 --diff` returns no error (in local)

Unfortunately that doesn't catch all issues - from the contributing guide: "One caveat about git diff upstream/master -u -- "*.py" | flake8 --diff: this command will catch any stylistic errors in your changes specifically, but be beware it may not catch all of them. "

> `./ci/code_checks.sh` seems to run fine (no errors in local).

flake8 is no longer in there. If you enable pre-commit in your workspace (`pre-commit install`) it'll find these kinds of errors before you make your commit.

Or you can just run `flake8` from the command-line on the files you've changed.

Even better, you can run `pre-commit run flake8 --files <files you've changed>` (or `pre-commit run flake8 --all-files`)
thanks @ukarroum
BUG: DataFrame.min/max dt64 with skipna=False
State: closed

- [x] closes #36907
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Datetime
description: Datetime data dtype

Comments:
whatsnew added + greenish
REF: avoid special case in DTA/TDA.median, flesh out tests
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Also fixes currently-incorrect empty case.

Labels:
name: Datetime
description: Datetime data dtype

Comments:
can you rebase
Rebased + greenish
thanks