BrandonFors/pandas-github-issues huggingface dataset:
ENH: Clarify error message when reindexing fails (#42000)
State: closed

- [x] closes #42000
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
@jreback Updated. Not green, but the failing check seems unrelated?
> @jreback Updated. Not green, but the failing check seems unrelated?

should now be fixed
i'll merge with a 1.4 milestone. but a trivial change so could backport.
Thanks @aberres
ENH: `Styler.apply(map)_index` made compatible with `Styler.to_excel`
State: closed

This is a follow-on to #41893.

It makes the new `Styler.apply_index` and `Styler.applymap_index` methods compatible with `Styler.to_excel`. If the methods are not used the output reverts to the old format.

![Screen Shot 2021-06-14 at 10 58 47](https://user-images.githubusercontent.com/24256554/121866799-93016100-ccff-11eb-8bc8-124fd768e550.png)

**MultiIndex Case**

```python
midx = pd.MultiIndex.from_product([['ix', 'jy'], [0, 1], ['x3', 'z4']])
df = pd.DataFrame([np.arange(8)], columns=midx)
def highlight_x(s):
    return ["background-color: yellow;" if 'x' in v else "" for v in s]
df.style.apply_index(highlight_x, axis="columns", level=[0, 2]).to_excel("mi_test.xlsx")
```

![Screen Shot 2021-08-12 at 09 24 15](https://user-images.githubusercontent.com/24256554/129155325-1fe07a25-3f82-4063-8eba-b31ba725ce56.png)



# NOTE on TESTS

The tests for styler_to_excel were broken and were xfailed out of production in 2019. Since they dont work I have replaced them with some other, cleaner, tests and add tests relevant to this PR also.

Closes #25351


Labels:
name: Enhancement
name: IO Excel
description: read_excel, to_excel
name: Styler
description: conditional formatting using DataFrame.style

Comments:
I discovered #25351 where basically the tests for `Styler.to_excel` had all been removed, and were pretty ugly anyway.

So I have replaced these with a minimalist set of non-comprehensive tests, and also added the tests relevant to this PR.
@jreback ping green
thanks @attack68
⬆️ UPGRADE: Autoupdate pre-commit config
State: closed

<!-- START pr-commits -->
<!-- END pr-commits -->

## Base PullRequest

default branch (https://github.com/pandas-dev/pandas/tree/master)

## Command results
<details>
<summary>Details: </summary>



</details>

## Changed files
<details>
<summary>Changed file: </summary>



</details>

<hr>

[:octocat: Repo](https://github.com/technote-space/create-pr-action) | [:memo: Issues](https://github.com/technote-space/create-pr-action/issues) | [:department_store: Marketplace](https://github.com/marketplace/actions/create-pr-action)

Labels:

Comments:
putting this off for now
> putting this off for now

This is not already done? #38376
yeah not sure why it wasn't picked up here
The isort is also upgraded. If that make significant changes, maybe worth doing so that we can backport now while 1.3.x and master are not significantly diverged so that we can reduce the potentially reduce failures on autobackports in the future.
a quick scan of the diff. didn't notice any changes to imports.
> The isort is also upgraded. If that make significant changes, maybe worth doing so that we can backport now while 1.3.x and master are not significantly diverged so that we can reduce the potentially reduce failures on autobackports in the future.

in this PR it's pyupgrade that's updated, but the change is really minor, isort is still at 5.8.0

---

Maybe I need to delete the branch for this to work properly, will try now
> in this PR it's pyupgrade that's updated, but the change is really minor, isort is still at 5.8.0

thanks @MarcoGorelli misread the diff
STYLE: Added from __future__ import annotations to required files
State: closed

- [X] closes #41901 
- [ ] No tests added
- [X] All linting passed (used `isort` to add imports)
- What's New:

Built a [script](https://gist.github.com/aacosta13/eb1cb4b22ece354c60f0e7aab90137b0) using `isort` to add `from __future__ import annotations` to necessary files as referenced by issue #41901.

Labels:
name: Stale

Comments:
@MarcoGorelli sorry about that, could have sworn that file was marked as deleted before the PR. Just removed it, feel free to let me know if there is anything else to fix or take note of.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
closing as stale, feel free to let us know if you'd like to continue work on this
CI: troubleshoot py310 build
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: CI
description: Continuous Integration
name: Python 3.10

Comments:
Hey @jbrockmendel Just giving the heads up that the Python 3.10 dev build was actually "fake" green and can be green even if failing. (was actually failing even when it was "green"). I'm trying to address that in #41595, so I guess you can ignore failures listed on the issue #41940 in this PR.

Also, FYI the Python dev build on tests on a single core, so you might find it helpful to change add an ``-n auto`` to the Pytest command. If you want to have CI fail on a failure, you can just remove ``continue-on-error`` flag.
Thanks @lithomas1.  IIUC what we really want is allow-failure which GH actions doesn't support (https://github.com/actions/runner/issues/2347).  For now I'll be happy if non-breaking PRs get a little green checkmark.
@jbrockmendel there's some mypy failures
> Also, FYI the Python dev build on tests on a single core, so you might find it helpful to change add an `-n auto` to the Pytest command.

The last time I tried, only single core works. We probably should try pytest-xdist master now.

Remaining failures look unrelated
tell you what. i like green, so if you can do a followup for the name cleaning (which we can backport to 1.3)
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
DEPS: drop Python 3.7 and NumPy 1.17 from tests (#41678)
State: closed

closes #41678

Most of the new minimum support versions were released between Oct 14, 2019, Python 3.8's initial release date, and June/July 2020.

Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Deprecate
description: Functionality to remove in pandas
name: Dependencies
description: Required and optional dependencies

Comments:
Thanks @fangchenli for getting this started. To close the issue, are you planning to update the release notes in this PR too?
> Thanks @fangchenli for getting this started. To close the issue, are you planning to update the release notes in this PR too?

I'll update the doc at last.
could update this to `--py38-plus`

https://github.com/pandas-dev/pandas/blob/0b68d87a4438a13f14a2ed5af2e432df02eb0b2c/.pre-commit-config.yaml#L59-L63
> could update this to `--py38-plus`

I think maybe code changes should be a follow-on. This PR should probably just be the ci setup and docs.
> update the release notes in this PR too?

I should have said install.rst too.

I think it would be easier for reviewers to see the significant changes, but I also appreciate that while getting things working is an additional burden, so add when ready.


once we do this, we won't be releasing from master until at least 26 December. Historically, we have had 6 monthly releases (except for 0.22, which was a special case for a api breaking change) so shouldn't be an issue. Are we ready for this?
> once we do this, we won't be releasing from master until at least 26 December. Historically, we have had 6 monthly releases (except for 0.22, which was a special case for a api breaking change) so shouldn't be an issue. Are we ready for this?

yep this is fine this early. going to have a look.
https://github.com/pandas-dev/pandas/pull/41989/checks?check_run_id=2984645939 random failure?
> https://github.com/pandas-dev/pandas/pull/41989/checks?check_run_id=2984645939 random failure?

Looks like a flaky test.
thanks @fangchenli very nice!
BUG: nan-objects lookup fails with Python3.10
State: closed

- [x] closes #41953
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them


This builds upon #41952, which should be integrated first.

We no longer can assume that the CPython's hash function for elements with NaN (float, complex, tuple) will do the right thing (see https://github.com/python/cpython/commit/a07da09ad5bd7d234ccd084a3a0933c290d1b592) and thus need to replace the default CPython's hash-function with our own, similar to the way we handle equality-operator.

We need to use old CPython hash functions for `float` and `complex`, because we would like to keep the equivalency `int(k)==float(k)==complex(k, 0.0)`.

We have more freedom with tuple's hash-function, but still use CPython's implementation, so we get old hash-values.


Labels:
name: Python 3.10

Comments:
No significant changes in performance in the asv-bench.
thanks @realead 

the reason merging these to 1.3 is since 3.10 will be out before 1.4/2.0 :->
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
TST: Un-xfail tests on numpy-dev
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Makes 3.8 npdev green. 

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
@simonjayhawkins @jreback are we still in a merge moratorium?  i think this fixes one of our broken builds
no this is fine (just needing to backport any 1.3 things)
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
@meeseeksdev backport 1.2.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.2.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 e3d57c983ba9c9e5ba659f48b1821d929a6a6c16
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #41987: TST: Un-xfail tests on numpy-dev'
```

4. Push to a named branch :

```
git push YOURFORK 1.2.x:auto-backport-of-pr-41987-on-1.2.x
```

5. Create a PR against branch 1.2.x, I would have named this PR:

> "Backport PR #41987 on branch 1.2.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
DOC: update array-like parameters if scalars accepted
State: closed

- [x] based on discussion in #41807 
- [x] tests added / passed
  - N/A since only doc changes
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
  - N/A since only doc changes
- [x] whatsnew entry
  - Not needed

Based on discussion, I went through all docstrings that had the regex `":.*array-like"` and added a scalar as an accepted argument where appropriate.



Labels:
name: Docs

Comments:
thanks @Dr-Irv
Backport PR #41966 on branch 1.3.x (CI: activate azure pipelines/github actions on 1.3.x)
State: closed

Backport PR #41966: CI: activate azure pipelines/github actions on 1.3.x

Labels:
name: CI
description: Continuous Integration

Comments:
i'll close for as don't wanted it merged between tagging locally and pushing tag. will reopen after rc1
Backport PR #41981 on branch 1.3.x (BLD: Update MANIFEST.in)
State: closed

Backport PR #41981: BLD: Update MANIFEST.in

Labels:
name: Build
description: Library building on various platforms

Comments:
no idea why ci doesn't run. #41966 has been merged. when doing the same for 1.2.x, #38360 we also backported, so maybe is necessary

will be testing 1.3.x before release so merging this.
BLD: Update MANIFEST.in
State: closed

xref https://github.com/pandas-dev/pandas/pull/41977#issuecomment-860171367

Labels:
name: Build
description: Library building on various platforms

Comments:
The pip-test and conda-test steps of the release process pass with this patch.
> The pip-test and conda-test steps of the release process pass with this patch.

I should have said locally.

I think the reason that the ci action https://github.com/simonjayhawkins/pandas-release/actions/runs/932775977 is failing for the conda-test maybe due to the conda recipe being removed from the pandas repo in 1.3. This is a separate issue and maybe an error in the "unofficial" ci scripts there. Will investigate further.



> I think the reason that the ci action https://github.com/simonjayhawkins/pandas-release/actions/runs/932775977 is failing for the conda-test maybe due to the conda recipe being removed from the pandas repo in 1.3. This is a separate issue and maybe an error in the "unofficial" ci scripts there. Will investigate further.

the scripts are failing for 1.2.5 now too. so something else has changed in the last few days. not relevant for this PR.
yeah we removed the conda recioe

i think we can remove this action as well as the conda recioe is maintained externally 
> i think we can remove this action as well as the conda recioe is maintained externally

the action is maintained by me on my personal branch to avoid surprises when doing the release and as a basis for the release automation #21050 and #36704

but that's not relevant to this PR. OK to merge?
sure
REF: Refactor assert_index_equal
State: closed

There are currently some issues in `_check_dtypes` in `assert_index_equal` related to #41153:
* `assert_attr_equal("dtype", left, right, obj=obj)` currently isn't run if `check_categorical` is False (even if the indexes are not CategoricalIndexes)
* `assert_class_equal` with `exact='equiv'` should not be restricted to accept equivalence of `Int64Index` and `RangeIndex`, but should accept equivalence of all subclasses of `NumericIndex` (in preparation for #41153).

So, comparing numeric indexes with `assert_index_equal(idx1, idx2, exact="equiv")`  we should for otherwise equal content have:
* RangeIndex and Int64Index should pass, because they're both numeric indexes and have the same dtype
* Int64Index and Float64Index should  fail, because even though they're both numeric indexes, they have different dtypes
* Int64Index and NumericIndex[int64] should pass, because they're both numeric indexes and have the same dtype
* RangeIndex and NumericIndex[int64] should pass, because they're both numeric indexes and have the same dtype
* Float64Index and NumericIndex[int64] should fail, because even though they're both numeric indexes, they have different dtypes

Conversely, with `exact=True` class comparison should be strict, so:
* RangeIndex and Int64Index should fail, because they're different classes, even though they have the same dtype
* Int64Index and NumericIndex[int64] should fail, because they're different classes, even though they have the same dtype
* etc.

This refactoring does not change behaviour in master, but will make #41153 easier.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Index
description: Related to the Index class or subclasses

Comments:
Everything passes localy, so seems like the CI failurees are unrelated.
Rebased.
thanks @topper-123
BLD: ignore multiple types of file in wheel
State: closed

xref #40169 


Labels:
name: Build
description: Library building on various platforms

Comments:
lgtm

@simonjayhawkins give this a try and merge / backport if good
thanks @fangchenli 
Thanks @fangchenli 

will open PR on pandas-wheels to test
> will open PR on pandas-wheels to test

https://github.com/MacPython/pandas-wheels/pull/135
we still need the pxi.in template files

```
    File "/tmp/pip-build-env-jawd5f8v/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 135, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-jawd5f8v/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 259, in run_setup
      self).run_setup(setup_script=setup_script)
    File "/tmp/pip-build-env-jawd5f8v/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 150, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 650, in <module>
      ext_modules=maybe_cythonize(extensions, compiler_directives=directives),
    File "setup.py", line 422, in maybe_cythonize
      build_ext.render_templates(_pxifiles)
    File "setup.py", line 91, in render_templates
      with open(pxifile) as f:
  FileNotFoundError: [Errno 2] No such file or directory: 'pandas/_libs/algos_common_helper.pxi.in'
```
The wheels on pandas-wheels look to be building OK, these must be built from (github) source. the failure in previous comment is from https://github.com/simonjayhawkins/pandas-release/actions/runs/932775977 (built from sdist)
BUG: UInt64Index.where with int64 value
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Index
description: Related to the Index class or subclasses

Comments:
bug? needs a release note. I would put in 1.3 for now. (it can be moved if needed after @jreback reviews)

using semver we can make backwards compatible bug fixes in patch releases but we tend to only fix regressions.

from https://semver.org/

Given a version number MAJOR.MINOR.PATCH, increment the:

MAJOR version when you make incompatible API changes,
MINOR version when you add functionality in a backwards compatible manner, and
PATCH version when you make backwards compatible bug fixes.


this is fine for 1.3
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
PERF: indexing on UInt64Index with int64s
State: closed

- [x] closes #41873
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Performance
description: Memory or execution speed performance
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
This fixes a regression on master so should be backported to 1.3.x
@jbrockmendel can you post some timings.
> can u add a test here

not actually changing any behavior

> can you post some timings

based on the asv mentioned in #41873

```
index = pd.UInt64Index(range(10**6))
ser = pd.Series(np.random.randn(10**6), index=index)

%timeit ser.loc[[800000]]
278 µs ± 12.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)   # <- PR
124 ms ± 1.13 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)   # <- master
```
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
ENH: nullables use Kleene logic for any/all reductions
State: closed

- [x] xref #41967
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

This gives consistency for nullable dtype treatment across `.any/.all` and the groupby versions which already used Kleene logic. Diff looks big, but this is just moving `BooleanArray.any` and `BooleanArray.all` to `BaseMaskedArray`

Labels:
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
lgtm. @jorisvandenbossche if you can give a quick glance.
thanks @mzeitlin11 
Thanks @mzeitlin11 !
CI: activate azure pipelines/github actions on 1.3.x
State: closed

This is milestoned 1.4 since the config is on master. (and we don't want the autobackport) 

Labels:
name: CI
description: Continuous Integration

Comments:
i'll merge this now. so that if we need to do any other backports to 1.3.x we can confirm the ci status
@meeseeksdev backport 1.3.x
> This is milestoned 1.4 since the config is on master. (and we don't want the autobackport)

looks not to be the case.
Bug in melt raising Error with dup columns as value vars
State: closed

- [x] closes #41951
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

@simonjayhawkins I think this changed in May last year, so probably not woth backporting?

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
thanks @phofl 

this seemed simple enough
> @simonjayhawkins I think this changed in May last year, so probably not woth backporting?

can confirm, first bad commit: [62c7dd3e771d5dc2921212cb363239b8f1447058] Fix MultiIndex melt when col_level is used (#34329)

This looks like it would be a straightfoward backport. I see no harm in backporting as it's a regression, but would need release notes moving first so that autobackports work.
Meaning opening a new PR for the Whatsnew before backporting this?
> Meaning opening a new PR for the Whatsnew before backporting this?

yes. @jreback thoughts on including this regression fix in 1.2.5? (it was a regression in 1.1)
prob not worth backport to 1.2.x
CI: enable auto-cancel for GitHub Actions
State: closed


Labels:
name: CI
description: Continuous Integration

Comments:
It's working correctly. I got email notifications when previous jobs were canceled.
What does auto-cancel do?
> What does auto-cancel do?

A new commit will cancel all unfinished builds. So the CI only runs for the latest commit.
thanks @fangchenli very nice!
Maybe we should turn this off for the master branch. Or find a way to mark canceled jobs as 'success'.
ahh yes if u can turn off for master
REGR: Series[dt64/td64].astype(string)
State: closed

- [x] closes #41409
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

This surfaces another bug in `StringArray.__getitem__` that I haven't looked into yet.

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.