BrandonFors/pandas-github-issues huggingface dataset:
CI: use pyarrow 0.17.1 instead of 0.17.0 for py37_macos build
State: closed

Closes #41937

Labels:
name: CI
description: Continuous Integration

Comments:
thanks merge when green (note we may need for 1.2.5 as well?)

cc @simonjayhawkins 
> note we may need for 1.2.5 as well?

will know for sure when I do the manual backport of #41924

or could backport anyway?
> @simonjayhawkins I think 1.2.5 is still on pyarrow 0.15 so shouldn't backport?

we don't really want ci constantly failing. if we need a patch for pandas 1.2.x to work with pyarrow 0.15 that's a different issue.
Sorry, should've been more clear.
The MacOS Python 3.7 job on Azure is on pyarrow 0.15.1 on 1.2.x.
https://github.com/pandas-dev/pandas/blob/322d54a9257e0d57a13dae6a443855df39e18ba3/ci/deps/azure-macos-37.yaml#L24
According to the JIRA issue, it seems this would only affect 0.17.0. I guess we should just first not backport, and then backport if CI is red on the MacOS job then.
> According to the JIRA issue, it seems this would only affect 0.17.0. I guess we should just first not backport, and then backport if CI is red on the MacOS job then.

sgtm
thanks @jorisvandenbossche
ENH: `sparse_columns` and `sparse_index` added to `Styler.to_html` 
State: closed

Checks an item off the list in #41693 




Labels:
name: Enhancement
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap
name: Styler
description: conditional formatting using DataFrame.style

Comments:
@attack68 does this need a release note?

you can keep adding to the 1.3 release notes. is Styler still experimental? I think we can merge these changes to 1.3 during the rc (if not done before)
> @attack68 does this need a release note?
> 
> you can keep adding to the 1.3 release notes. is Styler still experimental? I think we can merge these changes to 1.3 during the rc (if not done before)

No release note needed, since these args _could_ have been added in the previous PR that added the method to 1.3.0.

I think Styler is fully functional now, it has extensive docs, improved performance and features, high test coverage, and very close to being able to deprecate DataFrame.to_html and DataFrame.to_latex. Its also been around a number of years now so I would say well integrated into the package.

No reason this can't get into 1.3 as long as it fits your timeline.
> No reason this can't get into 1.3 as long as it fits your timeline.

i think OK for 1.3
don't add milestones
> don't add milestones

sure. if not 1.3 will need a release note. IIUC This sits on top an enhancement for 1.3.
and versionadded tags for the new parameters
@simonjayhawkins has this missed the boat for 1.3.0?

I can change to 1.4.0 if so.
> @simonjayhawkins has this missed the boat for 1.3.0?

not necessarily if additions to enhancements added in 1.3. The `to_html` of Styler is new for 1.3? I've not really followed the changes. Do we still have a `.render` method and does it share code?
> > @simonjayhawkins has this missed the boat for 1.3.0?
> 
> not necessarily if additions to enhancements added in 1.3. The `to_html` of Styler is new for 1.3? I've not really followed the changes. 

yes this is additions to a 1.3.0 enhancement. The only reason it wasn't added previously was to reduce the size of that other  PR and for this to be a simple follow on.

> Do we still have a `.render` method and does it share code?

Yes there is still a `.render` method currently but I will look to propose its deprecation for 1.4.0 since it becomes redundant. Its a reasonable amount of work to migrate the tests though thats why I didnt have time for 1.3.0.
This seemed to miss 1.3.0 so I have added the necessary tags and comments for 1.4.0 insted.
thanks @attack68
TYP: make numpy.typing available from pandas._typing
State: closed

xref https://github.com/pandas-dev/pandas/issues/41807#issuecomment-859372500

this does not update the `__array__` to `to_numpy` methods of EA, see https://github.com/pandas-dev/pandas/pull/41185#issuecomment-852469918 but does need a couple of ignores till that is done.

(we are removing 3 anyway, so a net reduction in ignores :smile:, and reducing false positives for users)



Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
does this require 1.21? (though passing for all so guess is no)
for runtime, `npt: Any = None` means that the numpy version is not relevant. For type checking, `import numpy.typing as npt` so needs at least 1.20, but would not be checking without 1.21 anyway otherwise will get other errors.
thanks @simonjayhawkins
[ArrayManager] Enable pytables IO by falling back to BlockManager
State: closed

This doesn't "fix" anything, but by falling back for now to the BlockManager-specific implementation of `to_hdf` writer, this avoids failures when using ArrayManager, which should make it easier to test it. 

Labels:
name: IO HDF5
description: read_hdf, HDFStore
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
@jorisvandenbossche only PRs for the rc are milestoned 1.3 for now to help track our readiness for the rc, otherwise they are milestone as they are merged.

Is this for the rc or can I remove the milestone for now?
this is fine. @jorisvandenbossche can you open an issue for the impl in AM .
It's mentioned as one of the bullet points in https://github.com/pandas-dev/pandas/issues/39146
STYLE: Use from future import annotations (pandas-dev#41901)
State: closed

- [X] closes #41901
- [ ]  no tests added
- [ ]  pre-commit hook, no python files added
- [ ] No docstrings added; docs not rebuilt. _**Where do pre-commit hooks get added to “What’s New"?**_


Implementation:
   * Name set to always-use-future-annotations (rather than always-use-from-future-annotations)
   * We filter with` types: [python]` [rather than](https://pre-commit.com/#filtering-files-with-types) `files: "\\.(py)$”` 
   * No minimum pre-commit version. 

What else is needed here for a merge? 



Labels:
name: Code Style
description: Code style, linting, code_checks
name: Typing
description: type annotations, mypy/pyright type checking
name: Stale

Comments:
edited comment to note that @aacosta13 has a script to add "from __future__ import annotations" to the 120 files in the codebase that currently don't have it (excluding tests, __init__.py, etc.) 

I suggested that should be a separate PR which can go in first. 
@ForestMars yeah - can exclude asv_bench and web too though
by way of update, I was waiting on a PR from @aacosta13 so that all targeted (ie. not excluded) files have the required import added, before we merge the pre-commit hook into the code base. I originally thought this was something that was going to be a quick fix, but haven't seen an update on #41901 in over a week. 
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
> by way of update, I was waiting on a PR from @aacosta13 so that all targeted (ie. not excluded) files have the required import added, before we merge the pre-commit hook into the code base. I originally thought this was something that was going to be a quick fix, but haven't seen an update on #41901 in over a week.

feel free to take over the other PR if you're still interested in working on this
Thanks for the PR. As mentioned, it appears a precursor PR is needed first (which we're a happy to have from you) first. Since this PR has gone stale in the meantime, going to close for now.
Use from future import annotations
State: closed

- [X] closes #xxxx
- [ ]  no tests added
- [ ]  pre-commit hook, no python files added
- [ ] No docstrings added; docs not rebuilt. _**Where do pre-commit hooks get added to “What’s New"?**_


Implementation:
   * Name set to always-use-future-annotations (rather than always-use-from-future-annotations)
   * We filter with` types: [python]` [rather than](https://pre-commit.com/#filtering-files-with-types) `files: "\\.(py)$”` 
   * No minimum pre-commit version. 

What else is needed here for a merge? 



Labels:

Comments:
I'm not happy that this pull request includes a master merge, and would prefer to redo it. 
master merges are fine
we squash on merging anyhow
TYP/BUG: fix CI
State: closed

Fix bug introduced in https://github.com/pandas-dev/pandas/pull/41903

Mypy doesn't seem to like generic types with default values?

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
thanks @twoertwein ping on green
@jreback green except for macOS
thanks @twoertwein
CLN: Remove useless Validation functions
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Not sure what the point of these were.

Labels:
name: Clean

Comments:
these were for backwards compat when we had a skipna change (long since deprecated / removed)
BUG: IntervalIndex.get_indexer with categorical both have nans
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Started with moving a fastpath from IntervalIndex to the base class.  In the process found a bug in the IntervalIndex version, which is now fixed.

Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Categorical
description: Categorical Data Type
name: Interval
description: Interval data type

Comments:
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
BUG: CategoricalIndex.get_loc(np.nan)
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Categorical
description: Categorical Data Type
name: Index
description: Related to the Index class or subclasses

Comments:
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
TST: Add tests for numeric_only default behaviour (GH41351)
State: closed

- [x] closes #41351
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Timedelta
description: Timedelta data type
name: Stale

Comments:
Hello @michaelneururer! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-06-10 20:50:13 UTC
Thanks @michaelneururer for the PR. The added test is failing ci.
@simonjayhawkins Oh, I thought that the issue was resolved on the master branch of pandas. I didn't realise that you run tests with various different numpy versions. The failure I reported in #41351 depends on the numpy version used (it works from numpy version 1.20.2). 

I am not sure I have the time to fix the issue if an older version of numpy is used. Could you advise on what I should do now?
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Thanks for the PR, but as you discovered, this PR would need a fix that depends on the numpy version in addition to tests. If you're still interested in investigating a fix in this case, let us know and we can reopen this PR, but closing for now.
REF: de-duplicate IntervalIndex._intersection
State: closed

made possible by #41863

Labels:
name: Refactor
description: Internal refactoring of code
name: Interval
description: Interval data type
name: Index
description: Related to the Index class or subclasses

Comments:
failures looks spurious 
This _might_ have caused a big slowdown, see https://pandas.pydata.org/speed/pandas/#index_object.IntervalIndexMethod.time_intersection_both_duplicate?python=3.8&Cython=0.29.21&p-param1=100000&commits=17391997-969688a9

(the [range of commits](https://github.com/pandas-dev/pandas/compare/17391997...969688a9) also includes some other commits)
> This might have caused a big slowdown [...] (the range of commits also includes some other commits)

The indexing bugfix _might_ play a part, but it does seem likely that this is the culprit.  Profiling it looks like drop_duplicates is the big time sink.
ive got a IntervalArray.unique branch in the works that cuts this this from 4x down to 2x, still a ways to go.
PERF: fix regression in reductions for boolean/integer data
State: closed

Retake of https://github.com/pandas-dev/pandas/pull/41911

Labels:
name: Performance
description: Memory or execution speed performance
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
Alternative for https://github.com/pandas-dev/pandas/pull/41920 (I still think my fix is slightly simpler though, but I won't object this one again ;-))
looks fine merge away
@meeseeksdev backport 1.2.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.2.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 b4dcf3b535d06dd3d597e37be46a4f178fe6b542
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #41924: PERF: fix regression in reductions for boolean/integer data'
```

4. Push to a named branch :

```
git push YOURFORK 1.2.x:auto-backport-of-pr-41924-on-1.2.x
```

5. Create a PR against branch 1.2.x, I would have named this PR:

> "Backport PR #41924 on branch 1.2.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.2.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 b4dcf3b535d06dd3d597e37be46a4f178fe6b542
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #41924: PERF: fix regression in reductions for boolean/integer data'
```

4. Push to a named branch :

```
git push YOURFORK 1.2.x:auto-backport-of-pr-41924-on-1.2.x
```

5. Create a PR against branch 1.2.x, I would have named this PR:

> "Backport PR #41924 on branch 1.2.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
DOC: add contributors to 1.3 release notes
State: closed

I think we should do this for the rc.

We will be publishing the docs at https://pandas.pydata.org/pandas-docs/version/1.3.0/ and linked from the rc announcement. (It's the same url for the actual release and the rc docs are replaced)

we haven't yet released 1.2.5 so the 1.2.5 tag does not yet exist. used 1.2.4 for now.

Labels:
name: Docs

Comments:
don't you need the tag to exist for this to work? (1.3)
> don't you need the tag to exist for this to work? (1.3)

it should be `.. contributors:: v1.2.5..v1.3.0|HEAD` for the actual release, but the v1.2.5 tag does not yet exist.

maybe it could be ``.. contributors:: v1.2.5|v1.2.4..v1.3.0|HEAD`` or we could merge this and update before the actual 1.3 release
PERF: fix regression in reductions for boolean/integer data (assume no missing values)
State: closed

Address performance regression in https://github.com/pandas-dev/pandas/pull/41711/files#r647515492 (https://pandas.pydata.org/speed/pandas/#series_methods.All.time_all?python=3.8&Cython=0.29.21&p-N=1000000&p-case='fast'&p-dtype='bool'&commits=9231b495-44237a58)



Labels:
name: Performance
description: Memory or execution speed performance
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
Ah, indeed you noticed the issue back then as well. Yeah, I think this is not lower level than the later PR #41711 (which was also backported), and IMO this should be safe for a backport.
@jorisvandenbossche I didn't notice #41911 (was on the wrong milestone) with a different fix. 
merged #41911
REF: de-duplicate Categorical _validate_foo_value
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

TypeError is "more correct" for these anyway, and we come within striking distance of sharing with datetimelike.

Labels:
name: Refactor
description: Internal refactoring of code
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Categorical
description: Categorical Data Type

Comments:
> TypeError is "more correct" for these anyway

release note? I think we normally add when changing the exception type (but not always for just a change in message)

Can you also explain why TypeError is "more correct".
> release note? I think we normally add when changing the exception type (but not always for just a change in message)

Just pushed with note

> Can you also explain why TypeError is "more correct".

My reasoning is that TypeError makes more sense than ValueError here because it is determined solely by the dtype.  By contrast, ValueError would make sense if we had a valid-for-the-dtype value that was an incompatible shape.  KeyError is only being replaced in a few places, and it just doesn't make sense at all in these non-indexing/lookup contexts.

> release note? I think we normally add when changing the exception type (but not always for just a change in message)

Just pushed with note

> Can you also explain why TypeError is "more correct".

My reasoning is that TypeError makes more sense than ValueError here because it is determined solely by the dtype.  By contrast, ValueError would make sense if we had a valid-for-the-dtype value that was an incompatible shape.  KeyError is only being replaced in a few places, and it just doesn't make sense at all in these non-indexing/lookup contexts.

@simonjayhawkins is it time to move whatsnew notes for existing PRs to the 1.4 file?
> @simonjayhawkins is it time to move whatsnew notes for existing PRs to the 1.4 file?

not sure @jreback ?

I started that task after we branched but was asked not to milestone things. Without being able to add milestones doing that task seemed pointless so I stopped.

If a PR has a release note then we need to assess whether we will be backporting or ask the contributor to move the release note.

If a PR doesn't have a release note then moving release note is not applicable and unless it's CI related then we probably are not backporting to 1.3.x

This PR is titled as a refactor. but has a change in behavior so probably needs a change in title as well.



> If a PR has a release note then we need to assess whether we will be backporting or ask the contributor to move the release note.

it's not just release notes, we also need to check PRs for versionadded and versionchanged tags.
let's target 1.4 with this
will have to re-loook can you rebase
thanks, makes sense
BUG: inconsistent validation for get_indexer
State: closed

By putting all the validation up-front, we'll be able to safely move some short-circuiting checks from the subclasses _get_indexer methods up to the single get_indexer method

Labels:
name: Bug
name: Index
description: Related to the Index class or subclasses

Comments:
bugfix should have a release note? 
whatsnew added + green
REF: give rank1d/2d same nan filling
State: closed

A piece broken off the `rank_2d` deduplication


Labels:
name: Refactor
description: Internal refactoring of code
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
@mzeitlin11 do the stubs need updating for the extra function and extra parameter?
> @mzeitlin11 do the stubs need updating for the extra function and extra parameter?

This added function (and the function with a new param) can only be called from `Cython` code, so compilation should handle type checking correctness. For another internal function `kth_smallest_c`, I don't see it typed in `algos.pyi`, so I don't think anything needs to be added.
> > @mzeitlin11 do the stubs need updating for the extra function and extra parameter?
> 
> This added function (and the function with a new param) will only be called from `Cython` code, so compilation should handle type checking correctness. For another internal function `kth_smallest_c`, I don't see it typed in `algos.pyi`, so I don't think anything needs to be added.



> > @mzeitlin11 do the stubs need updating for the extra function and extra parameter?
> 
> This added function (and the function with a new param) will only be called from `Cython` code, so compilation should handle type checking correctness. For another internal function `kth_smallest_c`, I don't see it typed in `algos.pyi`, so I don't think anything needs to be added.

I see. they are both cdef so not exposed to Python.
BUG: MultiIndex.difference incorrectly raising TypeError when indexes contain non-sortable entries
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: MultiIndex
name: Index
description: Related to the Index class or subclasses

Comments:
Can you check the performance impact here? (xref https://github.com/pandas-dev/pandas/pull/41929#issuecomment-860514675)
> Can you check the performance impact here?

ran asvs overnight and no obvious standouts.  re-running a subset now to be on the safe side
yep, all noise
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
REF: simplify Series.__setitem__
State: closed

Motivating goal is to get rid of the Series methods that use `self.index._engine.get_loc`, as these are a mine field (xref #41846).  The correct method would be to use `self.index.get_loc`, the trouble being that we are inconsistent about what kinds of exceptions we raise in engine.get_loc vs Index.get_loc (and Index subclasses).  This is a bite-size step towards this goal. 

Labels:
name: Refactor
description: Internal refactoring of code
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Series
description: Series data structure

Comments:
can you rebase
sure