BrandonFors/pandas-github-issues huggingface dataset:
TYP: testing.pyi
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
gentle ping @WillAyd, annotations PRs are piling up and becoming blockers (albeit mostly for further annotations, so no real hurry)
thanks @jbrockmendel 
booyah!   more of these .pyi files are waiting in the wings once the current crop of TYP PRs go through
TST Add test for loc on sparse dataframes
State: closed

Adds a test for loc on sparse dataframes.

- [x] closes #34687
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Sparse
description: Sparse Data Type

Comments:
The test `pandas/tests/io/xml/test_xml.py::test_parser_consistency_url` seems to be failing, but I am not seeing how I am touching this code by just adding another test. Does anyone know how I would be able to fix this?
> The test pandas/tests/io/xml/test_xml.py::test_parser_consistency_url seems to be failing, but I am not seeing how I am touching this code by just adding another test. Does anyone know how I would be able to fix this?

This is unrelated and OK to ignore here.  Just restarted that build.
@EricLeer the pre-commit is failing if you can have a look
thanks @EricLeer
REGR: where not copying on no-op
State: closed

- [x] closes #39595
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Regression
description: Functionality that used to work in a prior pandas version
name: DataFrame
description: DataFrame data structure

Comments:
thanks @mzeitlin11 
@meeseeksdev backport 1.2.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.2.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 f75a620d4401dc2f4daa6ae9f10170345a63ae92
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #40592: REGR: where not copying on no-op'
```

4. Push to a named branch :

```
git push YOURFORK 1.2.x:auto-backport-of-pr-40592-on-1.2.x
```

5. Create a PR against branch 1.2.x, I would have named this PR:

> "Backport PR #40592 on branch 1.2.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
PERF: Define Block.__init__ in cython
State: closed

Shaves about 10% off the benchmark discussed in https://github.com/pandas-dev/pandas/pull/40171#issuecomment-790219422

Labels:
name: Performance
description: Memory or execution speed performance
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
lgtm. maybe some of these checks are stuck.
green
can u rebase
rebased + green
REF: deduplicate group min/max
State: closed

ASVs look unaffected:
<details>

```
before           after         ratio
     [05a0e98a]       [16d06e22]
     <master>         <ref/deduplicate_grp_min_max>
         33.0±3ms         33.3±1ms     1.01  gil.ParallelGroupbyMethods.time_loop(2, 'max')
       32.2±0.4ms         34.5±3ms     1.07  gil.ParallelGroupbyMethods.time_loop(2, 'min')
         64.1±1ms       65.6±0.5ms     1.02  gil.ParallelGroupbyMethods.time_loop(4, 'max')
         65.3±3ms         65.5±1ms     1.00  gil.ParallelGroupbyMethods.time_loop(4, 'min')
         20.2±2ms       20.2±0.8ms     1.00  gil.ParallelGroupbyMethods.time_parallel(2, 'max')
         21.3±1ms       20.1±0.8ms     0.95  gil.ParallelGroupbyMethods.time_parallel(2, 'min')
       25.1±0.7ms       25.3±0.5ms     1.01  gil.ParallelGroupbyMethods.time_parallel(4, 'max')
       25.0±0.4ms         26.5±2ms     1.06  gil.ParallelGroupbyMethods.time_parallel(4, 'min')
         125±50ms          104±1ms    ~0.83  groupby.GroupByCythonAgg.time_frame_agg('float64', 'max')
          105±2ms          107±2ms     1.02  groupby.GroupByCythonAgg.time_frame_agg('float64', 'min')
```
</details>


Labels:
name: Refactor
description: Internal refactoring of code
name: Groupby
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
LGTM
thanks @mzeitlin11
COMPAT: add back dummy CategoricalBlock class
State: closed

See https://github.com/pandas-dev/pandas/issues/40226#issuecomment-804903010

@jbrockmendel this is unfortunate, but short term I think we need to keep the CategoricalBlock class. I am not directly sure what would be the best way to actually deprecate this. 

Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Categorical
description: Categorical Data Type

Comments:
>  I am not directly sure what would be the best way to actually deprecate this.

Could keep the deprecation in `internals.__init__.__getattr__`
Ah, yes, that's a good idea. Since we don't use that import path ourselves, we can still deprecate accessing the class that way. 
what is the user of this actual class? pyarrow?
yes, pyarrow uses `isinstance(block, CategoricalBlock)` right now
why didn't this break the CI? (the original change)
Because we don't have full coverage of pyarrow functionality in our tests (and we don't run the full pyarrow test suite), there were only a few failures in the pyarrow CI.
> Because we don't have full coverage of pyarrow functionality in our tests (and we don't run the full pyarrow test suite), there were only a few failures in the pyarrow CI.

ahh ok. is it possible to add something that exercises internals? (not blocking here)
COMPAT: make Categorical._codes settable again
State: closed

This is a follow-up on https://github.com/pandas-dev/pandas/pull/40033, which changes the `_code` attribute of a Categorical into a property without a setter.

This broke dask / fastparquet, so this PR is making it settable again. We could deprecate the setter, though, at the same time.

Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Categorical
description: Categorical Data Type

Comments:
> We could deprecate the setter, though, at the same time

+1
> This broke dask / fastparquet, so this PR is making it settable again. We could deprecate the setter, though, at the same time.

new PR? or this one?
Let's merge this to fix dask's CI. We can deprecate it in a follow-up / together with providing an alternative.
I opened https://github.com/pandas-dev/pandas/issues/40606 marked for 1.3, so we don't forget this
TST/CLN: remove redundant to_json test
State: closed

Should be covered by the test which follows it since `pd.NA` is in `nulls_fixture`

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO JSON
description: read_json, to_json, json_normalize
name: Clean

Comments:
Awesome thanks! FYI this module is kind of a mess compared to other IO areas. If you are interested in contributing more I think splitting it up into a few separate modules for reading / writing / roundtrip *might* make sense; open to your thoughts
> Awesome thanks! FYI this module is kind of a mess compared to other IO areas. If you are interested in contributing more I think splitting it up into a few separate modules for reading / writing / roundtrip _might_ make sense; open to your thoughts

Agreed, will look into it
TYP: tslibs.parsing.pyi
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
thanks, are these actively checked now, the .pyi?
> thanks, are these actively checked now, the .pyi?

mypy uses them, yes
BUG: groupby rank computing incorrect percentiles
State: closed

- [x] closes #40518
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Built on #40546, is a relatively minor change after that.

Labels:
name: Bug
name: Groupby
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
lgtm. can you merge master and ping on greenish
> lgtm. can you merge master and ping on greenish

@jreback greenish
thanks @mzeitlin11
STY: enable flake8-bugbear
State: closed

There are some cases where one of the values of a `zip()` was not used. I removed the `zip()`. If the two sequences for `zip()` have different lengths, this would change the behavior.

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
nice - does this solve all bugbear errors? If so, do you want to add it as an additional dependency to the flake8 check? https://github.com/pandas-dev/pandas/blob/ced764da36bbc493d3ada854983638a8552d36f6/.pre-commit-config.yaml#L39
> nice - does this solve all bugbear errors? 

There are still plenty of other warnings. It might be possible to enable bugbear but we would need to disable some rules/add ignore statements (especially when covering the test code).

bugbear could be enabled when the following warnings are ignored: B004, B005, B006, B008, B009, B010, B011, B015.

I feel there is something wrong with this test:
https://github.com/pandas-dev/pandas/pull/40570/files#diff-5974844fa02c9b81f3477198023bd02503bf82da1e3d15df6dfd62cdca48666bR132
but replacing `r` with `l` results in some errors.

I honestly do not understand how `v` is used in this test - is there a weird side effect (or I am blind)?
https://github.com/pandas-dev/pandas/pull/40570/files#diff-64777e769142fd7a85a91ae1809c0485036fb28ebb32cc6cccba9ae83646ac15R959

@MarcoGorelli I enabled almost all of bugbear's checks. The current way to handle B008 might decrease performance a bit (could either add ignore statements or have module-level variables).
I like getting rid of unnecessary zip/enumerates, bug things like requiring unused loop variable to be "_" seem annoying
> I like getting rid of unnecessary zip/enumerates, bug things like requiring unused loop variable to be "_" seem annoying

I will separate the removal of unnecessary statements into a new PR later today.
Would it make sense to have a new PR that enables only a smaller subset of rules that are less controversial?

Skip:
B005 (strip with multiple characters)
B006 (mutable default values)
B008 (function calls in default values)
B007 (unused loop variables)
B301 (iteritems - false positives)
and the already skipped rules in this PR

Personally, I would like to have B007 since it found unused zips/enumerates and weird tests. B005 would also be great: there is a comment in the function that uses `rstrip(".__call__")` that this function has a lower priority because it causes issues otherwise (speculating: might be because of this `rstrip` call).

sure happy to have some of these enabled in a non controversial PR

then can follow up enabling others
DEPR: use DeprecationWarning instead of FutureWarning for CategoricalBlock
State: closed

Follow-up on https://github.com/pandas-dev/pandas/pull/40527

There is no need for end-users to see this warning if an underlying library they use generates this, so using a DeprecationWarning instead of FutureWarning (as we did before with potentially noisy or internal warnings). At a later point we can change to FutureWarning.

Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Deprecate
description: Functionality to remove in pandas

Comments:
thanks @jorisvandenbossche
BUG: Fix query doesn't support equals or not equals with Python parse…
State: closed

…r (#40436)

- [X] closes #40436 
- [X] tests added / passed
- [X] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [X] whatsnew entry


Labels:
name: Bug
name: expressions
description: pd.eval, query

Comments:
Could some of the pandas maintainers let me know if the failed checks are normal ? They are unrelated to any changes I've made as far as I know.
> Could some of the pandas maintainers let me know if the failed checks are normal ? They are unrelated to any changes I've made as far as I know.

If you look at the failures in a run like https://github.com/pandas-dev/pandas/pull/40563/checks?check_run_id=2163292906, looks like they might be related?
They look related to me.  Do these tests pass for you locally?
> > Could some of the pandas maintainers let me know if the failed checks are normal ? They are unrelated to any changes I've made as far as I know.
> 
> If you look at the failures in a run like https://github.com/pandas-dev/pandas/pull/40563/checks?check_run_id=2163292906, looks like they might be related?

I understand that my changes had a broader impact than I had anticipated. Some of these tests hard coded the == and != operators as NotImplemented, which I have undone and this is causing them to fail. 

Will have to really dig around to understand if these changes I made are ok but simply failing tests which should be amended or if I may be introducing some bug(s). Will take much more time than anticipated.
Ok, I dug further into this and I both understand more and less about the issue. I now understand that both the `pandas` and `python` visitors inherit from the same `BaseExprVisitor` albeit with different parameters for `disallow` and different `preparser` function.

However, I do not understand why `In` and `NotIn` were specifically disallowed for the `python` engine even though its treatment would be identical under both `PythonExprVisitor` and `PandasExprVisitor`. This appears quite intentional as removing `In` and `NotIn` from the `PythonExprVisitor` disallow list makes a few tests fail as they specifically expect queries using these bin ops to raise `NotImplemented`.

Could someone please educate me as to why this is desirable ?
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
would take this if you can merge master and will look again
Thanks for the PR, but it appears to have gone stale. If interested in continuing, please merge master and a maintainer can have a second look.
ENH: applymap get kwargs #39987
State: closed

- [X] closes #39987 
- [X] tests added / passed
- [X] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [X] whatsnew entry


Labels:
name: Apply
description: Apply, Aggregate, Transform, Map
name: API - Consistency
description: Internal Consistency of API/Behavior

Comments:
thanks @alexprincel
CLN: remove unused axis keyword from Block.where
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Clean

Comments:
thanks. any perf changes? (e.g. you added `copy=False`)
> thanks. any perf changes? (e.g. you added copy=False)

we save a copy and make a mask more efficiently, but nothing earth-shaking
REF/PERF: deduplicate kth_smallest
State: closed

Ensuring a contiguous input seems to give about a 5-10% performance improvement on the existing benchmark `gil.ParallelKth`

Labels:
name: Performance
description: Memory or execution speed performance
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
@mzeitlin11 any top-level user visible perf worth mentioning in the what's new? (e.g. nlargest / nsmallest)
> @mzeitlin11 any top-level user visible perf worth mentioning in the what's new? (e.g. nlargest / nsmallest)

Based on the benchmark should speed up nlargest/nsmallest about 5-10%. Let me know if that's worth adding a whatsnew
thanks @mzeitlin11 

ok this was more of a refactor then.
TST: add test_ffill_with_string_column
State: closed

- [x] closes #40250 
- [x] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

#40250 is an issue which seems to have been fixed in master already through #39446. This PR adds a test for this issue.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Strings
description: String extension data type and string data

Comments:
Hey @mzeitlin11, no reason in particular - just thought I'd wait for the CI checks to pass before making it ready for review.

It does seem that one of the CI checks is failing (CI / Web and docs), but I have no reason to suspect that the failure has to do with the changes in this PR.

Also: it seems that the whatsnew entry may not be needed since this is just testing an already-fixed bug. Let me know if that's the case.

I have marked as ready to review! Thank you
> Hey @mzeitlin11, no reason in particular - just thought I'd wait for the CI checks to pass before making it ready for review.
> 
> It does seem that one of the CI checks is failing (CI / Web and docs), but I have no reason to suspect that the failure has to do with the changes in this PR.
> 
> Also: it seems that the whatsnew entry may not be needed since this is just testing an already-fixed bug. Let me know if that's the case.
> 
> I have marked as ready to review! Thank you

Yep, whatsnew is not necessary. The CI / Web and docs failure is unrelated and should be fixed by #40542
thanks @shiv-io
DOC: Fix minimum Python version in contributing docs
State: closed

- [x] closes #40485 


Labels:
name: Docs

Comments:
I think the failed check also fails in the current master. So, what should I do?
> I think the failed check also fails in the current master. So, what should I do?

nothing. just wait. it might get approved without, or you might be asked to rebase when master goes green again (gets fixed)
Thanks for the explanation. 
Hi @hasan-yaman - could you fetch and merge upstream/master please?
Thanks @hasan-yaman for updating - let's just wait for #40130 to be in before proceeding here
 #40130 is in - @hasan-yaman if you rebase and ping when ready this should be good to go 
@MarcoGorelli 

I moved the changes to contributing_environment.rst

Thanks for your help.
TST: `test_tooltip.py` convert to functional tests instead of class
State: closed

- [x] adds `pytest.fixture` instead of Class setup method.
- [x] removes `test_tooltip_reindex` since it is already being tested by `test_tooltip_render`
- [x] other tests no changes except adopting fixtures


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean
name: Styler
description: conditional formatting using DataFrame.style

Comments:
thanks @attack68
TYP: internals.pyi
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
can you rebase
rebased + greenish
cool. any ideas about the failures on master?

e.g. https://github.com/pandas-dev/pandas/runs/2220118625
> cool. any ideas about the failures on master?

looking into it now

BTW the other open TYP PRs all play together nicely
TYP: mostly core.arrays, some core.indexes
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
cool one comment above for potential followon
DOC: Fixing indentation of example of DataFrame
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Thanks @juliansmidek
CLN: Don't catch TypeError in FrameApply.agg
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Ref: #34224. Back when transform and agg shared code, this was introduced to stop transform from having an infinite recursive loop. Now they do not, it is no longer necessary and only serves to obfusticate the error message.

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Clean
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
@jbrockmendel - merged master CI is now green except code coverage, which appears incorrect to me. Throwing a `raise ValueError` just above the untested lines fails 385 tests in tests.apply.
thanks @rhshadrach
DOC: suppress warnings from CategoricalBlock deprecation
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
This fixes the docbuild, in conjunction with just-merged #40541 should get us back to green
tagging #40538 and #40540 so that when this is merged, I can merge with master and pass CI
@Dr-Irv pls update those PRs and ping when ready
Revert "PERF: increase the minimum number of elements to use numexpr for ops from 1e4 to 1e6"
State: closed

Reverts pandas-dev/pandas#40502

Troubleshooting the CI

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
Looks like this fixes all the builds except docbuild, which should be fixed by #40542
great merge when ready (and maybe open an issue about the perf change)
This fixes the memory issue I noted on gitter:

> Anyone else seeing larger memory usage when running the tests? Each pytest worker is using close to 4 GiB on me, I recall this being 2 GiB not too long ago.
Thanks @jbrockmendel 
> Anyone else seeing larger memory usage when running the tests? Each pytest worker is using close to 4 GiB on me, I recall this being 2 GiB not too long ago.


Now that you mention it... about 1.3 GB at the end of collection, 6.04 by the time it hits 1%!  2GB matches my recollection of what ive seen recently pre-yesterday