BrandonFors/pandas-github-issues huggingface dataset:
[ArrowStringArray] startswith/endswith using native pyarrow method
State: closed

```
[ 50.00%] ··· strings.Methods.time_startswith                                                                                                             ok
[ 50.00%] ··· ============== ==========
                  dtype                
              -------------- ----------
                   str        19.2±0ms 
                  string      16.0±0ms 
               arrow_string   2.42±0ms 
              ============== ==========

[ 50.00%] ··· strings.Methods.time_endswith                                                                                                               ok
[ 50.00%] ··· ============== ==========
                  dtype                
              -------------- ----------
                   str        19.3±0ms 
                  string      15.0±0ms 
               arrow_string   6.68±0ms 
              ============== ==========

```

Labels:
name: Performance
description: Memory or execution speed performance
name: Strings
description: String extension data type and string data

Comments:
nice!
DEPS/CLN: remove distutils usage
State: closed

- [x] close #41199


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Dependencies
description: Required and optional dependencies
name: Python 3.10

Comments:
hmm didn't realize `packaging` is a thing.

cc @pandas-dev/pandas-core any comments.
As mentioned on the issue, we can also vendor the part of the code of `packaging` that we need (https://github.com/pandas-dev/pandas/issues/41199#issuecomment-830063572)
Why not wait?  Python 3.12 is a long way off, and there maybe an official equivalent of LooseVersion in the standard library by then.
> Why not wait? Python 3.12 is a long way off, and there maybe an official equivalent of LooseVersion in the standard library by then.

deprecation warnings!

i don't think vendoring would be very hard (and this is in general a nice cleaning anyhow)
> deprecation warnings!

Could just import it once in a compat module and catch the deprecation warning. 
I don't think waiting (and add warning catching as workaround for now) would help much. We will have to otherwise do it in 2 years anyway, and they don't plan to add an alternative to the stdlib. 

For reference, this is the PR that vendored it in pyarrow: https://github.com/apache/arrow/pull/9849/files#diff-01d967ca0c7171567d28b14f96289b349089dcab833c61a83d88f3fb72303591
yeah +1 on vendoring. 
If they really aren't going to provide a simple mechanism to handle versioning OOB, then vendoring is the next best.  Still total PITA.
> If they really aren't going to provide a simple mechanism to handle versioning OOB, then vendoring is the next best. Still total PITA.

for sure! but this is *slightly* better than existing
looks good. can you rebase after https://github.com/pandas-dev/pandas/pull/41327 is merged and ping on green.
@fangchenli #41327 now merged
thanks @fangchenli very nice!
DOC: Add Lux to Pandas Ecosystem page
State: closed

#### Location of the documentation

List of systems under [pandas ecosystem](https://pandas.pydata.org/docs/ecosystem.html#pandas-ecosystem)

#### Documentation problem

Add [Lux](https://github.com/lux-org/lux) to the ["pandas ecosystem" documentation](https://pandas.pydata.org/docs/ecosystem.html#pandas-ecosystem)



Labels:
name: Docs

Comments:
can you run the precommit checks to fixup: https://github.com/pandas-dev/pandas/pull/41205/checks?check_run_id=2471833240

ping on green.
Thanks @dorisjlee CI failure is unrelated
BUG: Fix export .to_numpy() with nullable type
State: closed

- [x] closes #40630
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Under such a situation where a change causes cascading failed cases across multiple tests. Should I be modifying the other test cases?
If not, what strategy should I adopt?
> Under such a situation where a change causes cascading failed cases across multiple tests. Should I be modifying the other test cases?

There are a bunch of test failures, so I think the approach should be looked at more carefully first. Running tests locally should also always be the first step to catch test failures before using CI
TST: refactored tests to use parametrize instead of "for" loop
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks @ShaharNaveh
ENH: use native filesystem (if available) for read_parquet with pyarrow engine
State: closed

Some timings in https://issues.apache.org/jira/browse/ARROW-12428 suggest that it can be beneficial for reading parquet from S3 to use the pyarrow filesystem implementation, instead of pandas always using fsspec if installed (especially when selecting columns).

Labels:
name: Enhancement
name: IO Parquet
description: parquet, feather
name: Stale
name: Arrow
description: pyarrow functionality

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@jorisvandenbossche - was there much else that needed doing on this? Happy to help as this is functionality I use a lot :) 
Might want to document this behavior. If so, here's my docs from the geopandas PR:

```
        When no storage options are provided and a filesystem is implemented by
        both ``pyarrow.fs`` and ``fsspec`` (e.g. "s3://") then the ``pyarrow.fs``
        filesystem is preferred. Provide the instantiated fsspec filesystem using
        the ``filesystem`` keyword if you wish to use its implementation.
```
this is quite old, happen to reopen if actively worked on.
closing as stale
BUG: dataframe setitem error on size incompatible assignment
State: closed

- [ ] closes #40827
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

I've labeled the error something silly but that can be fixed later.

This pull request replaces this older pr #41008 

I've put all my tests inside one file for easier review.

### Issues
I'm unsure how to stop this test from failing `test_pivot_table_doctest_case` Could you help with this?

This test (`test_invalid_colormap`) also fails but when I rerun it individually, it passes ...

I also had an issue with flake8 and black not working together ... in my most recent commit so I just committed it with `--no-verify`.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Stale

Comments:
I may have worked something out so don't review this yet.
I've gotten `test_pivot_table_doctest_case` to pass.
@jreback Thanks for the review. I will move the tests and rename them later. How do I make sure that I'm not duplicating tests ...?
> @jreback Thanks for the review. I will move the tests and rename them later. How do I make sure that I'm not duplicating tests ...?

IIUC, you are essentially narrowing a case, so either try to find the tests that exercise that part of the code (might) be hard, or simply add in the logical place; by-definition you won't be duplicating.
@jreback Sorry I've been a bit busy. I'll return to this on the weekend when I have some time.
@jreback Hey so I've been trying to work on this but I've been getting an error. Do you have any ideas?
Steps to reproduce:
1. `git fetch upstream`
2. `git merge upstream/master`
3. `conda env update -n pandas-dev --file environment.yml  --prune`
4. `python setup.py build_ext -j 4`

I get this strange error:
```sh

  File "setup.py", line 249
    f"{extension}-source file '{sourcefile}' not found.\n"
                                                         ^
SyntaxError: invalid syntax
```

I'm not really sure why this is happening when it wasn't before. I'm using the new M1 chip on the mac if that matters.

Also, if I get this issue in the future, where should I be putting it? I don't think should be raising a Github issue for development build errors.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
> This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.

I will have a look and see if I can get this repository running in the next few days.
Closing as I do not want to work on this anymore.
REF: tests.indexes.numeric.py
State: closed

This PR ensures that all index instantiation calls in `tests.indexes.numeric.py` use the `_holder` class variable instead of the actual index class. This will make it easier to extend the existing tests to tests for the new `NumIndex` in #41153.

Also renames `_holder` to `_index_cls` for all of `tests.indexes`. I think this is a more descriptive name.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Index
description: Related to the Index class or subclasses

Comments:
Ok, I have changed those to fixtures.
WEB add phofl to maintainers list
State: closed

Am I OK to add a checklist (which would probably only have about 3 items: add to pandas-dev team on GitHub, add to mailing group, list on website) to the Wiki for next someone's added?

Labels:
name: Web
description: pandas website

Comments:
Thanks @MarcoGorelli 
thanks @MarcoGorelli and @phofl 
We should probably add updating the governance people list https://github.com/pandas-dev/pandas-governance/blob/master/people.md every now and then if you add a check list
Added! https://github.com/pandas-dev/pandas/wiki/Adding-new-maintainers
Deprecate inplace in Categorical.rename_categories
State: closed

- [x] xref #37643
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Categorical
description: Categorical Data Type
name: Deprecate
description: Functionality to remove in pandas

Comments:
need to do `tm.assert_produces_warning` in some of the existing tests i think
> need to do `tm.assert_produces_warning` in some of the existing tests i think

As far as I see, the failed CI task treats warnings as errors. I will check which tests should be changed
yeah pls assert the warning o nthe failing tests and ping on green
@jbrockmendel @jreback 
I fixed the uncaught warning issues, all checks are green.
thanks @OlehKSS
TYP: use DTypeLike alias from numpy.typing
State: closed


Labels:
name: Typing
description: type annotations, mypy/pyright type checking
name: Mothballed
description: Temporarily-closed PR the author plans to return to

Comments:
@simonjayhawkins I hope you an approve and merge in #40421 before this one gets merged, because many of the changes there are with respect to using `NpDtype` in the right places with respect to the `to_numpy()` calls in `ExtensionArray` classes and subclasses.
> @simonjayhawkins I hope you an approve and merge in #40421 before this one gets merged, because many of the changes there are with respect to using `NpDtype` in the right places with respect to the `to_numpy()` calls in `ExtensionArray` classes and subclasses.

I recall suggesting doing this as a precursor. some changes in #40421 would become unnecessary as the mypy errors are false postives. 
closing to clear queue. will hopefully circle back to typing next week.
@Dr-Irv @jbrockmendel I probably won't have time to revisit this for a while (probably after 1.3.1 to allow time to follow regressions from 1.3) so feel free to take this on.
I could be wrong, but I think this is necessary precursor to the EA typing work.
> I could be wrong, but I think this is necessary precursor to the EA typing work.

I think you could do it in either order.  If we can merge in the things I have been working on since late January, incorporating the changes in this PR will be straightforward.  On the other hand, if we merge in this PR, then I probably have to make changes in my 4 open PRs, which would be a PITA.
improved dataframe example to avoid confusion
State: closed

- [x] closes #41180
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Thanks @regmibijay. The CI failure looks unrelated
TST: run frame/series arithmetic tests with+without numexpr
State: closed

Follow-up on https://github.com/pandas-dev/pandas/pull/40463, doing the same for the `test_arithmetic.py` for frame and series.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
Failure is the flaky ResourceWarning
WIP: avoid zip and unzip to improve the speed of get_indexer
State: closed

- [x] closes #23735 #34531 
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] improve the speed for get_index by avoiding some redundant zipping and unzipping

Labels:

Comments:
@qiuwei is this something you want to continue working on?
> @qiuwei is this something you want to continue working on?

@datapythonista I realize there are a lot of implementations depending on the current `get_indexer` interface.
Changing this requires too much work for me at the moment.  

Close for now.
DEPR: Deprecate convert_float
State: closed

- [x] closes #41127
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

I added tm.assert_produces_warning for every existing test that uses the `convert_float` argument, and a git grep only found one reference to `convert_float` in doc/. Is anything else required for a deprecation? IIUC, the actual feature would get removed in a pandas 2.0, correct?

Labels:
name: IO Excel
description: read_excel, to_excel
name: Deprecate
description: Functionality to remove in pandas

Comments:
Hello @ahawryluk! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-05-25 16:38:24 UTC
I'm failing some checks due the warnings stacklevel. I think stacklevel=5 would be correct if the user calls pd.read_excel, but then it would be wrong for pd.ExcelFile.parse. What would you recommend?
cc @rhshadrach @WillAyd 
@ahawryluk can you merge master to resolve conflicts
@jreback @rhshadrach I've updated the warnings in pandas/io/excel/_base.py to use the recently improved find_stack_level from pandas.utils._exceptions. One of the test suites failed, but I don't think it's related to excel routines. One of the error messages was "TypeError: Cannot cast array data from dtype('O') to dtype('<U') according to the rule 'unsafe'", and all the others were "FutureWarning: Promotion of numbers and bools to strings is deprecated."
thanks @ahawryluk very nice
Thanks everyone for your help
PERF: implement NDArrayBackedBlock
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
can you rebase (haven't looked yet)
cool
fixed formatting on to_csv() for categorical datetypes
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv
name: Categorical
description: Categorical Data Type
name: Stale

Comments:
Thanks @kyleolaughlin for the PR. what is the issue number this PR is addressing?
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Thanks for contributing @kyleolaughlin, closing as stale for now, but please ping to reopen if you'd like to continue working on this.
fixed formatting on to_csv() for categorical datetypes
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:

Comments:
Hello @kyleolaughlin! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-04-26 23:42:42 UTC
TST: Regression test for #33765
State: closed

- [x] closes #33765
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them


Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: MultiIndex

Comments:
lgtm if you can make a not-draft can merge.
There is a test failure that seems unrelated.
The failing test is in:
```pandas/tests/reductions/test_reductions.py::TestSeriesReductions::test_empty_multi```

---

Is there a possibility that the changes here, causing that test to fail?
> There is a test failure that seems unrelated.
> The failing test is in:
> `pandas/tests/reductions/test_reductions.py::TestSeriesReductions::test_empty_multi`
> 
> Is there a possibility that the changes here, causing that test to fail?

no this occasionaly fails
thanks @ShaharNaveh
[ArrayManager] TST: remove more ArrayManager skips for JSON IO
State: closed

Some more JSON skips that I missed in https://github.com/pandas-dev/pandas/pull/41081 can now be removed

xref https://github.com/pandas-dev/pandas/issues/39146

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Internals
description: Related to non-user accessible pandas implementation
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
The failing test is the flaky ResourceWarning
thanks @jorisvandenbossche
PERF: numexpr doesn't support floordiv, so don't try
State: closed

Re-take of https://github.com/pandas-dev/pandas/pull/40727, see that PR for explanation.

Tests will still be failing (so don't merge yet ;)), but depend on #41161 to be fixed

Labels:
name: Performance
description: Memory or execution speed performance
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
so don't put the milestone as an indicator of not merge yet
I made it draft to be clear 
This is now passing, and the actual change was already reviewed / approved in #40727
REF: move check for disallowed bool arithmetic ops out of numexpr-related expressions.py
State: closed

The current `_bool_arith_check` helper function in `expressions.py` does 2 things: 1) check for operations we do not allow in pandas for boolean data (division, power) and 2) check for ops that are not supported by numexpr but have a pure python fallback.

That first aspect is not related to numexpr, though: we always check this (and raise an error if appropriate), whether numexpr is used or not. 

So this PR does:

- Move the check for disallowed bool ops out of the numexpr code to `array_ops.py`. This is a cleaner separation of concerns, and ensures that the check is done consistently (eg if not calling floordiv through numexpr (https://github.com/pandas-dev/pandas/pull/40727), or if in the future not calling expressions.py if we know we can't use numexpr, such as in https://github.com/pandas-dev/pandas/pull/41122)
- In `expressions.py`, move the remaining check for unsupported ops (by numexpr) inside an except block: this way, we don't check up-front (for all ops, for all types), and only do the types check if there was actually a NotImplementedError. That avoids paying the overhead costs for all other ops. 

Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Clean

Comments:
looks ok, can you rebase as merged the numexpr test updates.
Min max sparse
State: closed

- [x] closes #40921
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Sparse
description: Sparse Data Type
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
Hello @taytzehao! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-04-27 08:13:18 UTC
pre-commit is failing, pls run the hooks: https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#pre-commit
cc @jbrockmendel 
thanks @taytzehao 
@jreback this wasnt ready, in particular https://github.com/pandas-dev/pandas/pull/41159#discussion_r621602882 looks sketchy
ok feel free to revert
@taytzehao would you prefer to revert or to address this in a follow-up?
API: `Styler.hide_columns` replaced by `.hide_values` which also operates row-wise
State: closed

For showing and hiding columns/rows `Styler` currently has two methods:

 - `.hide_columns(subset)`: which hides a selection of columns and the associated data values.
 - `.hide_index()`: which hides the index keys completely and displays all the data values.

This PR adds a `.hide_values(subset, axis, show)` method, which is a superset of `.hide_columns`. It allows the same functionality to operate row-wise, and adds the kwarg `show` which allows an inverse method, exclusively showing instead of hiding.

### Follow-on:

#### Option 1

Keep `.hide_values(subset, axis, show)` as is and **deprecate** `.hide_columns`.

#### Option 2

Make `._hide_values(subset, axis, show)` a private module and include/add the public methods:

    - `hide_columns`      = `_hide_values(subset=subset, axis="columns", show=False)`.
    - `hide_rows`         = `_hide_values(subset=subset, axis="index", show=False)`.
    - `show_columns`      = `_hide_values(subset=subset, axis="columns", show=True)`.
    - `show_rows`         = `_hide_values(subset=subset, axis="index", show=True)`.



Labels:
name: Styler
description: conditional formatting using DataFrame.style
name: API - Consistency
description: Internal Consistency of API/Behavior

Comments:
closing this is favour of #41266 for now.