BrandonFors/pandas-github-issues huggingface dataset:
ENH: Add arrow engine to to_csv
State: closed

- [ ] closes #53618 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement
name: IO CSV
description: read_csv, to_csv
name: Arrow
description: pyarrow functionality

Comments:
OK, this is down to 34 failing tests now (out of 87).

I think I'll xfail all the failing tests and get this merge-ready for now soonish.

Most of the things failing are just subtle differences in the way Pyarrow/Python CSV engine write things.
(One particularly annoying thing is that pyarrow quotes strings even if the quotes are technically not needed. It doesn't affect roundtripping I don't think, but it makes a bunch of the tests fail).
pre-commit ci fix
pre-commit.ci autofix
I don't think https://github.com/pandas-dev/pandas/pull/54171/files#r1267066476 was addressed otherwise looks good
Just a merge conflict otherwise looks good
```
   /home/runner/work/pandas/pandas/pandas/io/formats/csvs.py:320:17 - error: Argument of type "IO[AnyStr@_save]" cannot be assigned to parameter "csvfile" of type "SupportsWrite[str]" in function "writer"
    "IO[AnyStr@_save]" is incompatible with protocol "SupportsWrite[str]"
      "write" is an incompatible type
        Type "(__s: AnyStr@_save, /) -> int" cannot be assigned to type "(__s: _T_contra@SupportsWrite, /) -> object"
          Parameter 1: type "_T_contra@SupportsWrite" cannot be assigned to type "AnyStr@_save"
            Type "str" cannot be assigned to type "AnyStr@_save" (reportGeneralTypeIssues)
```
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Since this feature has not been updated for a long time, I am wondering if it is still being worked on.
I think there is just a typing error left that I haven't resolved, I'll add an ignore if it's still there.
Looks like this PR has gotten stale so closing for now. Feel free to reopen when you have time to circle back
DEPR: argmin/argmax with all-NA or skipa=False
State: closed

- [x] closes #33941 (Replace xxxx with the GitHub issue number)
- [x] closes #33942
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Discussed on last week's dev call.

Analogous deprecation for idxmin/idxmax will be done separately in a follow-up.

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @jbrockmendel
DEBUG: CY3 test_add_out_of_pydatetime_range
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Unreliable Test
description: Unit tests that occasionally fail
name: Dependencies
description: Required and optional dependencies

Comments:
Looks like tests are passing here. I just modified the test so that it passes with cython3 and cython0.29. I made a note in https://github.com/pandas-dev/pandas/issues/34213 of the core issue that would need to be addressed for cython3.
DOC: Fixing EX01 - Added examples
State: closed

- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Towards https://github.com/pandas-dev/pandas/issues/37875

Note: `pandas.errors.AccessorRegistrationWarning` definition was supposed to be removed.
The warning was [removed](https://github.com/pandas-dev/pandas/pull/18827/commits/a308a2e29286c03c71e3566a881629d996b4821c) but definition left.
(Explanation courtesy of @MarcoGorelli)

Labels:
name: Docs

Comments:
Thanks @DeaMariaLeon
BUG: idxmin/max dtype
State: closed

- [x] closes #43587 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
Thanks @jbrockmendel
DOC: Fixing EX01 - Added examples
State: closed

- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Towards https://github.com/pandas-dev/pandas/issues/37875

We need to use /preview as the html for `groupby`can't be generated locally.


Labels:
name: Docs

Comments:
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/54162/
Thanks @DeaMariaLeon
BUG: read_xml stripping whitespace in string data
State: closed

- [X] closes #53811
- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [X] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO XML
description: read_xml, to_xml

Comments:
Thanks @ParfaitG
Bug 54134
State: closed

- [X] closes #54134
- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [X] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: metadata
description: _metadata, .attrs

Comments:
I have no idea why that that completely unrelated test that's marked as expected to fail is failing the checks.
Could you merge in main once more?
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
CLN/ASV: reduce size of join_merge.JoinIndex
State: closed

The existing `join_merge.JoinIndex.time_left_outer_join_index` ASV joins two frames with indexes containing lots of duplicates resulting in a large result (~25m rows). This is resulting in long (and inconsistent?) run times (700-3000 ms):

<img width="304" alt="Screen Shot 2023-07-16 at 12 36 34 PM" src="https://github.com/pandas-dev/pandas/assets/8519523/fd0fbd49-581b-4680-978c-3fb71a7b1560">

This PR reduces the size of the frames being joined such that the result is ~250,000 rows and the runtime is < 20ms. 


Labels:
name: Clean
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
Nice! Happy to have more of these benchmark tuning @lukemanley
Update grouper.py
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: Groupby

Comments:
Thanks @guoci
DEPR: Not passing tuple to get_group when grouping on length-1 list-likes
State: closed

- [x] closes #25971 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Similar in spirit to #53517, #51817, #50064

Labels:
name: Groupby
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @rhshadrach
DOC: Example missing in pandas.DataFrame.to_html #44945
State: closed

- [x] closes DOC: Example missing in pandas.DataFrame.to_html #44945
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Uses the suggestion in the issue report as an example.

@noatamir  @phofl  SciPy 2023 sprint


Labels:
name: Sprints
description: Sprint Pull Requests

Comments:
thx @vvaidy
ASV: update Fillna benchmarks for method parameter deprecation
State: closed

Updates various `Fillna` benchmarks to not use `method` as `method` has been deprecated in favor of `DataFrame.ffill` and `DataFrame.bfill`. 

Also, removed `reindex.Fillna` ASV as it was just a subset of `series_methods.Fillna`. 


Labels:
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
Thanks @lukemanley
DOC: Provide examples of using read_parquet #49739
State: closed

- [x] closes #xDOC: Provide examples of using read_parquet #49739
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

@noatamir  @phofl SciPy 2023 sprint


Labels:
name: Docs
name: IO Parquet
description: parquet, feather

Comments:
Can you remove `pandas.read_parquet \` from `code_checks.sh`?
Hi @mroeschke,

No problem. I did as you requested and reran the tests locally and they succeeded.

I must confess I don't quite understand the implications of that change, but I will look it over later tonight. In the meanwhile, it's done, tested locally and pushed.

Do LMK if you need anything else. Thx.
Thanks @vvaidy
TST: Added test for query method with string dtype and null values 
State: closed

- [x] closes #31516
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

scipy2023 sprint @noatamir @phofl 

Labels:
name: Sprints
description: Sprint Pull Requests

Comments:
Thanks @tntmatthews
Add mixed int string entry.
State: closed

- [X] closes #54072 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This is in reference to #54072. Currently running tests have been failing due to memory requirements on local machine. 
As suggested by @noatamir @phofl during SciPy 2023 opening as a draft to have it run tests. 


Labels:
name: Stale
name: Sprints
description: Sprint Pull Requests

Comments:
Updated relevant tests in "test_value_counts" by having it skip if the index had more than one type. 
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
DOC: fix formatting of the validate keyword in DataFrame join docstring
State: closed

- [x] closes DOC: fix formatting of the validate keyword in DataFrame join docstring
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

@noatamir  @phofl SciPy 2023 sprint

Labels:
name: Sprints
description: Sprint Pull Requests

Comments:
Thanks @vvaidy
TEST: Add test to check safe cast between UInt32 and np.uint32
State: closed

PR from the SciPy 2023 Sprint session. @noatamir @phofl 

I implemented a test to check that the casting from `float` type to `UInt32` pandas type raises an error.

This should be raising an error, as it is expected that Pandas types do not allow for casting to types that would remove information.

Labels:
name: Stale
name: Sprints
description: Sprint Pull Requests

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
move indexslice below multiindex
State: closed

- [x] closes #50496
- [x] closes https://github.com/pandas-dev/pandas/issues/28871
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

SciPy 2023 sprint and @phofl, @noatamir 

Labels:
name: Sprints
description: Sprint Pull Requests

Comments:
thx @coco90417
Removed indentation in documentation
State: closed

Removed indentation in documentation
https://github.com/noatamir/scipy-2023-sprint/issues/8

@phofl, @noatamir


Labels:
name: Sprints
description: Sprint Pull Requests

Comments:
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/54143/
By indenting the last bullet, sphinx is treating it as a separate block from the first line whereas I think we just want a natural continuation. Also, the first line of the last bullet is getting bolded.

![image](https://github.com/pandas-dev/pandas/assets/45562402/b666f0ff-f67e-423e-aca3-5266bc725ede)

[This SO answer](https://stackoverflow.com/a/54677891) seems to indicate your first commit here had the correct level of indentation; I'm wondering if the newline you've added at the top (before the list) was sufficient (I'm not sure).
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/54143/
thx @thvranken
DOC: remove double curly braces in the DataFrame.add_prefix docstring
State: closed

- [ x] closes "https://github.com/noatamir/scipy-2023-sprint/issues/10"
- [ x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

@noatamir  and @phofl Scipy2023 sprint

Labels:
name: Docs
name: Sprints
description: Sprint Pull Requests

Comments:
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/54142/
Thanks @vvaidy!
Add type literals to engine argument in groupby aggregations
State: closed

- [x] closes https://github.com/noatamir/scipy-2023-sprint/issues/7 groupby.py
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature

@noatamir @phofl SciPy 2023 Sprint!

Labels:
name: Typing
description: type annotations, mypy/pyright type checking
name: Sprints
description: Sprint Pull Requests

Comments:
thx @ankile
Typing improvements for MultiIndex
State: closed

SciPy 2023 sprint

@phofl @noatamir 

Typing improvements for MultiIndex #6

- [x] Add sortorder for MultiIndex.from_arrays
- [x] Add sortorder for MultiIndex.from_frame
- [x] Add names for MultiIndex.from_frame
- [x] Add name and names for MultiIndex.copy



Labels:
name: Sprints
description: Sprint Pull Requests

Comments:
Thanks @cnguyen-03 !
Scipy 2023 sprint: Added type hints
State: closed

Added type hints during the SciPy 2023 sprint.
https://github.com/noatamir/scipy-2023-sprint/issues/5


- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.

@phofl, @noatamir


Labels:
name: Stale
name: Sprints
description: Sprint Pull Requests

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.