BrandonFors/pandas-github-issues huggingface dataset:
PERF: sparse to_csv
State: closed

Improves `NDFrame.to_csv` performance for sparse dataframe by casting to dense before initializing `DataFrameFormatter`. Results in many fewer calls to `to_native_types` which saves time. Added a new ASV benchmark based on the example provided by OP in #41023.

Benchmark results:
```
[100.00%] ··· io.csv.ToCSVSparse.time_sparse_to_dense_to_csv                                        1.16±0.02s       before           after         ratio
     [56d82a9b]       [945a3525]
     <main~1>         <main>
-       13.6±0.1s       1.22±0.04s     0.09  io.csv.ToCSVSparse.time_sparse_to_csv

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```

- [x] closes #41023 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Performance
description: Memory or execution speed performance
name: IO CSV
description: read_csv, to_csv
name: Sparse
description: Sparse Data Type
name: Stale

Comments:
Revised asv benchmarks (upstream/main vs pr) after moving materialization to the chunk level to preserve memory. Chunk-level materialization takes longer than all-at-once materialization but is still a significant improvement over upstream/main.

```
[100.00%] ··· io.csv.ToCSVSparse.time_sparse_to_dense_to_csv                                         87.2±3ms
       before           after         ratio
     [9c9789c5]       [6d092ee6]
     <main^2>         <main>
-         641±4ms          285±3ms     0.44  io.csv.ToCSVSparse.time_sparse_to_csv

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```
Please don't wait for reviewers to resolve conversations
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
STYLE: fix pylint no-method-argument warning
State: closed

- [ ] Refers to one of the issues in #48855
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

I have removed "no-method-argument" from pyproject.toml and addressed all the lines that create a warning.  I followed this [documentation](https://docs.quantifiedcode.com/python-anti-patterns/correctness/method_has_no_argument.html) on anti-patterns and corresponding best practices. 

This is my first Hacktoberfest and I was wondering if you could add "hacktoberfest-accepted" label to this PR to count. 

Thanks, and please let me know if any changes are required. 😊

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
@MarcoGorelli, alright, thanks, I will wait then. Let me know if I should adjust anything. :)
thanks @gulyapulya !
Welcome! @MarcoGorelli, would it be possible to add "hacktoberfest-accepted" label to this PR? This will be much appreciated and help me in my first Hacktoberfest!
DOC: change 1.6.0 doc to 2.0.0
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
thx @jbrockmendel
API: retain non-nano dtype in DatetimeArray constructor
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Has one test that will fail until #49050.

Labels:
name: Datetime
description: Datetime data dtype
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
Thanks @jbrockmendel
REVERT caching in find_stack_level
State: closed

addresses https://github.com/pandas-dev/pandas/pull/48023#issuecomment-1275912681

Labels:
name: Warnings
description: Warnings that appear or should be added to pandas

Comments:
It undoes https://github.com/pandas-dev/pandas/pull/48023, but I did it manually instead of with `git revert` as there were too many conflicts to deal with
> It undoes #48023, but I did it manually instead of with `git revert` as there were too many conflicts to deal with

I did this again with `git revert` and braved all the conflicts, to be safer. Also checked through the whole diff and it looks fine
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.5.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 0106c26529900bad0561efb9c9180f7f016365b0
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #49053: REVERT caching in find_stack_level'
```

4. Push to a named branch:

```
git push YOURFORK 1.5.x:auto-backport-of-pr-49053-on-1.5.x
```

5. Create a PR against branch 1.5.x, I would have named this PR:

> "Backport PR #49053 on branch 1.5.x (REVERT caching in find_stack_level)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
BUG: DTA.isin with mismatched resos
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Also trims a bit off of the upcoming DTA._from_sequence non-nano PR

Labels:
name: Datetime
description: Datetime data dtype
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
Thanks @jbrockmendel
ENH: Add mask support for value_counts with dropna=False
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
BUG: guess_datetime_format doesn't guess 2018-01-01T00:00:00.000
State: closed

- [ ] closes #49043 (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Haven't added a whatsnew note as it's not yet user-facing

Labels:
name: Bug
name: Datetime
description: Datetime data dtype

Comments:
Thanks @MarcoGorelli
enhancement: Added 'errors=coerce' option in astype()
State: closed

- [ ] closes #48781  
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [errors=coerce in astype()](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/v1.6.1.rst` file if fixing a bug or adding a new feature.


Labels:
name: Stale

Comments:
Going to close for now pending the discussion in the original issue. Happy to have your thoughts in that issue if you are in support of this feature.
Why didn't we moved this PR ? Since it does not have a huge enhancement in the astype method. I would like to add this enhancement feature into astype (). Which help us to solve the issue. I hope! 
We would still be open to this enhancement given more support and discussion in #48781 first. 
Actually to stimulate discussion, I'll reopen this PR, sorry.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. Additionally this issue could use more discussion on the original issue before moving forward to closing to await resolution on the main issue
read_json `engine` argument integration
State: closed

Creating the PR so we can start the discussion. It's not complete yet I will keep updating the PR; however, any feedback of course is appreciated. 

My main question is, there is an `arrow_parser_wrapper.py` file has a `ArrowParserWrapper` class which mainly serves the `engine` argument of the `read_csv` method. I though of extending that classes functionality so it can serve both `read_json` and `read_csv` but wanted to make get your opinion about it.

I will also add tests but I am still reading the tests part from the contributing guidelines.

- [ ] closes #48893 (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO JSON
description: read_json, to_json, json_normalize
name: Arrow
description: pyarrow functionality

Comments:
> I though of extending that classes functionality so it can serve both read_json and read_csv but wanted to make get your opinion about it.

The CSV and JSON reading code is more-or-less distinct so I'd recommend against this.

I recommend just creating a new file in `pandas/io/json` like `arrow_parser.py` with the parsing class that can imported and used in `read_json`
Yeah that makes sense. Sounds good then, will do that. Thanks for the feedback!
I just wanted to give an update. I didn't forget about the issue, I'm working on it:) After days of staring (and exploring) between source code and me finally I was able to implement a logic the past two days and wrote a test for it. I have been trying to get the docker interpreter working so I can run the tests. After that I'll update the PR.
@mroeschke I have to close this PR since I messed up something in git flow and history got messed up while trying to squash commits. I have to create a new PR from main branch. Is there a closing process, any commands I should run?
Opened #49249 instead of this PR.
Closing in favor of https://github.com/pandas-dev/pandas/pull/49249
ENH: Implement io.nullable_backend config for read_parquet
State: closed

- [x] xref #48957 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.




Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: IO Parquet
description: parquet, feather
name: Arrow
description: pyarrow functionality

Comments:
@phofl any follow-ups here?
thx @mroeschke
STYLE check for ability to remove 'c-extension-no-member' from pylint exclusions
State: closed

- [ x] addresses one concern listed in #48855 .

Removed `c-extension-no-member` from `pyproject.toml`, ran the precommit checks in Docker build environment, found no instances of the error.

Looks like `c-extension-no-member` can be removed from the list of pylint exclusions in pyproject.toml


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
OK, that's decidedly odd -- nothing found in the code by pylint, but if I enable the check the pylint process fails on the Ubuntu workflow?    I'll have to look into this more.

it's almost certainly unrelated, don't worry about it
> it's almost certainly unrelated, don't worry about it

I'm choosing to view this as a Self Guided Learning Experience into the intricacies of Github Actions, and not as something to worry about.
thanks @roadswitcher
STYLE: Fix or suppress non-iterator-returned errors from pylint
State: closed

- [X] addresses one issue ( `non-iterator-returned` ) from #48855

Found one instance of a test that's deliberately returning an Iterator to fool `is_file_like`, suppressing the warning as a false positive based on nature of test.

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks @roadswitcher
TST/CLN: Avoid subprocess shell=True
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).



Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean

Comments:
LGTM pending green
thx @mroeschke
REG: fix regression in df.corrwith on tied data when method is spearman
State: closed

- [x] closes #48826
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
name: Stale

Comments:
The failure comes from `numeric_only`. Is this implement on purpose? I think both results should be empty df. @mroeschke @phofl 
```
>>> pd.DataFrame({"A": [True]}).select_dtypes(np.number)
Empty DataFrame
Columns: []
Index: [0]
>>> pd.DataFrame({"A": [True]}, dtype="boolean").select_dtypes(np.number)
      A
0  True
```
"boolean" is currently not considered numeric
@mroeschke please move this to milestone 2.0.
> @mroeschke please move this to milestone 2.0.

Update to 1.5.2 for now, thanks @GYHHAHA 
Moving to 2.0 as we just reverted the original PR in https://github.com/pandas-dev/pandas/pull/49140 and I think it's safer than "re-reverting" for 1.5.2
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
STYLE: suppress false positive pylint findings for 'not-an-iterable'
State: closed

- [ x] addresses one of the identified issues in  #48855 

Disabled pre-commit Pylint checks for `not-an-iterable` in `pyproject.toml`, found a few identified problems, all of which appear to be false positives.


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks for working on this - not overly keen on adding lots of inline comments, if these are all false positives and it's not easy to address them then I think it's ok to not turn off this check

if somewhere in the code there is iteration over something non-iterable, the tests should catch that anyway
> thanks for working on this - not overly keen on adding lots of inline comments, if these are all false positives and it's not easy to address them then I think it's ok to not turn off this check

Makes perfect sense.    I've moved on to another line item off the chart -- but if there's a "better" way that Pylint wants developers to handle iterables in these cases, I'm very curious to figure that out.

Apparently my branch had fallen behind `main` so far that when I synced it in Github interface, I chose wrong button and closed it?

Anyways, I have an idea that I'm testing locally now, if it works, I'll reopen the PR.

OK, pylint apparently assumes if you take a slice of something, it must be an iterable.

Only suppressed two warnings in one file, but in that function it's already got a `TODO` to refactor/combine with function from another file.
I think having to do `[:]` to get around this warning would be unclear to contributors - I don't think this particular error message helps us avoid bugs that wouldn't be caught by the test suite, so I think we can just keep it turned off
> I think having to do [:] to get around this warning would be unclear to contributors 

I think having to do `[:]` to suppress this raises questions about the pylint parser.
DOC: Remove mention of make
State: closed

The makefile was deleted in ca2d5ad6ad34ca2d3d9b82e89268b6a4f87fc68d.

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @akx
pylint: disable invalid-repr-returned in Series.__repr__
State: closed

- [x] closed one of the issues in #48855 
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Since the params passed are for getting the repr string. This should not be an issue and `to_string` will return a `str`


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Hi @MarcoGorelli 

Will the cancelled test be re-run?
yeah, don't worry about it - I'll take a look later in the week, curious to see if there's a way to avoid having to turn off the warning at all
🤔 not sure what the issue is - if it's already string, I'd rather not add an unnecessary `str(` to please pylint. I'm surprised that no error is caught in `frame.py` in the same function. I'll take a look
Thanks @vamsi-verma-s