BrandonFors/pandas-github-issues huggingface dataset:
REF: use single-test-class for datetimetz, period tests
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
```bash
mypy.....................................................................Failed
- hook id: mypy
- duration: 127.2s
- exit code: 1

pandas/tests/extension/test_period.py:95: error: Item "dtype[Any]" of "dtype[Any] | ExtensionDtype" has no attribute "freq"  [union-attr]
pandas/tests/extension/test_period.py:95: error: Item "ExtensionDtype" of "dtype[Any] | ExtensionDtype" has no attribute "freq"  [union-attr]
pandas/tests/extension/test_datetime.py:131: error: Item "dtype[Any]" of "dtype[Any] | ExtensionDtype" has no attribute "tz"  [union-attr]
pandas/tests/extension/test_datetime.py:131: error: Item "ExtensionDtype" of "dtype[Any] | ExtensionDtype" has no attribute "tz"  [union-attr]
Found 4 errors in 2 files (checked 1413 source files)
```
updated+green
Thanks @jbrockmendel
Smcbeth/series write parquet
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
-1 on adding api

there is no reason to support this as it's not round trippable - nor is Series a common format out there. just add a doc recipe.
@jreback ah okay Im happy to dump this if not worthwhile, would you mind taking a look / commenting on the issue [here](https://github.com/pandas-dev/pandas/issues/54638)? Your point about not being round trippable was brought up and debated
Closing as this feature probably needs more discussion
These kinds of inconsistencies in the pandas API and friggin' infuriating. Users don't particularly distinguish between a series and dataframe in working code. For example, the result of doing pd.concat on some selection from a dataframe might be a dataframe or a series depending on how many columns match the criteria. Borking a downstream serialisation because your frame turned into a series is not fun. Arguing that to_parquet on a series is not round-trippable so shouldn't be supported is childish arrogance.
CLN: assorted
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Clean

Comments:
```
2023-08-21T21:16:17.4537902Z /home/runner/work/pandas/pandas/pandas/core/indexes/base.py:docstring of pandas.core.indexes.base.Index.get_indexer:44: CRITICAL: Unexpected section title or transition.
2023-08-21T21:16:17.4538391Z 
2023-08-21T21:16:17.4538801Z -----
2023-08-21T21:16:17.4539546Z /home/runner/work/pandas/pandas/pandas/core/indexes/base.py:docstring of pandas.core.indexes.base.Index.get_indexer:44: CRITICAL: Unexpected section title or transition.
2023-08-21T21:16:17.4539976Z 
2023-08-21T21:16:17.4540160Z -----
2023-08-21T21:16:17.4540647Z /home/runner/work/pandas/pandas/pandas/core/indexes/base.py:docstring of pandas.core.indexes.base.Index.get_indexer:44: CRITICAL: Unexpected section title or transition.
2023-08-21T21:16:17.4541159Z looking for now-outdated files... none found
2023-08-21T21:16:17.4541409Z 
2023-08-21T21:16:17.4541582Z -----
```
Thanks @jbrockmendel
BUG: drop_duplicates raising for boolean arrow dtype with missing values
State: closed

- [x] closes #54667 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

cc @mroeschke 

More generally, how do we want to proceed with Arrow EA fixes after 2.1 is out?

Labels:
name: Arrow
description: pyarrow functionality

Comments:
> More generally, how do we want to proceed with Arrow EA fixes after 2.1 is out?

Since Arrow support isn't new anymore I suppose we shouldn't be backporting bug fixes as aggressively anymore, but I would still be okay with it
Ok then maybe lets backport low risk fixes and decide on the others on a case by case basis
Thanks @phofl
ENH: numba engine in df.apply
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Apply
description: Apply, Aggregate, Transform, Map
name: numba
description: numba-accelerated operations

Comments:
gentle ping @mroeschke.

I'm planning on following this up with general support for numba in df.apply. 
pre-commit.ci autofix
pre-commit.ci autofix
Nice! Thanks @lithomas1 
Thanks for the review!
DEPR: BaseNoReduceTests
State: closed

I'm really not interested in writing tests for the tests

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Deprecate
description: Functionality to remove in pandas

Comments:
Would be good to include this in the 2.2.0 whatsnew
> Would be good to include this in the 2.2.0 whatsnew

updated+greenish
Thanks @jbrockmendel
REF: remove is_simple_frame from json code
State: closed

cc @WillAyd the simple_frame path is kludgy, I expect fragile to non-numpy blocks.  This just rips it out to simplify the code.

Labels:
name: Refactor
description: Internal refactoring of code
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
Any perf impact?
```
import numpy as np
import pandas as pd

arr = np.random.randn(10**4, 10)
df = pd.DataFrame(arr)

In [4]: %timeit df.to_json()
12.8 ms ± 1.72 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <- main
12.4 ms ± 448 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <- PR

In [5]: %timeit df.T.to_json()
12 ms ± 413 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <- main
19.2 ms ± 309 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  # <- PR
```

So for very-wide cases this takes a hit, but indistinguishable for normal cases.
I think that is fine. Not worth optimizing to_json for very wide frames at the ms level
Thanks @jbrockmendel
Backport PR #54586 on branch 2.1.x (REF: Refactor conversion of na value)
State: closed

Backport PR #54586: REF: Refactor conversion of na value

Labels:
name: Strings
description: String extension data type and string data
name: Arrow
description: pyarrow functionality

Comments:
@phofl 
Does this need to be backported?

It says this is a pre-cursor for #54585, and that seems to be a pre-cursor for #54533 which will probably miss 2.1.


No none of those should miss the release, lets merge this, it's only copy-paste things around anyway which is not user visible
BUG: _apply_rule() ignores tz info on an empty list of observances
State: closed

- [x] closes #54580 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Timezones
description: Timezone data dtype

Comments:
This seemed eerily familar -- reminded me of #49075.

Figured out that `_apply_rule()` was silently ignoring the timezone data if you passed it an empty list of holiday observances that were computed given start and end dates that had timezone info.

Thanks @roadswitcher
DOC: fix build failure with sphinx 7 and SOURCE_DATE_EPOCH
State: closed

When SOURCE_DATE_EPOCH is set (which is the default when building Debian packages), Sphinx 7 [**requires** a space or comma after the year in 'copyright'](https://sources.debian.org/src/sphinx/7.1.2-2/sphinx/config.py/#L444).

[Debian bug report, with failure log](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1042672).

- [not known to be reported here ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ n/a] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ CI says yes] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [n/a ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [no ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks @rebecca-palmer
Add nan_count in Describe Functions with all conditions, made all request.
State: closed

Hey, 
I am Aditya tomar from India.
I found an issue #54076. 
I added nan_count in describe function for dataframe, series and timestamps.

Kindly request to you please check it out.
for your feedback 

Mail : adityatomarsvnit@gmail.com


Labels:

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
ENH: Add on_bad_lines for pyarrow
State: closed

- [x] closes #54480 
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Update documentation

This adds the `on_bad_lines` argument to the `pyarrow` engine for the `from_csv` parser that closely follows the behaviour of the `python` engine. Internally utilizes [pyarrow's invalid_row_handler](https://arrow.apache.org/docs/python/generated/pyarrow.csv.ParseOptions.html#pyarrow.csv.ParseOptions.invalid_row_handler). The built-in callable implementation slightly differs for pyarrow, so the difference is appropriately documented by pointing to pyarrow's documentation.

Usage Example:

example.csv:
```
a,b,c
acol1,bcol1,ccol1
acol2,ccol2
```

example.py 
```
df_arrow = pd.read_csv(r"example.csv", engine='pyarrow', dtype_backend='pyarrow', on_bad_lines='warn')
print(df_arrow)
```

Console output
```
ParserWarning: Expected 3 columns, but found 2: acol2,ccol2
       a      b      c
0  acol1  bcol1  ccol1
```

Labels:
name: IO CSV
description: read_csv, to_csv
name: Arrow
description: pyarrow functionality

Comments:
Does this fix any tests?

There are some xfailed tests for ``on_bad_lines`` I think somewhere (sorry that I don't remember where) for the pyarrow engine.
@lithomas1 The tests have been refactored to not fail for both the "python" engine and the "pyarrow" engine (see: [pandas/tests/io/parser/test_unsupported.py](https://github.com/pandas-dev/pandas/pull/54643/files#diff-2a0f93df26fa0ee1d10f216142db735ff227fcfb02f161ad165fdfcb3f767cd1)) 

I didn't find any other tests that were invalidated (as confirmed by the CI). 
@lithomas1 btw, is there any specific action I should be taking to request a review?
Rebased from master to pull in compat workaround so that CI Passes
Can you add a whatsnew entry to 2.2.0.rst?
> @lithomas1 The tests have been refactored to not fail for both the "python" engine and the "pyarrow" engine (see: [pandas/tests/io/parser/test_unsupported.py](https://github.com/pandas-dev/pandas/pull/54643/files#diff-2a0f93df26fa0ee1d10f216142db735ff227fcfb02f161ad165fdfcb3f767cd1))
> 
> I didn't find any other tests that were invalidated (as confirmed by the CI).

Try looking in pandas/tests/io/parser/common/test_read_errors.py

Those tests are currently skippped, but on_bad_lines is being used there. You should be able to make at least some of them pass.

(You can try xfailing the tests there by changing the "pyarrow_skip" to a "pyarrow_xfail")
@lithomas1 I've gotten all the tests within `test_read_errors.py` applicable for this change to pass. 
@lithomas1 Are there any other changes that you'd like me to look at from the test perspective?
Any updates on this PR?
@lithomas1 Anything else that needs to be covered in this pull request?
@lithomas1 I've chained the ArrowInvalid exception with ParserError so that the tests are uniform. Does this match what you had in mind?
@lithomas1 Awaiting your review
Thanks @amithkk
BUG: getitem indexing wrong axis
State: closed

- [x] closes #54622 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Regression
description: Functionality that used to work in a prior pandas version
name: Blocker
description: Blocking issue or pull request for an upcoming release

Comments:
Thanks @phofl
Rewording a sentance in tech docs
State: closed

I fixed a sentence grammatically to sound better.

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @StEmGeo
Updated `ruff pre-commit` version and modified few lines according to it.
State: closed

#### Ruff 0.0.285 has been released (https://github.com/astral-sh/ruff-pre-commit).
So, I have updated the ruff version in `.pre-commit-config.yaml` file, ran `ruff .` and corrected few files according to the errors it showed.

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
@mroeschke sir, can you please review the PR, Thank you.
Thanks @Sai-Suraj-27
BUG [fix] prevent potential memory leak
State: closed

- [x] closes #54628
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Performance
description: Memory or execution speed performance
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
DEPR: make arguments keyword only in to_clipboard
State: closed

- [x] xref #54229 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/v2.2.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: Deprecate
description: Functionality to remove in pandas

Comments:
@mroeschke is anything more required here? 
edit: reference to your [comment]( https://github.com/pandas-dev/pandas/issues/54229#issuecomment-1684166854)
@mroeschke added test for the same.
Just a merge conflict that needs fixing
@mroeschke resolved
Thanks @rsm-23
DEP: remove python-snappy and brotli as optional dependencies (no longer used)
State: closed

See https://github.com/pandas-dev/pandas/issues/32417#issuecomment-1684356585 for context.

The `python-snappy` and `brotli` packages were in the past a dependency of fastparquet, but that is no longer the case (https://github.com/dask/fastparquet/commit/2e657224a7250d49aa2e4b8c457a98e008687c21, replaced by `cramjam` that provides both compression options). 
I am not sure if there was another reason that we included this in our optional dependencies, but I didn't find any other usage in our codebase. 

Closes #32417

Labels:
name: Dependencies
description: Required and optional dependencies

Comments:
thanks @jorisvandenbossche
DEPR: deprecated nonkeyword arguments in to_csv
State: closed

- [x] xref #54229 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/v2.2.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO CSV
description: read_csv, to_csv
name: Deprecate
description: Functionality to remove in pandas

Comments:
@mroeschke can you please review this?
Thanks @rsm-23
DEPR: deprecated nonkeyword arguments in to_parquet
State: closed

- [x] xref #54229 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/v2.2.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: Deprecate
description: Functionality to remove in pandas
name: IO Parquet
description: parquet, feather

Comments:
@mroeschke can you please help me understand what's causing this error?
>AssertionError: Warning not set with correct stacklevel. File where warning is raised: /home/runner/micromamba/envs/test/lib/python3.9/site-packages/pyarrow/pandas_compat.py != /home/runner/work/pandas/pandas/pandas/tests/io/test_parquet.py. Warning message: is_sparse is deprecated and will be removed in a future version. Check `isinstance(dtype, pd.SparseDtype)` instead.
You can set `check_stacklevel=False` for your test. Seems like a 3rd party warning is messing up the stacklevel checking
@mroeschke still getting some error like this. Can you please help on this?

> pandas/tests/io/test_parquet.py::test_parquet_pos_args_deprecation - ImportError: Unable to find a usable engine; tried using: 'pyarrow', 'fastparquet'.
A suitable version of pyarrow or fastparquet is required for parquet support.
Thanks @rsm-23
DEPR: make arguments keyword only in to_dict
State: closed

- [x] xref #54229 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/v2.2.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
@mroeschke should I remove the `*` and add the deprecation decorator here as well?
Yes
@jorisvandenbossche @mroeschke I am done with the changes. 
Thanks @rsm-23