BrandonFors/pandas-github-issues huggingface dataset:
BUG: avoid segfault in tz_localize with ZoneInfo
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Timezones
description: Timezone data dtype

Comments:
> does this need a whatsnew note if it fixes a segfault?

Nope, only relevant in main, not in any released version.

> Also, is there a way to reproduce the segfault locally?

```
dti = pd.date_range(start="2015-03-29 02:00:00", periods=60, freq="min")
tz =  zoneinfo.ZoneInfo("Europe/Warsaw")
dti.tz_localize(tz, nonexistent="shift_forward")
```
Ah I was getting `AttributeError`, perhaps because I was using the zoneinfo backport on python3.8
```
In [1]: from backports import zoneinfo

In [2]: dti = pd.date_range(start="2015-03-29 02:00:00", periods=60, freq="min")
   ...: tz =  zoneinfo.ZoneInfo("Europe/Warsaw")
   ...: dti.tz_localize(tz, nonexistent="shift_forward")

AttributeError: 'NoneType' object has no attribute 'total_seconds'
```
I'll try in python3.9
Thanks @jbrockmendel
CI_check_correctness_notebooks
State: closed

This PR is related to the PR #49874.
Creating new PR, that checks correctness notebooks, as was discussed with @MarcoGorelli .
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

CI didn't fail when documentation notebook executed with an error. I added new notebooks section in `ci/code_checks.sh` to check if the notebook execution passes.

Labels:
name: CI
description: Continuous Integration

Comments:
> Could you also add a step to `.github/workflows/code-checks.yml` to run this check in the CI?

Thanks @mroeschke , added this in the second commit.
Awesome, thanks @natmokval
Bug: Incorrect IntervalIndex.is_overlapping
State: closed

- [x] closes #49581 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.



Labels:
name: Interval
description: Interval data type

Comments:
Also, is `pandas/_libs/intervaltree.pxi.pdf` supposed to be committed?
@mroeschke Fixed.

Edit: Strangely it fails the "Python Dev / actions-311-dev (macOS-latest) (pull_request)" integration test.
Edit: Now passed.
> Could you remove `pandas/_libs/intervaltree.pxi.pdf`?

I've removed this file, is there any thing I've to do now?
thanks @Hikipeko
BUG: metadata does not always propagate in binary operations 
State: closed

- [x] closes #49916 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: metadata
description: _metadata, .attrs
name: Stale

Comments:
If this works I'm pretty sure it'll close #34373.

When we're ready to discuss deprecations for 3.0, I think we should kill off `__finalize__`.  There's no real prospect of getting it fully-working.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
I'll get back to this later

> When we're ready to discuss deprecations for 3.0, I think we should kill off __finalize__. There's no real prospect of getting it fully-working.

yeah agreed
CLN: clean core.construction._extract_index
State: closed

Return early in the case of len(data) == 0, thereby avoiding needing a long `else` clause. Also clean type hints.

EDIT: The PR looks a bit messy at first sight, but it's mostly just dedenting stuff. Checking "Hide whitespace" makes the PR much clearer, as suggested by @MarcoGorelli in #49921.

Labels:
name: Clean

Comments:
Updated.
BLD: use nonvendor versioneer
State: closed

This PR:
1. move versioneer config to pyproject.toml
2. use nonvendored version of versioneer

After this, we could enable PEP 517. 

Labels:
name: Build
description: Library building on various platforms

Comments:
@lithomas1 
Does meson-Python require versioneer to be vendored? 
> Does meson-Python require versioneer to be vendored?

Seems to work fine.
Awesome, thanks @fangchenli. Always great to not vendor things if we don't have to
API: ensure read_json closes file handle
State: closed

Use context manager instead of closing manually with `self.close` to ensure file always closes.

Not sure if this should have a whatsnew entry and where.

Labels:
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
> easier to review with the "hide whitespace" option

Nice one, I didn't know about that one. It does indeed makes changes that are mostly changing the indention level easier to review.
Yeah I don't think this needs a whatsnew entry - leaving open for a bit in case others have comments / disagree
Could note in the `Other API changes` that `read_json` will always close the file if there was an error in reading the file?
Thanks for the comments.

I've added a whatsnew entry + updated the PR to also improve the handling of closing files in `JsonReader.__next__` also.
> Would it be easy to test that the files handles are closed when an exception is encountered?

not sure, maybe serving a malformed file (though haven't checked if that is tested already or maybe caught in the init)?
error looks unrelated ("ERROR: Could not find a version that satisfies the requirement setuptools==59.2.0")
Thanks @topper-123. If you have any inspirations on how to test that files are closed on exceptions happy to have a follow up
Json fix normalize
State: closed

closes #49861 


Labels:
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
Thanks @WillAyd 
This patch may have induced a potential regression. Please check the links below. If any ASVs are parameterized, the combinations of parameters that a regression has been detected appear as subbullets. This is a partially automated message.

 - https://asv-runner.github.io/asv-collection/pandas/#io.json.NormalizeJSON.time_normalize_json
   - orient='columns'; frame='df'
   - orient='columns'; frame='df_date_idx'
   - orient='columns'; frame='df_int_float_str'
   - orient='columns'; frame='df_int_floats'
   - orient='columns'; frame='df_td_int_ts'
   - orient='index'; frame='df'
   - orient='index'; frame='df_date_idx'
   - orient='index'; frame='df_int_float_str'
   - orient='index'; frame='df_int_floats'
   - orient='index'; frame='df_td_int_ts'
   - orient='records'; frame='df'
   - orient='records'; frame='df_date_idx'
   - orient='records'; frame='df_int_float_str'
   - orient='records'; frame='df_int_floats'
   - orient='records'; frame='df_td_int_ts'
   - orient='split'; frame='df'
   - orient='split'; frame='df_date_idx'
   - orient='split'; frame='df_int_float_str'
   - orient='split'; frame='df_int_floats'
   - orient='split'; frame='df_td_int_ts'
   - orient='values'; frame='df'
   - orient='values'; frame='df_date_idx'
   - orient='values'; frame='df_int_float_str'
   - orient='values'; frame='df_int_floats'
   - orient='values'; frame='df_td_int_ts'
@rhshadrach awesome bot. Will take a look - moving the import to the global space might help
DOC: Table Visualization gets Error for cell #49874
State: closed

- [x] closes #49874
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Building the documentation passes locally.


Labels:
name: Styler
description: conditional formatting using DataFrame.style

Comments:
Thanks, @MarcoGorelli. I fixed the problem with `hide_index()` and added a new commit for that. Now the notebook executes without errors. 

I wanted to add the command `jupyter nbconvert --execute doc/source/user_guide/style.ipynb --to notebook` in [code_checks.sh](https://github.com/pandas-dev/pandas/blob/main/ci/code_checks.sh), but didn't find a suitable place for it. Checking notebook compilation doesn't seem to be `doctests` or `single-docs`.  Could you advise me? Maybe we can check for failure during the documentation building process (`python make.py`)?

Sure - I think a new `notebooks` section would be good. It should be fine for it to just run `jupyter nbconvert --execute doc/source/user_guide/style.ipynb --to notebook`, and that should already exit `1` if it fails and `0` if it succeeds

I haven't tried it, but something like `jupyter nbconvert --execute $(find -name '*.ipynb') --to notebook` might work

Anyway, that'd be for a separate PR, thanks for your contribution here!
BLD: Move options to pyproject.toml
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Spat out from #49115.
Versioneer needed an upgrade to read from pyproject.toml, but the update should be pretty low-risk/benign.


Labels:
name: Build
description: Library building on various platforms

Comments:
Thanks @lithomas1
DOC: update running the test suite documentation
State: closed

- [x] closes #48060
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

I updated the [Running the test suite](https://pandas.pydata.org/docs/dev/development/contributing_codebase.html#running-the-test-suite) documentation as discussed in [#48060 (comment)](https://github.com/pandas-dev/pandas/issues/48060#issuecomment-1321066179).

Labels:
name: Docs

Comments:
Thanks @marenwestermann
Fix some little-endian assumptions in the tests
State: closed

See previous [PR#46681](https://github.com/pandas-dev/pandas/pull/46681).

- [ ] closes #xxxx **N/A**
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature **N/A, fixes some existing test failures on s390x**
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions. **N/A**
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature. **N/A**


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
can you run the pre-commit checks please?
Long line broken and force-pushed; pre-commit passes now.
Thanks @musicinmybrain
API: various .value_counts() result in different names / indices
State: closed

- [ ] closes #49497 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

---

screenshot of whatsnewnote:

![image](https://user-images.githubusercontent.com/33491632/204052033-a7b16b21-7f47-4ae1-a942-90895869d214.png)


Labels:
name: API Design
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
Good point on `rename`, thanks, sorry for having missed that - have updated
gentle ping 😄 do we still want to make this change?
cheers - @jbrockmendel any further comments?
thanks for your reviews, much appreciated!

merging then, before more conflicts come in
STYLE: Enable unnecessary list index lookup
State: closed

- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Associated with #48855.  Enables the Pylint type "R" warning `unnecessary-list-index-lookup`.  Also updates the list of intentionally disabled checks.


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
@MarcoGorelli , are you talking about the changes to `pyproject.toml`?  I took the liberty of updating the list of intentionally disabled warnings: `wrong-import-order` appeared twice in the list, while `broad-except` and `overridden-final-method` were in the list at the top of #48855 but not in the section for intentionally disabled warnings in `pyproject.toml`.

If you'd like, I can change them back.
ah, that's fine then, thanks for explaining!
WIP STYLE:  Suppress false positives for pylint W0703 ( broad-except) 
State: closed

- [ ] partially addresses style issues in #48855 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Removed `broad-except` from `pyproject.toml` in order to see if the warning can be successfully enabled.



Labels:

Comments:
Per comment from @MarcoGorelli on #48855  closing the PR -- leaving this disabled for now is probably for the best.
BUG: SeriesGroupBy.apply sets name attribute if result is DataFrame
State: closed

- [ ] closes #49907 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.



Labels:
name: Bug
name: Groupby
name: Regression
description: Functionality that used to work in a prior pandas version
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
@MarcoGorelli I added back the Bug tag. I've been doing that on regressions to differ between e.g. a performance regression and a behavior regression. Let me know if you think differently.
> @MarcoGorelli I added back the Bug tag. I've been doing that on regressions to differ between e.g. a performance regression and a behavior regression. Let me know if you think differently.

Hey - sounds good to me
Thanks @MarcoGorelli!
adding convert_cell to read_excel
State: closed

- [x ] closes #49872
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
The converters do the convertion after the reading (i'm not sure of that) and doesn't have access to the cell format. 
In my case i have a  convert a cell with the number 2345, with numeric_format 00000 to the string '02345'. In Excel you see this as a number with leading zeros, wich isn't valid python. If a use a converter i will convert the number 2345 to the string '2345'.
I'm working normalizing financial documents, and in a lot of cases they use format as a data, for example another color for negative numbers(not as condicional, as simple formatted), and things like that.
I don't know if this is useful for a lot of people. Another option, less agressive is just add in the read_excel documentation an explanation that, in the case you want to access cell format you have to override _convert_cell(cell). I'm new in the community, is better have the discussion of the pertinence of the request in the pull request, the issue or the slack?
Discussion should be on the issue. Please wait with an implementation until there is agreement
Thanks for the pull request, but agreed discussion is needed in the original issue first before moving forward with a PR, so closing until a decision has been made there
STYLE enable pylint: undefined-variable
State: closed

Issue #48855. This PR enables pylint warning: `undefined-variable`
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Labels:

Comments:
To fix the warning I had to refactor the function `determine_clipboard()`. Hopefully, I did it the right way.
STYLE pre-commit autoupdate, and other cleanups
State: closed

Summary of changes (separated out into separate commits for ease of review):

- run `pre-commit autoupdate`, bumping a few versions
- It's mostly using double-quotes in Cython files (cc @jbrockmendel 's suggestion in https://github.com/pandas-dev/pandas/pull/49866#discussion_r1030853463)
- remove some unused variables / imports from `pandas/_libs/parsers.pyx`
- rename variables named `l`, as that's a new flake8 check that disallows that, in:
  - `pandas/tests/frame/indexing/test_getitem.py`
  - `pandas/tests/io/pytables/test_round_trip.py`
  - `pandas/tests/io/pytables/test_round_trip.py`
- move `black` to local hooks, because like this it runs twice (!) as fast, [as they now compile with mypyc ](https://ichard26.github.io/blog/2022/05/compiling-black-with-mypyc-part-1/)
- add `--verbose` argument to CI job, so we can see how long each hook takes in CI
- move `yesqa` to the manual stage (so it only runs in CI), as it's kind of slow and unnecessary most of the time (I don't think I've ever had a local commit fail because of it)

---

If reviewing, I'd suggest looked at each commit separately as they're each self-contained

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
IMO would be better if each commit is a pull request as it would be easier to revert a specific change if for some reason there was an issue with one particular change here
that's a fair point - I'll split it up then, thanks!
DEPR: Change numeric_only to False in various groupby ops
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Labels:
name: Groupby
name: Deprecate
description: Functionality to remove in pandas
name: Nuisance Columns
description: Identifying/Dropping nuisance columns in reductions, groupby.add, DataFrame.apply

Comments:
does this finish off the nuisance column stuff?
@jbrockmendel 

> does this finish off the nuisance column stuff?

No - we still have some grouby ops (those that go through _op_via_apply) and then resampler / window.
small question, otherwise lgtm
Thanks @rhshadrach