BrandonFors/pandas-github-issues huggingface dataset:
BUG: Fix date format identification for dict
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

cc @MarcoGorelli 

Labels:
name: Datetime
description: Datetime data dtype
name: IO CSV
description: read_csv, to_csv

Comments:
that was fast 🏃 
Thanks @phofl
DEPR: silently ignoring unrecognized timezones
State: closed

- [x] closes #18702 (Replace xxxx with the GitHub issue number)
- [x] closes #51476
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


WIP part is need to figure out #51476 before the deprecation warning will be issued by to_datetime.
Update: addressed.

Labels:
name: Timezones
description: Timezone data dtype
name: Deprecate
description: Functionality to remove in pandas

Comments:
updated to handle strptime inference cc @MarcoGorelli 
> WIP part is need to figure out https://github.com/pandas-dev/pandas/issues/51476 before the deprecation warning will be issued by to_datetime.

Looks like you've figured this out? `guess_datetime_format` now raises the deprecation warning
> Is the idea just to raise the warning on the pandas side, so that we have control over it and can raise in the future?

yes.

> Looks like you've figured this out? guess_datetime_format now raises the deprecation warning

yes.
note was added for 2.0, do we want to backport or move the note?
probably better to move the note.  will add to my next Assorted branch
thx
ERR: Improve error message in validate_percentile
State: closed

- [x] closes #51436 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This PR improves the validation of percentiles in pandas. Previously, when an out-of-range value was detected, the code would suggest a wrong value. This suggestion could be confusing to users and was not always necessary. This PR simplifies the suggestion message to directly state that the value should be between 0 and 1. 

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
@jbrockmendel 
I apologize for the mistake I made earlier. 
@phofl, Is this ok?
thx @ShashwatAgrawal20
DEPS: Partial Matplotlib 3.7 support
State: closed

xref https://github.com/pandas-dev/pandas/issues/51394

These pass locally with 3.7, the remaining issue will be address with https://github.com/matplotlib/matplotlib/issues/25219

Labels:
name: Visualization
description: plotting
name: Dependencies
description: Required and optional dependencies

Comments:
You may have to do version checks for `legend_handles` vs `legendHandles` if you don't want to pin to mpl3.7, the snake case version is new, but is preferred going forward, the camel case version was not actually meant to be fully public (it wasn't documented, but it also didn't have an `_`, so it got used)
thx @mroeschke 

I think we should backport, feel free to close the pr if you think differently
Agreed would be worth backporting
DOC: Remove 1.0.0 versionadded/versionchanged
State: closed

- [ ] xref #51071 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Removing 1.0.0 is non-controversial based on the discussion in the op. So want to get this removed for 2.0 to clean up our docs a bit

Labels:
name: Docs

Comments:
```
Error: /home/runner/work/pandas/pandas/pandas/core/indexes/multi.py:597:PR09:pandas.MultiIndex.from_product:Parameter "names" description should finish with "."
Error: /home/runner/work/pandas/pandas/pandas/core/arrays/integer.py:62:SS03:pandas.arrays.IntegerArray:Summary does not end with a period
Error: /home/runner/work/pandas/pandas/pandas/core/arrays/integer.py:62:SS06:pandas.arrays.IntegerArray:Summary should fit in a single line
Error: /home/runner/work/pandas/pandas/pandas/core/arrays/integer.py:155:SS06:pandas.Int8Dtype:Summary should fit in a single line
Error: /home/runner/work/pandas/pandas/pandas/core/arrays/integer.py:162:SS06:pandas.Int16Dtype:Summary should fit in a single line
Error: /home/runner/work/pandas/pandas/pandas/core/arrays/integer.py:169:SS06:pandas.Int32Dtype:Summary should fit in a single line
Error: /home/runner/work/pandas/pandas/pandas/core/arrays/integer.py:176:SS06:pandas.Int64Dtype:Summary should fit in a single line
Error: /home/runner/work/pandas/pandas/pandas/core/arrays/integer.py:183:SS06:pandas.UInt8Dtype:Summary should fit in a single line
Error: /home/runner/work/pandas/pandas/pandas/core/arrays/integer.py:190:SS06:pandas.UInt16Dtype:Summary should fit in a single line
Error: /home/runner/work/pandas/pandas/pandas/core/arrays/integer.py:197:SS06:pandas.UInt32Dtype:Summary should fit in a single line
Error: /home/runner/work/pandas/pandas/pandas/core/arrays/integer.py:204:SS06:pandas.UInt64Dtype:Summary should fit in a single line
```
thx
Thanks @phofl
CI: Cancel 32 bit when another job is started
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

saw this on a pr of mine. Job got cancelled after I merged main

Labels:
name: CI
description: Continuous Integration

Comments:
Thanks @phofl
CoW: Ignore copy=True when copy_on_write is enabled
State: closed

- [ ] closes #50535 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Copy / view semantics

Comments:
This basically closes https://github.com/pandas-dev/pandas/issues/50535? (although maybe not every method listed there is already covered)
I actually forgot about this issue. I double checked, we covered every case except transpose (added a test for this). So should be good to close when this is merged
Thanks @phofl
REF: move SelectN from core.algorithms
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
Thanks @jbrockmendel
WEB: Add code highlighting in PDEPs and render tables correctly
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Discovered that tables were rendering really nastily and there was no code highlighting so PDEPs were looking a little bland while working on PDEP 7.

Note: The codehilite.css file that I added was generated by pygments, and is based on the ``friendly`` style
(I'll try to find the pydata-sphinx-theme style css file though).

Labels:
name: Web
description: pandas website

Comments:
Nvm, we basically have the same thing as pydata-sphinx-theme, the only difference is that we don't have the box around the code block so it still looks kinda nasty.
<img width="539" alt="image" src="https://user-images.githubusercontent.com/47963215/219721819-ae8bb99f-7d7c-46ae-8479-7f15399ece77.png">

This is what it looks like.
We clearly have been noticing the same things ;) The last days, I was thinking we could also use sphinx (with myst) instead to solve many of those issues. I wrote up https://github.com/pandas-dev/pandas/issues/51461 about that
Preview docs at: https://pandas.pydata.org/preview/51458/
Looks like you're changing the background color of the navigation bar, not sure if that's intended. I like how it looks like actually, but feels like this PR shouldn't change it, right?

Also, we can address it in a follow up PR, but I think we can add some padding and maybe a border to the code boxes: https://pandas.pydata.org/preview/51458/pdeps/0004-consistent-to-datetime-parsing.html

Happy to take care of that myself in a follow up PR, but probably easy to add it here too.
> Looks like you're changing the background color of the navigation bar, not sure if that's intended. I like how it looks like actually, but feels like this PR shouldn't change it, right?

My bad. Just realized now that I'm in a different monitor, where the color looks darker. Interesting that everything else looks the same, except of our navigation bar. Sorry about the noise.
> Also, we can address it in a follow up PR, but I think we can add some padding and maybe a border to the code boxes: https://pandas.pydata.org/preview/51458/pdeps/0004-consistent-to-datetime-parsing.html

Thanks @lithomas1. I think we can address the above comment in a follow up, but nice improvement to the website!
TST: un-xfail NaT test on npdev
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
npdev is green, the failing test is an unrelated file leak. merging.
DOC: Add user guide section about copy on write
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: Copy / view semantics

Comments:
Thx for reviewing. We will have do explain this more extensively in follow-ups I guess, but I wanted to move the optimization out of the whatsnew before the rc is due 
merging, will add another part about the side-effect things in a follow up
DOC: fix EX02 errors in docstrings
State: closed

Related to the issue #51236

This PR enables functions:
`pandas.core.groupby.DataFrameGroupBy.take`
`pandas.Timestamp.fromtimestamp`

Labels:
name: Docs

Comments:
> thanks @natmokval - looks like `DataFrameGroupBy.take` was already passing then?
> 
It was fixed when I deleted `pandas.core.groupby.SeriesGroupBy.take` function from the ignore list in the previous PR #51432. It looks like the same changes in `pandas/core/groupby/generic.py` are required.
API: Use lowest precision floating type found in reindex with columns
State: closed

- [ ] closes #51059 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

The problem there was that the ``.values`` call here
https://github.com/pandas-dev/pandas/blob/a131266c8ab7c83f73331b5be5ea49d9a628f075/pandas/core/reshape/reshape.py#L746
upcasts the values, so the float32 will become float64.

I had to modify reindex in order to make this possible, since I don't think you can easily tell whether an all-nan col was there before or was created because of re-indexing. 
   - I guess could modify stack, but then I'd be duplicating reindex logic.


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
I think I overcomplicated this.
CI/TST: Error on PytestUnraisableExceptionWarning instead of using psutil to check open resources
State: closed

Using `psutil` to check for open resources in tests has been a false positive for a while, picking up unrelated connections such as the database services.

Erroring on `ResourceWarnings`, captured by `PytestUnraisableExceptionWarning`, should be sufficient for validating that resources are not left open as a result of a test. https://github.com/pytest-dev/pytest/issues/9825

Labels:
name: CI
description: Continuous Integration
name: Unreliable Test
description: Unit tests that occasionally fail

Comments:
there have been times in the past where we _did_ leave files open incorrectly, which I think the check this removes would catch.  does the check this adds still catch those?
Correct for example

```
 % git diff
diff --git a/pandas/tests/frame/test_api.py b/pandas/tests/frame/test_api.py
index e5787a7f16..c280fb9748 100644
--- a/pandas/tests/frame/test_api.py
+++ b/pandas/tests/frame/test_api.py
@@ -379,3 +379,7 @@ class TestDataFrameMisc:
         df = DataFrame()
         with tm.assert_produces_warning(None):
             inspect.getmembers(df)
+
+
+def test_open():
+    open("/dev/null")

% pytest pandas/tests/frame/test_api.py

...

pandas/tests/frame/test_api.py .................................F

...
    def unraisable_exception_runtest_hook() -> Generator[None, None, None]:
        with catch_unraisable_exception() as cm:
            yield
            if cm.unraisable:
                if cm.unraisable.err_msg is not None:
                    err_msg = cm.unraisable.err_msg
                else:
                    err_msg = "Exception ignored in"
                msg = f"{err_msg}: {cm.unraisable.object!r}\n\n"
                msg += "".join(
                    traceback.format_exception(
                        cm.unraisable.exc_type,
                        cm.unraisable.exc_value,
                        cm.unraisable.exc_traceback,
                    )
                )
>               warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E               pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
E
E               Traceback (most recent call last):
E                 File "...pandas/tests/frame/test_api.py", line 385, in test_open
E                   open("/dev/null")
E               ResourceWarning: unclosed file <_io.TextIOWrapper name='/dev/null' mode='r' encoding='UTF-8'>

../opt/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarnin
```
awesome, thanks for explaining
Looks like this caught actual issues with open files in `ExcelWriter`
merging, I guess we should backport to get rid of these on the 2.0.x branch?
thx @mroeschke 
nice!
ENH: Improve ref-tracking for group keys
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

cc @jorisvandenbossche 

Follow-up on https://github.com/pandas-dev/pandas/pull/49450

The equals check causes failures in Dask. I think we are better off moving to reference checking anyway. If you are ok with the general logic here, I'd refactor this check down to the Manager level.

Labels:
name: Copy / view semantics

Comments:
This would make `df.groupy(df['a'])` behave differently from `df.groupby(df['a'].copy())`, is that right? In general I'd like to move away from "in-axis" vs "out-axis" groupers behaving differently (ref: #49519), so I agree this is the right direction, but it might cause some odd edge cases to behave differently until we resolve those issues.

Still - I'm +0 here; checking equality gives some odd behaviors as well.
Yes those 2 behave different, but this pr establishes the same behavior as in the non-cow mode for this case. So that's actually a plus imo
Yes, I think this is a good solution for now (essentially the same as what was suggested in https://github.com/pandas-dev/pandas/pull/50730). And it indeed aligns with the behaviour we have for non-CoW.

Longer term, I agree with @rhshadrach that the current semantics are not really great. We should probably at some point move to not doing such check at all, at which point this discussion about identity vs equality also just doesn't matter anymore?

Yep this sounds sensible, but I think this needs a deprecation cycle?

I’ll move the check to the BlockManager then
Implemented it on the manager level
Thanks!
CI: Fix failing ci for arrow list test
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

cc @mroeschke @jbrockmendel 

I think this got in today

Labels:
name: CI
description: Continuous Integration
name: Arrow
description: pyarrow functionality

Comments:
circle ci is green, so merging to get ci running again
DOC Fix EX01 issues in docstrings
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Towards #37875

Labels:
name: Docs

Comments:
Thanks @DeaMariaLeon 
Thank you
PERF: Improve performance for pyarrow to_numpy with na and na_value
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

saw this in the engine pr, avoids a warning and improves performance:

```
pr
%timeit ser.to_numpy(dtype="int64", na_value=1.5)
1.67 ms ± 11.2 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

main
%timeit ser.to_numpy(dtype="int64", na_value=1.5)
/Users/patrick/PycharmProjects/pandas/pandas/core/arrays/arrow/array.py:893: RuntimeWarning: invalid value encountered in cast
  result = np.asarray(self._data, dtype=dtype)
2.42 ms ± 30.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Arrow
description: pyarrow functionality

Comments:
greenish