BrandonFors/pandas-github-issues huggingface dataset:
CI: Test Python 3.12
State: closed

- [ ] xref #53665  (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Build
description: Library building on various platforms
name: CI
description: Continuous Integration

Comments:
For those interested, we have
```208 failed, 172066 passed, 18996 skipped, 982 xfailed, 84 xpassed, 56 warnings in 2448.38s (0:40:48) =```

Not bad, given most are parametrized, but definitely much more than last year 😭 .
I pushed up a fix to use `timezone.utc`. I think `datetime.UTC` is > PY311
Thanks for updating this, and sorry for losing track of this PR.

I have the Docker image for Python 3.12 set up on my laptop, so I'll try to get some debugging in tomorrow for the Index failures, unless you beat me to it again :).
Looks like the major change is 

> [slice](https://docs.python.org/3.12/library/functions.html#slice) objects are now hashable, allowing them to be used as  dict keys and set items. (Contributed by Will Bradshaw, Furkan Onder, and Raymond Hettinger in [gh-101264](https://github.com/python/cpython/issues/101264).)
Down to 4 tests
```
FAILED pandas/tests/computation/test_eval.py::TestEval::test_true_false_logic - DeprecationWarning: Bitwise inversion '~' on bool is deprecated. This returns the bitwise inversion of the underlying int object and is usually not what you expect from negating a bool. Use the 'not' operator for boolean negation or ~int(x) if you really want the bitwise inversion of the underlying int.
FAILED pandas/tests/frame/indexing/test_where.py::TestDataFrameIndexingWhere::test_where_invalid - DeprecationWarning: Bitwise inversion '~' on bool is deprecated. This returns the bitwise inversion of the underlying int object and is usually not what you expect from negating a bool. Use the 'not' operator for boolean negation or ~int(x) if you really want the bitwise inversion of the underlying int.
FAILED pandas/tests/series/indexing/test_getitem.py::test_getitem_generator - AssertionError: Series are different

Series length are different
[left]:  0, Index([], dtype='object')
[right]: 14, Index(['1bZFplgxJt', 'dgf6MOzmcc', 'nXA7LxB4pD', '7xialahw4M', 'eHSMzF5vTv',
       'ohno0T7Zu6', 'sK2uiqyfCh', 'ccmUonUyaI', 'KNYdKnS7IO', 'jQ9UjUVpex',
       'NGazj6cYBr', 'XHqmZevyyF', 'SQXHWlGTY5', 'ZmyLvG2mLO'],
      dtype='object')
FAILED pandas/tests/tseries/offsets/test_year.py::test_add_out_of_pydatetime_range
```

I think I regressed one of the indexing tests in my fixes.
This should be ready now.

cc @jbrockmendel @phofl for indexing changes 

Not sure who to ping for eval/sql changes.
ENH: Include `df.attrs` metadata in `to_csv` output
State: closed

- [ ] closes #53577
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

---

This new functionality writes DataFrame metadata stored in `df.attrs` as header/comment lines at the top of a CSV file.

1. The `to_csv()` method had a `comment` param added to its call, matching the `read_csv()` method signature
2. If this param is set all key/values will be written to the beginning of the CSV file in the following format `[comment][key]:[value]`


Labels:
name: IO CSV
description: read_csv, to_csv
name: metadata
description: _metadata, .attrs

Comments:
Given other discussions around df.attrs #52166 this implementation does not use or depend on `__finalize__` at all.
Closing to cleanup my forked branch. Will re-open as a new PR
BUG: resampling empty series loses time zone from dtype
State: closed

- [x] Closes #53664
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests)
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit)
- [x] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst` file


Labels:
name: Timezones
description: Timezone data dtype
name: Resample
description: resample method

Comments:
Thanks @Charlie-XIAO
Backport PR #53727 on branch 2.0.x (CI: fix pytest scripts)
State: closed

backports #53727

(cherry picked from commit 3fb757beb6ca6bb30007ba1db130cc83f27f9363)

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
~oops - trying again~

EDIT: done
CI: fix numba typing
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: CI
description: Continuous Integration
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
you're right, thanks - just gotta `# type: ignore` it
updated to get the other fixes in
thanks for review - merging then to get CI green 
Can't Dooooo.... It seem like this is already backported (commit is empty).I won't do anything. MrMeeseeks out.
> Can't Dooooo.... It seem like this is already backported (commit is empty).I won't do anything. MrMeeseeks out.

right, Matt had already made this change on 2.0.x
ENH: EA.fillna copy=True
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

cc @phofl for comments on the CoW handling in Block code
cc @jorisvandenbossche for thoughts on deprecation path for 3rd party EAs

Edit: also worth discussing the "author's discretion" on respecting copy=False.

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Updated to ignore "copy" in the base class fillna method in ffill/bfill cases.

Added a warning+test for 3rd party EAs that dont yet have the copy keyword in the signature.
i think comments have been addressed
Sorry I thought that we had already merged this. Will merge when CI is green
thx @jbrockmendel
CI: fix `pytest scripts`
State: closed

seems that
```python
def pytest_addoption(parser):
    parser.addoption(
        "--strict-data-files",
        action="store_true",
        help="Unused. For compat with setup.cfg.",
    )
```
isn't needed anymore, and actually errors now:
```python
ValueError: option names {'--strict-data-files'} already added
```

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.0.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 3fb757beb6ca6bb30007ba1db130cc83f27f9363
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #53727: CI: fix `pytest scripts`'
```

4. Push to a named branch:

```
git push YOURFORK 2.0.x:auto-backport-of-pr-53727-on-2.0.x
```

5. Create a PR against branch 2.0.x, I would have named this PR:

> "Backport PR #53727 on branch 2.0.x (CI: fix `pytest scripts`)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
thx @MarcoGorelli 

I guess we should backport?
yeah probs
Why did pytest needed to be bumped? Technically we call out bumping these dev dependencies in the minor/major release whatsnews
thanks @mroeschke , I'll make a PR to update this in the whatsnew

the reason is that the
```
def pytest_addoption(parser):
    parser.addoption(
        "--strict-data-files",
        action="store_true",
        help="Unused. For compat with setup.cfg.",
    )
```
options in `scripts/conftest.py` fail with a more modern pytest, so I figured it was easiest to just remove that and bump pytest
I think I am getting a related error in some PRs: https://github.com/pandas-dev/pandas/actions/runs/5383684555/jobs/9770736568?pr=53828

```
ERROR: usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: --strict-data-files
  inifile: /home/runner/work/pandas/pandas/pyproject.toml
  rootdir: /home/runner/work/pandas/panda
```

Do you know why this is happening?

Also it appears `minversion = "7.0"` is still specified in pyproject.toml
the logs show that that job is using:
```
pytest                         7.4.0 
```

My guess is that the issue might be in the `pytest-localserver` plugin?
CI: ignore experimental warnings from numba
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Tried to catch the warning within `online_ewma` but then I get
```python
numba.core.errors.CompilerError: Failed in nopython mode pipeline (step: Handle with contexts)
Unsupported context manager in use

File "pandas/core/window/online.py", line 58:
    def online_ewma(
        <source elided>
        nobs = (~np.isnan(weighted_avg)).astype(np.int64)
        with warnings.catch_warnings():
```

Labels:
name: Docs
name: CI
description: Continuous Integration

Comments:
Yeah, nopython mode do be like that.

Apparently the warning is a numba bug to be fixed in 0.57.1
https://github.com/numba/numba/issues/8936.
thanks for the quick review

this fixes the docs build, so merging
Can't Dooooo.... It seem like this is already backported (commit is empty).I won't do anything. MrMeeseeks out.
> Can't Dooooo.... It seem like this is already backported (commit is empty).I won't do anything. MrMeeseeks out.

u wot m8
ah looks like @mroeschke had already made the same fix, but on the 2.0.x branch https://github.com/pandas-dev/pandas/pull/53372/files

all good then
TST: add test for reindexing rows with matching index uses shallow copy with CoW
State: closed

xref https://github.com/pandas-dev/pandas/issues/49473

This was already working, but I think it was not yet explicitly tested (at least not in the specific `copy_view` tests). And this addresses a TODO note in https://github.com/pandas-dev/pandas/issues/49473

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Copy / view semantics

Comments:
thx @jorisvandenbossche
API / CoW: shallow copy of DataFrame/Series (.copy(deep=False)) also returns shallow copy of the index/columns
State: closed

Broken off a subset of https://github.com/pandas-dev/pandas/pull/53699, only focusing on the behaviour of `df.copy(deep=False)` (i.e. a "shallow copy"). And address the `copy(deep=False)` part of https://github.com/pandas-dev/pandas/issues/53721.

Currently, a shallow copy of a DataFrame/Series only creates a new DataFrame/Series object, but still shares the same data (with tracking references, to ensure copy-on-write) and Index objects. 
Because Index objects have some mutable aspects (not their values, but for example there `name`), that means that sharing an identical object in the resulting shallow copy can propagate mutations to those mutable parts of the index.

See https://github.com/pandas-dev/pandas/issues/53721 for a longer description of the context.

This PR just updates the DataFrame/Series `copy()` method to also take a shallow copy of the index/columns Index object, in case of `deep=False`.

And based on the tests I had to update to get this passing, it illustrates some of the consequential changes in other places because of changing the shallow copy.


- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: API Design
name: Copy / view semantics

Comments:
Thanks could use a whatsnew note in 2.1
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks @jorisvandenbossche
BUG: Aggregation on arrow array return same type
State: closed

- [ ] closes #53030  (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Groupby
name: Apply
description: Apply, Aggregate, Transform, Map
name: pyarrow dtype retention
description: op with pyarrow dtype -> expect pyarrow result

Comments:
would #53089 address this?  in general we want to get logic _out_ of maybe_cast_pointwise_result
> would #53089 address this? in general we want to get logic _out_ of maybe_cast_pointwise_result

@jbrockmendel , I have tried pr53089. It can't fix issue 53030. But I can try to work on pr53089 to fix issue 53030. 
@jbrockmendel , I have done some debug, found something about convert_dtypes function.
The convert_dtypes function's first parameter input_array could be np.ndarray with object dtype. Then the line:
`
inferred_dtype = lib.infer_dtype(input_array)
`
will be executed. Since lib.infer_dtype returns a str, then inferred_dtype also is str. So following lines will be executed:
```
if isinstance(inferred_dtype, str):
    # If we couldn't do anything else, then we retain the dtype
    inferred_dtype = input_array.dtype
```
At this time, inferred_dtype is object dtype.
In the 'pyarrow' backend codes part, all if condition failed. So this line is executed:
`
base_dtype = inferred_dtype
`
base_dtype is object dtype also. Then after `pa_type = to_pyarrow_type(base_dtype)`, pa_type is None since base_dtype is object.
So in this situation, convert_dtypes will return object dtype as long as input_array is np.ndarray with object dtype.
what if you do infer_objects() followed by convert_dtypes?
I have tried using infer_objects with following codes:
```
            if result.dtype == object:
                df = pandas.DataFrame(result)
                result = df.infer_objects().values
            pyarrow_type = convert_dtypes(result, dtype_backend="pyarrow")
```
It doesn't work. The dtype of result still can be object after infer_objects.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
CI: Adjust tests for release of numpy 1.25
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: CI
description: Continuous Integration

Comments:
Gonna merge since this at least fixes the unit tests.

Docbuild and Code Checks will need fixing separately.
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.0.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 43d12a23484f9da3509eb4dda991070bfe92b281
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #53715: CI: Adjust tests for release of numpy 1.25'
```

4. Push to a named branch:

```
git push YOURFORK 2.0.x:auto-backport-of-pr-53715-on-2.0.x
```

5. Create a PR against branch 2.0.x, I would have named this PR:

> "Backport PR #53715 on branch 2.0.x (CI: Adjust tests for release of numpy 1.25)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
DEPR: Deprecate DataFrame.last and Series.last
State: closed

- [X] closes #53692 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [X] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
@MarcoGorelli Is something going on with the CI environment? The tests that are failing for this PR are not failing locally for me when I run pytest. 

The same tests also started failing in #53439 this morning after I updated my entry in whatsnew v2.1.0. None of the failing tests seem to have anything to do with whats being changed in the PR hmmmm
@MarcoGorelli The Doc build test failure seems to be unrelated

The docstring test is failing because the warning message that I imeplemented is executing. Since this method is being deprecated what would be the recommended solution in this instance? 

Should I add an additional note in the docstring to  create a mask and filter using `.loc` instead of `last` and then add code that shows how to do that? I don't believe this was handled in the PR for deprecating [series.first and DataFrame.first](https://github.com/pandas-dev/pandas/pull/53419/files#diff-1a2e3df0db7dd8bddc2ec4bff9de8a7a55e328e6c32e2cecde761dc9549fcd46)

thanks @rmhowe425 - does it work to `doctest: skip` the failing doctest?
@MarcoGorelli Doc tests passing now. Ready for review. :smile: 

The two failing unit tests still don't seem to be related to the PR.

Just out of curiosity, when updating doc strings for methods that will be deprecated, is the correct approach to skip doc tests for those methods?
BUG: period_range gives incorrect output 
State: closed

- [x] closes #53687
- [x] [Tests added and passed]
- [x] All [code checks passed]
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file


Labels:
name: Period
description: Period data type

Comments:
I couldn't find any test case for `period_range()`, so where should the test case go in?
> I couldn't find any test case for `period_range()`, so which should the test case go in?

`pandas/tests/indexes/period/test_constructors.py`
done.
Thanks @jayendra-patil33
CoW: Return read-only array in Index.values
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Index
description: Related to the Index class or subclasses
name: Copy / view semantics

Comments:
thx for noticing, rebased