BrandonFors/pandas-github-issues huggingface dataset:
Ensure HDFStore read gives column-major data with CoW
State: closed

Fixing a failing test that turned up in https://github.com/pandas-dev/pandas/pull/55732. 
This PR specifically fixes `pandas/tests/io/pytables/test_store.py::test_hdfstore_strides`, which tests that the data coming back from HDF are column-major (the "idea" of the test was to ensure the layout is _preserved_, i.e. when starting with a DataFrame that is column-major, you should get back one with the same layout. But in practice we don't do that, we just simply always return column-major data because of `concat` making a copy by default (see https://github.com/pandas-dev/pandas/issues/22073#issuecomment-1383843005), and we only test the column-major case in the test)

We have ran into this issue before: in pandas/tests/io/pytables/test_store.py::test_hdfstore_strides, @phofl initially just skipped the test for CoW. But then in the final version of that PR, we removed that skip in favor of adding a `copy=True` inside the `concat` call in HDFStore.read. 
However, later, we then changed `concat` to ignore the `copy` keyword alltogether when CoW is enabled, even when specifically passing `copy=True` (done in https://github.com/pandas-dev/pandas/pull/51464). Because of that, this test is failing again (the reason this wasn't caught in our CI, s because the specific test was in the meantime moved to "single_cpu", which we don't run on the CoW CI builds)

This PR implements the option to manually copy inside HDFStore.read when CoW is enabled to achieve the same end result, but two other options would be:

1) Copy manually in HDFStore.read to ensure column-major data (i.e. current version of this PR)
2) Don't copy, and live with HDFStore returning row-major data (and update the test to reflect that for CoW). This gives a faster read, but lower performance afterwards (probably not the best option)
3) Change `pd.concat(..)` again to _do_ honor the `copy=True` option, even in the case of CoW



Labels:
name: IO HDF5
description: read_hdf, HDFStore
name: Copy / view semantics

Comments:
Thanks @jorisvandenbossche
BUG: fix nanmedian for CoW without bottleneck
State: closed

Fixing a failing test that turned up in https://github.com/pandas-dev/pandas/pull/55732. 
This PR specifically fixes `pandas/tests/frame/test_reductions.py::test_reduction_axis_none_returns_scalar[float64-True-median]`, i.e. calling `df.median(axis=None)`, for builds that don't have bottleneck installed.

If the underlying data of the DataFrame or Series is float64 data, the `values` inside `nanops.nanmedian(..)` will be the original values from the DataFrame, and with CoW set to be read-only (for other dtypes, we cast to float64 inside the function, and so that will always give a copy). 

And then we do `values[mask] = np.nan`, which raises an error if values is read-only. In practice, this didn't give any problems with mutating the original DataFrame, because if you start with float64 values, the `mask` always corresponds with NaN values already (so we were actually mutating the calling dataframe, just without any effect).

Labels:
name: Bug
name: Copy / view semantics
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
thx @jorisvandenbossche
POC/ENH: infer resolution in array_to_datetime
State: closed

xref #55564 implements the relevant logic in array_to_datetime, does not change any user-facing behavior yet.

Have not yet done any profiling, but we have to expect a slowdown.

Labels:
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
@mroeschke ok here?  this goes a long way toward #55901
Thanks @jbrockmendel
REF: misplaced formatting tests 
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks @jbrockmendel
BUG: fix online ewma with CoW
State: closed

Fixing some failing tests that turned up in https://github.com/pandas-dev/pandas/pull/55732. This PR specifically fixes `pandas/tests/window/test_online.py::TestEWM::test_online_vs_non_online_mean` for Copy-on-Write enabled.

The numba function takes the underlying frame's array of values, and then writes into those values:

https://github.com/pandas-dev/pandas/blob/c36e302c7b30bc7945119a77f255d0302307847e/pandas/core/window/online.py#L80

(this `weighted_avg` is a view in the passed values, from `weighted_avg = values[0]`)

That currently doesn't cause a bug, because before getting the values from the frame and passing it to the numba func, we cast to float64:

https://github.com/pandas-dev/pandas/blob/c36e302c7b30bc7945119a77f255d0302307847e/pandas/core/window/ewm.py#L1071

This `astype()` call currently always returns a copy by default (even if you already have all float64 columns), but with CoW enabled, this avoids making a copy. Currently, with CoW enabled, the numba func fails because of the numpy array being marked as read-only. But if we wouldn't manually copy the data and just ignore this read-only flag, we would be mutating the calling dataframe (so marking as read-only prevented a new bug here!).

Labels:
name: Bug
name: Window
description: rolling, ewma, expanding
name: Copy / view semantics
name: numba
description: numba-accelerated operations

Comments:
Thanks @jorisvandenbossche
DEPR: errors='ignore'
State: closed

- [x] closes #54467 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @jbrockmendel
[NOT FOR MERGE] Test enabling Copy-on-Write by default
State: closed

xref https://github.com/pandas-dev/pandas/issues/48998

We already test with CoW enabled in a few CI builds, but let's see what this gives when enabling it by default for the full test matrix.

Labels:
name: Copy / view semantics
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
I think with https://github.com/pandas-dev/pandas/pull/56633 we have effectively started down this path so closing
BUG: Comment in ODS-file gets included in string cells
State: closed

- [x] closes #55200
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This PR fixes only annotations in string cell. For time cell see #55324.

Labels:
name: IO Excel
description: read_excel, to_excel

Comments:
Thanks @dimastbk
TST: add test for mixed dtypes of df col index #47382
State: closed

- [x] closes #47382
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Modified the code snippet reproducing the error into a test under tests/frame/test_arithmetic.py.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks @mliu08
DEPR: groupby.fillna
State: closed

- [x] closes #55718
- [x] closes #48792 
- [x] closes #48395
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Groupby
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Deprecate
description: Functionality to remove in pandas
name: Transformations
description: e.g. cumsum, diff, rank

Comments:
Thanks @rhshadrach
BUG: IntervalIndex.from_arrays with mismatched datetimelike resos
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Surfaced while implementing #55564

Labels:
name: Interval
description: Interval data type
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
Thanks @jbrockmendel
BUG: Index.view with non-nano
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Index
description: Related to the Index class or subclasses
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
Thanks @jbrockmendel
REF: Avoid np.can_cast for scalar inference for NEP 50
State: closed

Looks like using `np.can_cast` for scalar casting inference is deprecated. We should probably use our own scalar casting inference routines instead

Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
Looks like the test failures are timeouts hopefully unrelated to this change. Going to backport this change as this could be useful for 2.1.x
Thank you! When do you plan to push out a new dev wheel with this patch? 🙏 
Wheels uploads run nightly at 3:27 UTC, so hopefully the dev wheels should be available tomorrow
I can confirm the dev wheel is up and it works now. Thanks again!
Backport PR #55619 on branch 2.1.x (BUG: Groupby not keeping string dtype for empty objects)
State: closed

* BUG: Groupby not keeping string dtype for empty objects

* Fix

Backports PR #55619


Labels:
name: Groupby
name: Strings
description: String extension data type and string data

Comments:
Just needed to move a test for this one.
Ack, looks like there's a segmentation fault here.
(not sure if it's spurious or not, it happens after the tests finish?)

see https://github.com/pandas-dev/pandas/actions/runs/6654698565/job/18083267496?pr=55705#step:9:467.

@phofl I'm gonna bump this off the milestone then.
Yeah no worries 
Nevermind about the segfault, it seems to be unrelated to this PR (I saw it happen on the 2.1.x branch too, and I made a comment about it on the release issue).
https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=812339&view=logs&j=385a3385-ae8b-54d1-9abe-3c614f74cb73&t=d0913101-e169-5874-b37d-533167693e73&l=1989.

I'll send a PR to move the release note.
Oops, looks like I accidentally hit close on this for some reason.

(I think I meant to hit merge).
Merging, 

I need to move the release note now.
BUG: there is no year 0
State: closed

- [ ] closes #55462
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:

Comments:
Thanks for the PR, but as the issue stated there needs to be more discussion on the correct behavior before proceeding here so closing in the meantime. Happy to have you efforts on other issues labeled `good first issue`
ENH: Support for list of formats in pd.to_datetime() (pandas-dev#55226)
State: closed

Trying to convert format list for each data list element.

The idea is to make the format argument more flexible to accept a list, in order to specify the format for each date in a list.


Labels:
name: Stale

Comments:
Hello @MarcoGorelli we need a review for this code.
Thanks for attention
Hello @MarcoGorelli , I reverted a commit with many graph images that had nothing to do with the issue. Can you review for us? Thank you for your patience.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
This fix allows a DataFrame to retain key order from a dictionary 
State: closed

This fix allows a DataFrame to retain key order from a dictionary with only one column instead of sorting them alphabetically.

Note: I had to change the order for the rename tests in `test_rename.py` but the order there seemed trivial.  If I'm wrong in my assumption please let me know. I also had to change the index order of the control df in `indexing/multiindex/test_setitem.py` but again this seemed trivial/innocuous change in this circumstance.

- [x] closes #55683
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: DataFrame
description: DataFrame data structure

Comments:
Not sure why this Closed.
Thanks @paulreece
BUG/DOC: Update dtype_backend documentation and API
State: closed

- [x] closes #55647 
- [x]  Corrects [https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html](documentation) incorrectly stating that the default `dtype_backend` was `numpy_nullable`
- [x] Adds `numpy` as an option for `dtype_backend`, equivalent to the current default (`lib.no_default`). Previously it was impossible to achieve the default behavior by passing in an explicit parameter, except by passing in `pandas._libs.no_default`. Passing the string "numpy" is now equivalent to passing `pandas._libs.no_default`.


Labels:
name: Stale

Comments:
It looks like tests specifically disallow users from passing `dtype_backend=numpy`. I'm curious why, and if it is intended that there be no easy way of explicitly passing in a parameter equivalent to the default. If that is the case, how should the default be labelled in the documentation? @lithomas1 
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
REGR: fix read_parquet with column of large strings (avoid overflow from concat)
State: closed

- [x] closes #55606
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: IO Parquet
description: parquet, feather

Comments:
thanks @jorisvandenbossche.
REGR: fix roundtripping datetimes with sqlite type detection
State: closed

- [x] closes #55554
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

https://github.com/pandas-dev/pandas/pull/54985 was already a related regression fix in 2.1.1, but that only fixed the adapter (conversion to sqlite), not the converter (conversion back from sqlite to python). This converter only gets used when explicitly asking sqlite to detect dtypes.

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
The changes LGTM assuming this is green.
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.1.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 aeb3644567a10296cc09aed03bf7e0a3780e67ee
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #55690: REGR: fix roundtripping datetimes with sqlite type detection'
```

4. Push to a named branch:

```
git push YOURFORK 2.1.x:auto-backport-of-pr-55690-on-2.1.x
```

5. Create a PR against branch 2.1.x, I would have named this PR:

> "Backport PR #55690 on branch 2.1.x (REGR: fix roundtripping datetimes with sqlite type detection)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Manual backport -> https://github.com/pandas-dev/pandas/pull/55702
Update doc for parse_dates arg to read_csv
State: closed

Added note that values are joined with a space before parsing for the combined column cases.

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @mvirts
DOC: Update title caps validation script to step through directories
State: closed

Add words that should remain capitalised:
``` 
"RangeIndex",
"SparseArray",
"SparseDtype",
"HTTP",
"XML"
```

Before:
when specifying a directory in the form: `doc/source` for the script, it would throw an error, because it was looking for only files

After:
can now enter:
`script/validate_title_capitalization.py doc/source` and it will traverse all the directories below it. 

- [X ] related to  #32550 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ X] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: CI
description: Continuous Integration

Comments:
Regarding the checks: 3 are being cancelled. The last one, ci/circleci: test-arm, I don't know why that's failing. I updated my local  repo to latest from upstream/main and ran the same test, which passed
@skregas thanks for the work here, do you still want to continue with this PR? There is the comment above that you may want to check if that's the case
Closing as stale, but thanks for the work @skregas, and if you eventually want to address the comments, please let me know and I can reopen the PR.
REF: organize Timedelta, Index tests
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Timedelta
description: Timedelta data type
name: Index
description: Related to the Index class or subclasses

Comments:
Thanks @jbrockmendel