BrandonFors/pandas-github-issues huggingface dataset:
update offsets.pyx to fix #60647
State: closed

- [x ] closes #60647 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Sorry for not adding a test case for this. I'll set it up once I get home and can work on my PC—my current laptop is too slow to handle the full testing environment.

Also, I noticed the unit tests failed. Again, I will fix them once I get home.
Thanks for working on this fix for #60647!

This change successfully removes the code path containing the specific line (`holidays = holidays + calendar.holidays().tolist()`) identified in the issue and the comments.

However, there are a few points that probably deserve some attention:

**Removed Functionality:** This fix works by removing the `if-else` block. Was the functionality within that block (handling a non-numpy `calendar` object passed with `holidays`) intended or ever valid? Removing it is a potentially breaking change that should ideally be documented.

**Exception Type:** Raising `ApplyTypeError` here is a bit unusual, as it's typically for arithmetic operations returning `NotImplemented`. Would a `ValueError` or `TypeError` be more appropriate to signal invalid arguments during the setup phase?

Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
BUG: DataFrame.resample is changing thBUG: DataFrame.resample is changing the index type to MultiIndex when the dataframe is empty (pandas-dev#61174) * BUG: DataFramee index type to MultiIndex when the dataframe is empty (pandas-dev#61174) * BUG: DataFrame    solved
State: closed

- [x] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
I'm closing this PR because it deletes the entire pandas source.
BUG: DataFrame.resample is changing the index type to MultiIndex when the dataframe is empty
State: closed

BUG: DataFrame.resample is changing the index type to MultiIndex when the dataframe is empty

very easy bug to reproduce. the bug has been opened for 1 year.

```
# repro
import pandas as pd

df = pd.DataFrame(columns = ["a", "b"], index = pd.DatetimeIndex([]))
sampled = df.resample("8h").apply(pd.Series.mean)

print(df.index)
print(sampled.index)
```

output on pandas 1
```
DatetimeIndex([], dtype='datetime64[ns]', freq=None)
DatetimeIndex([], dtype='datetime64[ns]', freq='8H')
```

output on pandas 2 and main branch
```
DatetimeIndex([], dtype='datetime64[ns]', freq=None)
MultiIndex([], )
```


- [X] closes #55572 (Replace xxxx with the GitHub issue number)
- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [X] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: MultiIndex
name: Resample
description: resample method

Comments:
Thanks @morotti
BUG: Handle overlapping line and bar on the same plot
State: closed

- [x] closes #61161
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This code
```python
import pandas as pd
import matplotlib.pyplot as plt

index = pd.period_range('2023', periods=3, freq='Y')
df = pd.DataFrame({
    'col1': [10, 20, 30],
    'col2': [40, 25, 10]
}, index=index)

fig, ax = plt.subplots()
df['col1'].plot(kind='bar', ax=ax)
ax2 = ax.twinx()
df['col2'].plot(kind='line', ax=ax2, color = 'r')
plt.show()
```
from the linked issue gives this plot:
![image](https://github.com/user-attachments/assets/9732188c-5478-4e93-ae3c-ae18e77776f3)

And swapping the lines (line before bar)
```python
df['col2'].plot(kind='line', ax=ax2, color = 'r')
df['col1'].plot(kind='bar', ax=ax)
```
gives
![image](https://github.com/user-attachments/assets/35815445-dd3d-4a95-8c27-5a4dd07810be)

I believe the difference of results stems from the fact that the figure is always cropped to match the x-range of the last plot. We can certainly adjust the PR to make the first figure behave like the second one if desired.

Please provide feedback on the overall approach. Once confirmed, I'll expand and cover with tests.


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
@rhshadrach I covered the change with a test. I believe the PR is now ready for review and then merge.
Only a maintainer is allowed to merge a PR, right?
Thanks @MartinBraquet
DataFrame.dtypes.to_json OverflowError: Maximum recursion level reached
State: closed

- [x] closes #61170  (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: IO JSON
description: read_json, to_json, json_normalize
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
ENH: Accept no fields for groupby by
State: closed

- [ ] closes #61160 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
@udit710  Thanks for the Pull Request, I appreciate !
Thanks for the PR, but there's still discussion needed on the issue before proceeding with a PR so closing for now.
DOC: Add punctuation to basics.rst in User Guide
State: closed

Add punctuation to `basics.rst` in User Guide.

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @star1327p
WEB: Remove self (Marco Gorelli) from "active maintainers" list
State: closed

This is a bit overdue, but I haven't been active (barring minor PRs) for some time. This isn't anything personal against anyone, nor against pandas, I'm just recognising that my involvement has waned, and that unfortunately I don't have capacity for pandas maintenance

Thanks everyone for having involved me, I really appreciate it. A highlight for me was meeting many of you in Basel, and on the technical side a personal highlight was working on PDEP-4 and PDEP-6

There are some changes I've made which affect 3.0, and I'm happy to be pinged and to take responsibility for them, but I can do so even just as contributor, no need for me to be in the "active maintainers" list for that

All the best 🤗 

Labels:
name: Web
description: pandas website

Comments:
Thank you Marco for all the amazing work in pandas, it's been amazing working with you and having your conttibutions. And looking forward to continue working together in what makes sense, regardless of your availability to maintain pandas.

For the workgroup, @Dr-Irv is the expert for what I know, maybe it can advice if this group should simply be removed or what's the plan with the new governance. Or if the CI and web rendering doesn't break, maybe just leave it empty. Also ok to put me there to get this merged, and we see later if needed.

For this PR, can you also remove the quansight logo image please? It was noted to me in #61121, I also forgot it. Other than that, lgtm

> For the workgroup, @Dr-Irv is the expert for what I know, maybe it can advice if this group should simply be removed or what's the plan with the new governance. Or if the CI and web rendering doesn't break, maybe just leave it empty. Also ok to put me there to get this merged, and we see later if needed.
> 

It's on my list to update the various teams in a few places.

For that workgroup (communications), include the following people:

@WillAyd Will Ayd
@rhshadrach Richard Shadrach
@jorisvandenbossche Joris Van den Bossche




Thanks for all the great contributions to pandas, @MarcoGorelli! Happy to have you back at any time 😄  but I know you're working on great things in the ecosystem 🚀 
Thanks all, very kind and much appreciated!

(sorry I didn't get to removing the logo in time, thanks for getting this through)
Thanks for all your contributions - and pandas and everything else!
ENH: Support `Series[bool]` as indexer for `iloc.__getitem__`  
State: closed

- [x] closes #60994 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
Hi! The test on lines 742-743 calls `.iloc` with a boolean mask successfully:

```python
result = df.iloc[np.array([True] * len(mask), dtype=bool)]
tm.assert_frame_equal(result, df)
```

I noticed I should add another test with false values so I have done so.

```python
result2 = df.iloc[np.array([True, False, True, False, True], dtype=bool)]
tm.assert_frame_equal(result2, DataFrame({"a": [0, 2, 4]}, index=["A", "C", "E"]))
```
While reviewing this, I realized my tests address `.iloc` when passing `boolean` masks rather than `Series[bool]` as #60994 originally intended. I will be updating my tests accordingly. 

That being said the new tests improve coverage for `boolean` mask behavior for `.iloc`, which I think is helpful.
Thanks @arthurlw
Fix bug in `Series.describe` where the median is included any time the `percentiles` argument is not None
State: closed

- [x] closes #60550 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This PR aims at fixing the bug mentioned in the issue referenced above. Multiple PRs (#61024, #61023 , #60986 , #60557) have already attempted to resolve it, yet all of them got closed or stale for more than 2 weeks. So I allowed myself to apply here all the comments in the issue and previous PRs.

One last thing left hanging pertains to backward incompatibility. The only case in which the result will change is when `percentiles` is a list that does not contain 0.5. Before the PR, it adds 0.5 to the result; after the PR, it does not include it. 
If we add a warning message in such case, it would be considered as:
* useful for the people who read the median even though they didn't include it in the list of percentiles
* spamming / undesirable for the rest

So, what is pandas' policy regarding warning messages and its potentially large number of false alerts?


Labels:
name: Series
description: Series data structure

Comments:
@mroeschke Thanks for the feedback! I applied your comments. Lmk if there's anything else to update.
Thanks @MartinBraquet
Preserve Complex Data Types for to_csv
State: closed

This Pull Request solves the issue outlined in:
https://github.com/pandas-dev/pandas/issues/60895

Complex data types like numpy arrays can now be stored in csv format and later recovered. 
A new parameter, preserve_complex, is introduces, and when it is set to true in your to_csv function call, the complex data types will be preserved and can be recovered from the csv.

The way this works is by serializing Numpy arrays into JSON format for preserve_complex=True. To get them from the csv, we can set the same parameter in read_csv, and the original Numpy array will be returned.

Please refer to tests in scripts/tests/test_csv.py to see how this is used.

Please refer to the original issue for more information on the problem definition.



Labels:
name: Enhancement
name: IO CSV
description: read_csv, to_csv
name: Complex
description: Complex Numbers
name: Needs Discussion
description: Requires discussion from core team before further action

Comments:
@Jaspvr is there an existing issue that this PR addresses? If so, could you list it in the description? If not, please create an issue describing the bug or proposed enhancement so it can be reviewed by a team member.
> @Jaspvr is there an existing issue that this PR addresses? If so, could you list it in the description? If not, please create an issue describing the bug or proposed enhancement so it can be reviewed by a team member.


Hey, just updated the description. This is in relation to https://github.com/pandas-dev/pandas/issues/60895
This PR introduces a useful feature for preserving complex data types like NumPy arrays during CSV serialization/deserialization via the `preserve_complex` flag. The implementation integrates well with the existing I/O functions, and the new tests seem cover the basic functionality (I haven't run them).

Two minor points for consideration:

**Serialization Logic (`pandas/io/formats/csvs.py`):** The current logic checks only the *first* value in an object column to decide if it contains arrays/lists needing serialization. This might be fragile if the first row has NaN or a different type. Would checking the first *non-NaN* value or using `pd.api.types.infer_dtype` be more robust, balancing robustness vs. performance?
**Deserialization Logic (`pandas/io/parsers/readers.py`):** The `_restore_complex_arrays` function uses a heuristic (`startswith("[")` / `endswith("]")`) combined with checking `all()` non-null values. This seems safer than just checking brackets, but could still potentially misidentify columns if they contain a mix of valid JSON strings and other strings. Perhaps a `try-except json.JSONDecodeError` within the `apply` could be an alternative? (though likely slower if many non-JSON strings exist)
Thanks for the pull request, but it appears to have gone stale. Additionally it appears that there needs more discussion whether we want to support this type in exporting CSV. So going to close for now
BUG: Clip corr edge cases between -1.0 and 1.0
State: closed

Closes #61120 

Clips correlation coefficient in `DataFrame.corr()` between `-1.0` and `1.0`

Adds `test_corr_within_bounds` to ensure coefficient within bounds

- [x] closes #61120 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: cov/corr

Comments:
I do not think `nancorr_spearman` requires this change because it computes the divisor by first ranking the raw data rather than using the raw data itself like `nancorr`. This results in a stable value for the divisor. In the case of `y = [1.35951, 1.3595100000000007]`, the divisor equals 0.5 for `nancorr_spearman` but equals almost 0 for `nancorr`.
Added a whatnew note under "Notable Bug Fixes"
Thanks again @j-hendricks 
@j-hendricks I'm seeing different behavior, depending on whether missing values are present. Can you confirm whether the following behavior is expected?

At 83979d6a0c5a223bac2af8ef706c5ff8d432bcca

```python
In [50]: x = pd.DataFrame({"A": [1, 2, None, 4], "B": [2, 4, None, 9]})

In [51]: x.cov()
Out[51]:
     A    B
A  1.0  1.0
B  1.0  1.0

In [52]: x.dropna().cov()
Out[52]:
          A     B
A  2.333333   5.5
B  5.500000  13.0

In [53]: pd.__git_version__
Out[53]: '83979d6a0c5a223bac2af8ef706c5ff8d432bcca'
```

With 2.2.3:

```python
In [67]: x = pd.DataFrame({"A": [1, 2, None, 4], "B": [2, 4, None, 9]})

In [68]: x.cov()
Out[68]:
          A     B
A  2.333333   5.5
B  5.500000  13.0

In [69]: x.dropna().cov()
Out[69]:
          A     B
A  2.333333   5.5
B  5.500000  13.0

In [70]: pd.__version__
Out[70]: '2.2.3'
```

Depending on my digging in https://github.com/dask/dask/pull/11857, there might be some more differences, but this seemed to be the largest.
It appears this clipping should have only happened if `cov=False`. @j-hendricks do you mind opening a follow up PR with this fix?
ENH: Adds `dict` & `**dict` support for `pd.set_option` [#61093]
State: closed

- [x] closes #61093
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


@rhshadrach & @yasirroni I have Two Approaches one adds support for `pd.set_option(options)` and other adds support for both `pd.set_option(options) & pd.set_option(**options)`


Labels:

Comments:
{@rhshadrach @yasirroni} Finally All tests Passed 🎊

OK Lets, Talk About Both approaches i think we can go with 1st approach for now and if required in future we can also go with the second approach 

Let's have a Brief Discussion on this Topic...
Once perfect approach chosen by members then i will move to write test case for that and update the changelog 
Thanks for the PR, but it appears https://github.com/pandas-dev/pandas/pull/61151 is further along with discussion and development so closing this PR in favor of consolidating there