BrandonFors/pandas-github-issues huggingface dataset:
BUG: fixes Arrow Dataframes/Series producing a Numpy object result
State: closed

- [x] closes #53979 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

## Notes: 
`.dot()` 's returned object dtype is now the result of  `dtypes.cast.find_common_type()`.

It's worth noting the new object's dtype's is of a higher precision but this seems line up what's expected: https://arrow.apache.org/docs/python/pandas.html#pandas-arrow-conversion



Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Arrow
description: pyarrow functionality

Comments:
Hey @phofl do you know who I should tag for a review on this?  
> Could you add a test for masked types as well? I suspect they failed before as well

Sure! I made an assumption about what you meant by the masked types `Int8, Int16, Int32...`. 

After I updated the tests I noticed another potential edge case that when calling `dot` with two different masked dtypes, for example `Float64` and `float[pyarrow]`. I get return type Object

How would we want to handle this case? 
> After I updated the tests I noticed another potential edge case that when calling dot with two different masked dtypes, for example Float64 and float[pyarrow]. I get return type Object

Let's ignore this case for now
Thanks @mcgeestocks
BUG: Parameter col_space of to_html method not working with multi-level columns 
State: closed

BUG: Parameter col_space of to_html method not working with multi-level columns 

- [X] closes #53885
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
Just needs a whatsnew note in 2.1.0.rst
> Just needs a whatsnew note in 2.1.0.rst

done, have updated the whatsnew
Thanks @dannyi96
DEPR: passing mixed offsets with utc=False into to_datetime
State: closed

xref #50887

parsing datetimes with mixed time zones will raise a warning if `utc=False`. We advice users to use `utc=True` to silence this warning. 

Labels:
name: Datetime
description: Datetime data dtype
name: Deprecate
description: Functionality to remove in pandas

Comments:
@MarcoGorelli, could you please take a look at this pr?
Just xrefing https://github.com/pandas-dev/pandas/issues/54194 as an example of a user wanting to maintain mixed timezones in a different operation
thanks for the reference

this would still be possible, just not via `to_datetime`

still, I'm curious to ask what their use-case is
nice

I'd have thought the warning would show up in at least some `read_csv` cases - is there no test case for `read_csv` where the input data has mixed offsets? If not, could we add a test for that please (either here or in a separate PR), so we can check that no warning is raised there (as users wouldn't be able to silence it)?
> I'd have thought the warning would show up in at least some `read_csv` cases - is there no test case for `read_csv` where the input data has mixed offsets? If not, could we add a test for that please (either here or in a separate PR), so we can check that no warning is raised there (as users wouldn't be able to silence it)?

Thank you for the comment. Seems like there was no test for `read_csv`  with mixed offsets. I made a separate pr for this.
Sure, thanks - I don't want to force this onto the project, I really want us all to be happy with what we're doing

Suggestion: we get this into 2.1, and then if there's complaints or issues we hadn't foreseen (or use cases for which the workaround suggested isn't enough), then we revert for 2.2 / 3.0? Getting it in for 2.1 would give more time to hear people's potential complaints. Happy to admit I've been wrong about this, and to revert course, if that happens
Great work @natmokval 

I do like that this will not catch users by surprise with `object` dtype from `to_datetime`. Yeah let's see if we get feedback from users in 2.1
> Great work @natmokval
> 
> I do like that this will not catch users by surprise with `object` dtype from `to_datetime`. Yeah let's see if we get feedback from users in 2.1

thank you. I hope users will find this change reasonable.
Thank you @MarcoGorelli for helping me with this PR.
Why are we filtering warnings internally here? My understanding is that this will raise in the future, so we should give users the time to address things

cc @natmokval @MarcoGorelli 
it's in a place where the user can't do anything about it, and where in the future it just won't get converted (the input will be preserved) as we'll no longer end up with an `object` Series of `Timestamp`s
CLN: Remove unnecessary pa version check
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Clean
name: Arrow
description: pyarrow functionality

Comments:
Thanks @phofl
API/CoW: Return copies for head and tail
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

this is something we talked about to avoid messing up references in Jupyter environments (xref https://github.com/pandas-dev/pandas/issues/48998)

Labels:
name: Copy / view semantics

Comments:
I would not add a keyword, I am not convinced that there is a proper use-case that would require such a keyword
DOC: Added to 10mins guide
State: closed

- [ ] closes #47282 
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Docs

Comments:
Adding as reviewers: @datapythonista @phofl @mroeschke 
Thanks @rsm-23
CLN: is_mixed_type, is_homogeneous_type
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Clean

Comments:
Thanks @jbrockmendel
ENH: Use explicit methods instead of regex pattern in arrow strings
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Strings
description: String extension data type and string data
name: Arrow
description: pyarrow functionality

Comments:
green now
Thanks @phofl
Correct check when slicing non-monotonic datetime indexes
State: closed

The intention of #37819 was to deprecate (removed in #49607) the special case behaviour of non-monotonic datetime indexes, so that if either slice bound is not in the index, a KeyError is raised.

However, the check only fired correctly for the case where the lower bound was not in the index and either the upper bound was None or it was _also_ not in the index.

Correct the logic here and adapt the one test that exercises this behaviour.

Closes #53983.

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- ~Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.~
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Datetime
description: Datetime data dtype
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
cc @phofl.

This _might_ need to go through a deprecation cycle since although the intention was to warn for this behaviour through 1.5 and remove in 2, users who only had upper slice bounds that were missing would not have seen that warning.
Any thoughts here?
> I think this would be okay to note as a "bug fix" in 2.1 since the original behavior was deprecated but not thoroughly enforced.

Thanks, added a bugfix entry.
Awesome, thanks @wence-
REF: implement EA.pad_or_backfill
State: closed

- [x] closes #53621 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Sits on top of #53989.

The deprecation is a little bit awkward.  If a 3rd party EA has implement fillna but not pad_or_backfill, we warn and call that.  DecimalArray tests that deprecation.

The "method" paths for our existing EA.fillna methods could be ripped out as they are no longer used.  I left them untouched for now.

In a follow-up, can de-duplicate Block.pad_or_backfill with EABlock.pad_or_backfill.

Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
> /home/runner/work/pandas/pandas/pandas/core/arrays/base.py:docstring of pandas.core.arrays.base.ExtensionArray:119: WARNING: autosummary: stub file not found 'pandas.api.extensions.ExtensionArray.pad_or_backfill'. Check your autosummary_generate setting.

@mroeschke help?  i dont understand what this is asking for
@MarcoGorelli any idea whats going on with the docbuild?  i have a premonition its going to be really obvious as soon as someone points it out to me
My guess is that you need to add this new method to doc/source/reference/extensions.rst ?
That solved it, thanks
Thanks @jbrockmendel
CLN: tests.groupby.test_any_all
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: Clean

Comments:
thx @rhshadrach
TST: add test to check dtype after replacing values in categorical Series inplace
State: closed

- [x] closes #53358

- added a test in order to check if the `dtype` is being updated, when we replace values in a categorical Series with `inplace=True`.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
@mroeschke, I added a test to check if the `dtype` is being updated, when we replace values in a categorical Series `inplace`. Could you please take a look at my pr?
Thanks @natmokval
BUG: ignoring sort in DTA.factorize
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
thx @jbrockmendel
CLN: remove unreachable, unnecessary axis kwd
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Clean

Comments:
thx @jbrockmendel
BUG: missing fstring
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
thx @jbrockmendel
DOC: Clean up CSV sniffing and chunking examples
State: closed

Sniffing:
- Had an unused "to_csv" call

Chunking:
- Used an unnecessary "sep"
- Had two no-op statements, which seem like they were meant to be printed

Both
- Put the index as a column unnecessarily

---

- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Labels:
name: Docs

Comments:
thx @wjandrea
REF: Separate groupby, rolling, and window agg/apply list/dict-like
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Should enable work on #53839.

Labels:
name: Refactor
description: Internal refactoring of code
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Just two suggestion about explaining the return value types for `compute_list_like` and `compute_list_like`, because return value types can't be expressed in the type system and IMO it's not always super clear how we can know the return value types in `result_data` from the input data.

Else looks good to me.
There's a conflict that needs to be resolved, else looks good.
Thanks, @rhshadrach.
WEB/DOC: use Plausible for analytics (using Scientific Python server)
State: closed

This proposes to switch from our current usage of Google Analytics to Plausible.

(currently it fully switches, but could also first add it in addition so we have both for some time)

Plausible is a privacy-friendly and open source alternative for Google Analytics (https://plausible.io/). 
The server we are using here is a self-hosted one by the Scientific Python team (using one from plausible.io costs some money). But the disclaimer I got from @stefanv is that it's a small server without professional sysadmin support and is currently not automatically backed up (but we could set up a cron job to query the data and dump them somewhere). I think, given that this is also being used by other projects (numpy, scipy, scikit-learn, ..), that this is fine, and if there come up problems, we will be able to look for solutions together.

The main advantage is that privacy-friendly (not collecting data for a big corporation ..). I think our current usage of Google Analytics is actually not correct (I assume we should have a pop-up to ask for permission). 
The consequence of that is that Plausible cannot track "unique visitors" over time (only per day), for example per month (https://plausible.io/data-policy#how-we-count-unique-users-without-cookies).

Labels:
name: Web
description: pandas website
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
DOC: Fixing EX01 - Added examples
State: closed

- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Towards https://github.com/pandas-dev/pandas/issues/37875

I could not make the html of each method, so I'm not sure if everything looks well.
I'll use `/preview` here. 

Labels:
name: Docs

Comments:
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/53982/
Nice! Looks like there's just a merge conflict to resolve
Test CoW for multiple Python versions
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Copy / view semantics

Comments:
We could also run a small subset on every build (like adding a step with `PANDAS_COPY_ON_WRITE=1 pytest pandas/tests/copy_view`). That avoids the build setup/install time for those additional builds
Yep, I'd add another stage in this case. But let's get this in so that we can rebase the fillna pr
Update ecosystem.md
State: closed

Hi @lithomas1 ,

This pull request has been raised to update the documentation file - ecosystem.md. As discussed in issue #53451, I have replaced the deprecated qgrid with the modin-spreadsheet in the ecosystem.md file. I am new to open source, kindly let me know if any modifications need to be made.

Labels:

Comments:
Hi @datapythonista ,

I have updated the documentation, and all the CI/CD checks have passed. Kindly review it.
Hi @datapythonista ,

Kindly review this.
Hi @datapythonista  @lithomas1 ,

Kindly review this.
thanks @sqali
ENH: Don't fragment manager if convert is no-op
State: closed

- [x] closes #53853 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

cc @jbrockmendel thoughts here? This isn't ideal but since this is probably the code path that's taken 99% of the time this might make sense?

Labels:
name: Performance
description: Memory or execution speed performance
name: replace
description: replace method

Comments:
This feels a bit too deep in the internals for a whatsnew
REF: helper for merge casting
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
thx @jbrockmendel