BrandonFors/pandas-github-issues huggingface dataset:
BUG: groupby with CategoricalIndex doesn't include unobserved categories
State: closed

- [x] closes #49354 (Replace xxxx with the GitHub issue number)
- [x] closes #49223
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

@gt-on-1234 - my apologies here; I was meaning to just tackle #49354 but found that my changes leaked over into #49223. Take a look and let me know what you think.


Labels:
name: Groupby
name: Categorical
description: Categorical Data Type

Comments:
> * [x]  closes [BUG: groupby with categorical index doesn't include unobserved categories #49354](https://github.com/pandas-dev/pandas/issues/49354) (Replace xxxx with the GitHub issue number)
> * [x]  closes [BUG: groupby with categorical result order differs for index type #49223](https://github.com/pandas-dev/pandas/issues/49223)
> * [x]  [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
> * [x]  All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
> * [x]  Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
> * [x]  Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.
> 
> @gt-on-1234 - my apologies here; I was meaning to just tackle #49354 but found that my changes leaked over into #49223. Take a look and let me know what you think.

No problem!  I think this should pretty much cover the changes in #49318 so will close my PR.  
Thanks @rhshadrach
DEPR: object-dtype bool_only
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @jbrockmendel
ENH: Implement io.nullable_backend config for read_csv(engine="pyarrow")
State: closed

- [x] xref #48957 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement
name: IO CSV
description: read_csv, to_csv
name: Arrow
description: pyarrow functionality

Comments:
thx @mroeschke
DEPR: Remove deprecated Timestamp.freq
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Small nit here -  you are removing `freq`, not deprecating it, because it was already deprecated, right?  Might want to change the name of the PR??
> Might want to change the name of the PR??

This is the pattern we've stumbled into for deprecation-enforcement PRs.
> > Might want to change the name of the PR??
> 
> This is the pattern we've stumbled into for deprecation-enforcement PRs.

For my own education, what's the pattern for a PR that deprecates as opposed to removal?
> For my own education, what's the pattern for a PR that deprecates as opposed to removal?

The patterns are not distinct.  I'm open to suggestions.
> > For my own education, what's the pattern for a PR that deprecates as opposed to removal?
> 
> The patterns are not distinct. I'm open to suggestions.

Maybe use "DEPR:  Deprecated foobar" when deprecating, and "DEPR: Remove deprecated foobar" to correspond to the deprecation and removal of the deprecated behavior of foobar, respectively.
Updated the PR title. Also lately we have had a unofficial moratorium on merging deprecating functionality PRs until we've cleared the 1.x deprecations since the checklist issue is unfortunately not comprehensive.
Thanks @jbrockmendel
DOC: Moved the sort_values doc from generic.py to frame.py
State: closed

Hello,

this is my first pull request to pandas and on GitHub in general.
While looking into an issue in sort_values in frame.py I saw the
TODO about moving the sort_values doc and did it.

Cheers, Felix

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks. Could you build the docs locally and share a screenshot of how the docs look after this change? https://pandas.pydata.org/pandas-docs/stable/development/contributing_documentation.html
Thanks for your PR - looks like this is no longer active (no activity in > 1 month), so I'll close it for now

If you still interested in working on this and addressing the review comments, please let us know and we can reopen
BUG: array_equivalent_object with mismatched shapes
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug

Comments:
Thanks @jbrockmendel
DEPR: disallow int fill_value in shift with dt64/td64
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @jbrockmendel
DEPR: DTA(float_data, dtype=dt64tz)
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @jbrockmendel 
@jbrockmendel looks like the arm build started failing after this one. Is this possible?
its plausible.  IIRC there was something about `np.array([np.nan]).astype(np.int64)` behaving weird on ARM.
could try changing `data = data.astype(np.int64)` to `data = data.astype("datetime64[ns]").view("i8")` and see if that does the trick
BUG: join on column with index
State: closed

- [x] closes #28243 
- [x] closes #15692
- [x] closes  #17257

 (below can be closed if agree on expected result)
xref #28220
xref #33232
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Lot's of existing tests were changed, i think there are multiple reported bugs on this.. I might be missing a few others as well.

The changed tests could be broadly classified as:

1. The result contains value in that was not present in the original column. 
2. The index of the merged df was different. According to the docs, when merged on a index it should pass on.
3. ~~The data type of the expected df was object instead of say datetime, or someother~~ 
4. ~~In merge_asof I think since it defaults to left join, it should inherit the index from left df~~ (these are working fine, there was some trouble with local build initially)

One particular test, which I am little puzzled about the expected output is the below. Joining on index on one df, and partial index and partial column in the other df. I was not able to figure out, so skipped for the time
https://github.com/pandas-dev/pandas/blob/f9ff3796329e4bedb4a5477739f5eb8d2e40761d/pandas/tests/reshape/merge/test_merge_index_as_string.py#L166-L189


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Stale

Comments:
Most of ~~them~~ the tests are. For the rest I'm not very sure about the expected behaviour. 
All the three bugs are kind of similar. 
You're joining / merging two dfs using a column and an index. These are just specific cases I believe.  

EDIT: Even the two xref bugs are similar. But I am not very sure about the expected dfs from those. 
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
DEPR: non-keyword arguments
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @jbrockmendel (the ASV failures that was affecting others PRs seems to have gone away)
API: stop silently ignoring parsing failures with dtype=dt64
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

It's a bit ambiguous whether to call this a deprecation-enforcement or API change.  In 1.3.0 we have

```
- Deprecated behavior of :class:`DataFrame` constructor when a ``dtype`` is passed and the data cannot be cast to that dtype. In a future version, this will raise instead of being silently ignored (:issue:`24435`)
```

but the PR that did that deprecation added that warning in a different code path, so we don't currently issue a warning for this case.  (and the whatsnew didn't say anything about Series).

Labels:
name: Datetime
description: Datetime data dtype
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
> It's a bit ambiguous whether to call this a deprecation-enforcement or API change

Since there wasn't a deprecation warning (well, not for all cases), I think it's fine to just list it as a breaking change (as you did right now I think).
Thanks @jbrockmendel
ENH:  add new feature of yyyyww for .isocalendar()  to resolve issue #49353
State: closed

- [x] closes #49353   (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
> 



> Is there an open Github issue associated with this?
> 
> I would be -1 for this change. `isocalendar` should align with `datetime.datetime` which doesn't return `yearandweek`

Yeah, this is an issue associated with this. Please see the link here: https://github.com/pandas-dev/pandas/issues/48947

Thanks for the suggestion about datetime.datetime. 
 
-1 as well this is easily computable and doesn't warrant a method
Thanks for the PR, but closing as it appears this approach is undesirable.
Thanks for the time for reviewing the code.
________________________________
From: Matthew Roeschke ***@***.***>
Sent: 01 November 2022 04:03
To: pandas-dev/pandas ***@***.***>
Cc: Rui Huang ***@***.***>; Author ***@***.***>
Subject: Re: [pandas-dev/pandas] ENH: add new feature of yyyyww for .isocalendar() to resolve issue #49353 (PR #49355)


Thanks for the PR, but closing as it appears this approach is undesirable.

—
Reply to this email directly, view it on GitHub<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fpandas-dev%2Fpandas%2Fpull%2F49355%23issuecomment-1297396333&data=05%7C01%7Cu5927645%40anu.edu.au%7C80573ea4400d4492577708dabb61e5ce%7Ce37d725cab5c46249ae5f0533e486437%7C0%7C0%7C638028326398403178%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=CBAgYqARprOSk5BMgc3ALFnoggcBWKQPJint3RokEQg%3D&reserved=0>, or unsubscribe<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FA2PTNSI4MJIVJ5SCT7MBTKDWF73XVANCNFSM6AAAAAARQCVYGM&data=05%7C01%7Cu5927645%40anu.edu.au%7C80573ea4400d4492577708dabb61e5ce%7Ce37d725cab5c46249ae5f0533e486437%7C0%7C0%7C638028326398403178%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XRqAYbyFzdo3aWrOWQh2a%2BF0ZTQUFEys%2BmI2pnOi8Qc%3D&reserved=0>.
You are receiving this because you authored the thread.Message ID: ***@***.***>
API: allow mixed-datetimes-and-ints in to_datetime, DatetimeIndex
State: closed

- [x] closes #49037 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

cc @jreback this goes a long way towards getting Series/Index constructors to match, but i dont see the reason for the current behavior, do you remember?

Labels:
name: Datetime
description: Datetime data dtype

Comments:
I commented on the issue (https://github.com/pandas-dev/pandas/issues/49037), I think I am also +1 on allowing the mixture (although I don't like that specific behaviour, it seems the most consistent if we want to have generally applicable rules)

@jbrockmendel one thing I am wondering: the `allow_mixed` keyword, that's not useful to keep for potential stricter behaviour? (although I assume it's not actually useful for "only datetime objects", since `allow_mixed=False` can also mean that you have only ints)
> one thing I am wondering: the allow_mixed keyword, that's not useful to keep for potential stricter behaviour? (although I assume it's not actually useful for "only datetime objects", since allow_mixed=False can also mean that you have only ints)

Assuming you mean in array_to_datetime.  I don't think its particularly useful, but its also not harmful (except for not being tested).  For only-datetime-objects (i.e. the strict from_scalars) i think it may be cleaner to implement a new dedicated function than further overloading array_to_datetime.  
failures unrelated
Thanks @jbrockmendel
REF: simplify sanitize_array
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
Thanks @jbrockmendel
DEPR: to_datetime('now') match Timestamp('now')
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @jbrockmendel
DEPR: Enforce disallowing loc with positionals
State: closed

Introduced in https://github.com/pandas-dev/pandas/pull/31840

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
looks like sql asvs failed
the asvs failing here are also failing on some of my PRs, so must be unrelated
> the asvs failing here are also failing on some of my PRs, so must be unrelated

It looks to have failed on main for https://github.com/pandas-dev/pandas/commit/6b4fa02e10480c4ddae0714e36b7fe765fa42eac, so might be related?
Merged and green
thanks @mroeschke
REF: simplify maybe_infer_to_datetimelike
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
Thanks @jbrockmendel
STYLE: fix pylint simplifiable-if-expression warnings
State: closed

Related to https://github.com/pandas-dev/pandas/issues/48855
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.



Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks @Moisan
STYLE fix: pylint "consider-using-from"
State: closed

Refers to one of the issues in https://github.com/pandas-dev/pandas/issues/48855

- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

- [x] Fixed imports triggering pylints "consider using from" message 


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Looks good, could you just resolve the conflict please?
not sure the merge conflict was quite resolved right, there's now an empty file `pandas/io/excel/_xlwt.py`, could you remove it please?
Awesome, thanks @fabinca