BrandonFors/pandas-github-issues huggingface dataset:
BUG: groupby with dropna=False drops nulls from categorical groupers
State: closed

- [x] closes #36327 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

The function nunique here is faster than using `pd.unique`, but only works on integers:

```
arr = np.random.randint(0, 300, size)
%timeit len(unique(arr))
%timeit nunique_ints(arr)

size: 10
8.94 µs ± 133 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
2.94 µs ± 12.2 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

size: 100
9.82 µs ± 22.2 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
3.01 µs ± 5.16 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

size: 1000
11 µs ± 76.5 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
3.74 µs ± 9.7 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

size: 10000
27.9 µs ± 66.8 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
10.4 µs ± 9.43 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

size: 100000
187 µs ± 214 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
78.8 µs ± 408 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
```



Labels:
name: Groupby
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Categorical
description: Categorical Data Type

Comments:
Thanks @rhshadrach
STYLE: fix pylint reimported warnings
State: closed

Related to https://github.com/pandas-dev/pandas/issues/48855
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
> (related, we should probably make a check that standardlibrary imports are always at the top of the file)

I believe `isort` already covers this with the `pyproject.toml` config:
```
sections = ["FUTURE", "STDLIB", "THIRDPARTY" ,"PRE_LIBS" , "PRE_CORE", "DTYPES", "FIRSTPARTY", "POST_CORE", "LOCALFOLDER"]
```
isort just checks the order, rather than where imports are
BUG: groupby.describe with as_index=False incorrect
State: closed

- [x] closes #49256 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

The describe bug was found while comparing differences between `_selected_obj` and `_obj_with_exclusions` (#46944). By removing `as_index` from the conditions that determine `_selected_obj`, we fix the describe bug, the apply inconsistency (see below), and bring `_selected_obj` and `_obj_with_exclusions` closer so that the latter can eventually be removed.

Currently `groupby(...).apply` with `as_index=True` will try to use the group keys, then if that fails with a TypeError will try to redo the computation without the group keys. In main, this is also attempted when `as_index=False`, however in that case the selection context does not exclude the group keys and so it's really just attempting the same exact computation again. This PR fixes this inconsistency by making `as_index=False` adopt the `as_index=True` behavior. In the one test that picked this up, we now agree with the original behavior that was desired in #7002.

Labels:
name: Bug
name: Groupby
name: Apply
description: Apply, Aggregate, Transform, Map
name: Nuisance Columns
description: Identifying/Dropping nuisance columns in reductions, groupby.add, DataFrame.apply

Comments:
> Is there a test where multiple keys are passed into groupby as well e.g. `df.groupby(["A", "B"], as_index=False).describe()`?

There is now :smile:. Added for Series and DataFrame. There are tests for multiple groupings, but I think all cases were not in-axis, so made sense to add these too.
Nice thanks @rhshadrach (the groupby describe output looks more sane now)
DOC: Gitpod documentation
State: closed

xref #48107  

Documentation for using the development Gitpod.

Labels:
name: Docs

Comments:
I got some warning about the title capitalization in pre-commit. But it was relating to `Gitpod` and `Liveserve`, and words after the colon, which I wasn't use about the style guide there. Is that a problem? What's your recommendation here? I would prefer the brands to be capitalized in the titles. 
For the capitalization, there is a list of exceptions that get ignored here:  https://github.com/pandas-dev/pandas/blob/e41b6d7827720df90a51ff05caa689333d7e02af/scripts/validate_rst_title_capitalization.py#L21
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
@jorisvandenbossche current pre-commit errors are about changing "I" or "I'm" in headlines to lower case "i". 
Should I need to add those to the capitalization script as well?
I think it makes sense to add I as an exception
Merging, we have a sprint tomorrow and would like to reference to this.

Please ping with suggestions, if we should improve anything
thx @noatamir
WARN introduce FutureWarning for value_counts behaviour change
State: closed

- [ ] xref #49497 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

As discussed in the dev call:
- the new behaviour would be more desirable than the current one;
- warning about it if users don't pass `name=` would be very noisy
- if we just warn for `1.5.2`, then there's at least some warning and it won't be there for too long as `2.0.0` should be out in a few months' time

Note that no warning is introduced for 
```
df.groupby(col, as_index=False).value_counts()
```
as that's the one that the other `value_counts` will conform to

Labels:

Comments:
there's conflicting reviews now 😄 https://github.com/pandas-dev/pandas/pull/49640#discussion_r1020205763

Having been through adjusting all the warnings in this PR, I am concerned about noise: `df.groupby(col).apply(lambda df: df.value_counts())` in particular would emit multiple warnings

@rhshadrach would you be OK with adding a `name` parameter to ease the deprecation cycle? The only other alternative (other than not doing this) I can think of would be to make the change without warning
closing for now as it's not the right solution, will a comment to the issue with further thoughts
for #49638 updated the doc
State: closed

added the `git fetch upstream` command

- [x] closes #49638
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
thanks @ramvikrams 
> thanks @ramvikrams 

Thanks sir, feels great by being able to contribute.
BUG/API: Indexes on empty frames/series should be RangeIndex
State: closed

- [x] closes #40077
- [x] closes #49572

I've looked into #49572 and it seems that the fix is reasonable simple, see included code. In short, currently if the user hasn't supplied `index` or `columns` values, then the index/columns is a `RangeIndex` if the data has lenght > 0, while they're `Index[object]` if the length is 0. After this PR, a `RangeIndex` will be used in both cases. This will simplify type inference etc. testing etc.

However, this fix requires changes to a lot of tests (>500), of which I've fixed about 100 (not included in this version of the PR), so it's quite a lot of work to get this fixed up. Test fixing so far seems to be only a matter of ensuring that various empty dataframes/Series have `RangeIndex` rather than a `Index[object]`, so relatively simple, but also tedious work.

I didn't get much response to #49572 from the core devs, so before I spend more time on this rather big item, could you guys chime in on if you agree that this a good thing to do? IMO having the same index dtypes for series/frame of length 0 and >0 will conceptually simplify pandas and decrease the amount of surprises that users will encounter.

I've included the code  that fixes the issue in this PR, as mentioned, and ATM this PR fails, because I haven't fixed the tests, but I will get them fixed if I get positive response. If you can respond either here or in #49572, I'll of course read them both.



Labels:
name: Index
description: Related to the Index class or subclasses

Comments:
@pandas-dev/pandas-core any comment?

IMO, this simplify the construction logic nicely, so will be good to get in.
I think this is also a sensible change for 2.0.0.  
this doesn't break any tests?
@jreback , this break a lot  of tests, as empty series/frames are in a lot of tests. So I  wanted to know it if others approve of this general idea/change, before really going through the tests.
I would be supportive of this change; IMO it would needs its own section in the 2.0 whatsnew
closes #40077?
Still need docs + some cleanup, otherwise is ready if passes the CI.
> closes #40077?

Yes, this makes it so the index of the `df` is a `RangeIndex`, so the example in #40077 passes after this PR.
This is ready for review now.
Updated.
Updated.
Rebased & updated.
Gentle ping. I've rebased (no code changes) just in case.
Thanks @topper-123
make more comparisons work across dataframes of different manager types
State: closed

- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This PR fixes issues with binary operators and `.equals` across two DataFrames with different managers (one with BlockManager, one with ArrayManager). 

1. When comparing using ` df (BlockManager) == df (ArrayManager)`, there would be a cryptic `AttributeError: 'ArrayManager' object has no attribute '_slice_take_blocks_ax0'`. Instead, we raise a ValueError exception that this comparison cannot be done.


2. Comparing using `df (ArrayManager) == df (BlockManager)` and `df (ArrayManager).equals( df(BlockManager) )` now works. It did not work before because BlockManager.arrays returns a nested list, whereas ArrayManager.arrays returns a flat list. Now, we flatten BlockManager.arrays which makes the comparisons work.


## Summary

Before, ArrayManager <-> BlockManager comparisons and operations did not work at all and would give cryptic error messages. 

Now, ArrayManager -> BlockManager operations work. BlockManager -> ArrayManager operations don't work but give a helpful `ValueError`.

Labels:
name: Stale

Comments:
thanks for the tip! With `mgr_to_mgr` both comparisons work now. 
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
BUG: Fix large floats in Excel losing precision when converted to integer
State: closed

- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

When opening Excel files with large floating point values like `1E50`, Pandas will convert these values to integer, resulting in integers like `100000000000000007629769841091887003294964970946560`. Converting large floating points to integers results in erroneous values. 

This PR only converts floats to integers if they are smaller than `1E22`, which is about the maximum float that can fully represent an integer. If any floats are higher than that cutoff, then they are not converted to integers.



Labels:
name: IO Excel
description: read_excel, to_excel

Comments:
Thanks for the PR @ng-henry !
Is there a related bug report? If not, could you please open one
Thanks for the pull request, but given the issue and solution, I think more discussion is needed in a dedicate issue first. Closing this for now, but happy to reopen if other core developers think this is an adequate solution in the issue
DEPR: Remove method and tolerance in Index.get_loc, bump xarray
State: closed

Introduced in https://github.com/pandas-dev/pandas/pull/42269

Requires xarray bump as their `get_loc` usage was cleaned up in 0.21.0

Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Dependencies
description: Required and optional dependencies

Comments:
ive come around to just ignoring the partial-indexing cases.  want to revive this?
small comment, otherwise lgtm pending green
nice!
CI: Upgrade mypy to 0.990
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
i upgraded mypy to 0.990 the other day and now im getting

```
% mypy pandas
/usr/local/lib/python3.10/site-packages/dask/utils.py:366: error: Parenthesized context managers are only supported in Python 3.9 and greater  [syntax]
Found 1 error in 1 file (errors prevented further checking)
```

<b>update</b> downgrading to 0.982 didn't make the error go away, so no idea whats going on
Thanks @phofl
DEPR: Enforce default of numeric_only=False in DataFrame methods
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Deprecate
description: Functionality to remove in pandas
name: DataFrame
description: DataFrame data structure
name: Nuisance Columns
description: Identifying/Dropping nuisance columns in reductions, groupby.add, DataFrame.apply

Comments:
Thanks @jbrockmendel - changed the removal of deprecated to versionchanged and added a line about Series.rank in the whatsnew - that's the only Series method that didn't default to numeric_only=False. 
thanks @rhshadrach
TST/CI: Follow up fix test_write_fspath_all
State: closed

Follow up to https://github.com/pandas-dev/pandas/pull/49509, was not hitting the more stable `pd.read_excel` path

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Unreliable Test
description: Unit tests that occasionally fail

Comments:
Green (except doc build which is still running and should not be impacted)
thanks @mroeschke
BUG: groupby with sort=False still sorts an ordered categorical
State: closed

- [x] closes #42482 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Groupby
name: Categorical
description: Categorical Data Type

Comments:
Thanks @rhshadrach
BUG: date_range with freq="C" (business days) return value changed on 1.5.0
State: closed

- [ ] closes #49441 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Frequency
description: DateOffsets

Comments:
Awesome, thanks @lohmanndouglas
API: Series(floaty, dtype=inty)
State: closed

- [x] closes #49599 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: API Design
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
If a user want to cast to int with float data, would they have to do `ser.round().astype(int)` (assuming the "safe casting" astype isn't implemented)? 
> If a user want to cast to int with float data, would they have to do ser.round().astype(int) (assuming the "safe casting" astype isn't implemented)?

Yes.
Cool could you add this "if you want to do this" behavior in the whatsnew?
cc @seberg just noting that pandas probably will move forward with this behavior which doesn't currently align with numpy
thx @jbrockmendel
REF: remove infer_datetimelike_array
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
Thanks @jbrockmendel
DEPR: Enforce Series(float_with_nan, dtype=inty)
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @jbrockmendel
STYLE Enable Pylint statement import-self
State: closed

Issue #48855.  This PR enables the pylint type "W" warning `import-self`.
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks @uzzell