BrandonFors/pandas-github-issues huggingface dataset:
BUG / CoW: Series.view not respecting CoW
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This isn't a particularly good solution, but ties into the problem we will have to solve for setitem with a Series anyway so should be good for now.

Labels:
name: Series
description: Series data structure
name: Copy / view semantics

Comments:
On a separate note, IMO we should also just deprecate this Series.view method: https://github.com/pandas-dev/pandas/issues/20251
New feature: Read first n_rows of Parquet File
State: closed

- [x] closes #51830 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Why is this PR closed? Can we discuss on how/if to implement this feature?
ERR: Raise ValueError when non-default index is given for orc format
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

orc can't handle indexes right now (they are dropped), so we should raise like we did in feather

Labels:
name: IO Parquet
description: parquet, feather
name: Arrow
description: pyarrow functionality

Comments:
> orc can't handle indexes right now (they are dropped)

Just confirming, they are dropped without a warning/exception from pyarrow?
Yes they are just gone when roundtripping
Thanks @phofl
DOC: sort has no effect on filters
State: closed

- [x] closes #51692

Updated docs for `DataFrame.groupby` and `Series.groupby`, pointed out that the parameter the `sort` parameter has no effect on filters.


Labels:
name: Docs
name: Groupby

Comments:
In the API docs for DataFrame.groupby and Series.groupby the fact that `as_index` has no effect on filters is not mentioned. 
Should I mention it? 

@MarcoGorelli - any further comments?
Thanks @MarcoGorelli and @rhshadrach for helping me with this pr.
Fix to_excel not rendering None values in MultiIndex
State: closed

- [x] closes #51252
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

---

The change fixes the issue of not rendering `NaN`-valued column cells when exported to Excel.
```py
import pandas as pd
import numpy as np

Cols = pd.MultiIndex.from_tuples(
    [
        ("Item", "Type 1", "Class A"),
        ("Item", "Type 1", "Class B"),
        ("Item", "Type 2", "Class A"),
        ("Item", "Type 2", "Class B"),
        ("Item", "Type 3", None),
    ]
)

df = pd.DataFrame(np.random.randn(10, 5), columns=Cols)
df.to_excel("Test.xlsx")
df
```
<img src="https://user-images.githubusercontent.com/107754076/223540765-c4789732-eed3-4b71-aee1-8a4abfa688be.png" width="250"/>
<img src="https://user-images.githubusercontent.com/107754076/223540913-f3b11548-52de-4fcc-98aa-ddc0a666aaa3.png" width="250"/>

However, when I do roundtripping of the Excel sheet back into a `DataFrame` I get this:
```py
pd.read_excel("Test.xlsx", header=[0, 1, 2], index_col=[0])
```
<img src="https://user-images.githubusercontent.com/107754076/223541702-5a7e047f-32a6-43ed-8d3c-85b7a7ee2aad.png" width="250"/>

So I have a question. Should I update the logic of `read_excel` to also apply the conversion of `NaN` values of regular cells (specified by the `na_values` parameter) to column values as well? If so, would it make more sense to do it [here](https://github.com/GodelTech/pandas/blob/issue-51252-nan-values-multiindex/pandas/io/parsers/base_parser.py#L503-L512) (where regular values are converted), or [here](https://github.com/GodelTech/pandas/blob/issue-51252-nan-values-multiindex/pandas/io/parsers/python_parser.py#L373-L375) (where column values assigned the `Unnamed` values)? Or there's some reason why the current functionality is not applied for column values and the user is expected to convert them manually (like [here](https://stackoverflow.com/questions/40839609/rename-unnamed-multiindex-columns-in-pandas-dataframe))?


Labels:
name: IO Excel
description: read_excel, to_excel

Comments:
I don't believe we can differentiate between `''` and an NA value in the Excel file. Would it make more sense to assume the value is `''` instead of an NA value?
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
PERF/REF: make window/groupby imports lazy
State: closed

Maybe makes 20ms difference, hard to measure.  But separating this out from other import-speed stuff bc the docstring movement makes the diff big.

Labels:

Comments:
Restored groupby imports in core.api bc i think dask accesses `pd.core.groupby.GroupBy`.  Restoring that basically undoes the perf improvement, so this is just a refactor at this point
DEPR: scalar index for length-1-list level groupby
State: closed

- [x] closes #51583 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions. (nothing new here)
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Previously, iterating over a Groupby object that had been initialized with a length-1-list ```level``` parameter (ex. level = [0]) would return scalar indexes as opposed to tuples with only the first element defined (ex. _0_ as opposed to _(0, )_ ). This behavior differs from when using the ```by``` parameter and so the change is motivated.

Labels:
name: Bug
name: Groupby
name: MultiIndex
name: Deprecate
description: Functionality to remove in pandas

Comments:
> Thanks for the PR! This PR doesn't deprecate the behavior, it changes the behavior.
> 
> We need to first warn users that the behavior will change _without_ changing the current behavior.

Well that's very reasonable, should have thought of that...  What's the best way to go about this? Maybe just make a new PR (with an actual "depreciation") and then leave/close this branch so that it can be used for reference when the time is right? 
I would suggest just modifying this PR to deprecate the behavior; as you've already demonstrated the changes to enforce that deprecation won't be extensive.
I'm not sure how to resolve this failing mypy check, using `is_list_like(level)` together with `len(level)` does not cause any trouble in grouping.py. Anyone got a clue?
 
Looks okay to add a type ignore here. We should eventually use a type guard with is_list_like but might now be able to do that just yet.
Thanks @august-tengland!
add concat function dtype test
State: closed

- [ ] closes #32934
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

The old PR closed, so created new one to close #32934.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
1 check failed with current code,Ubuntu/Data Manager. It means that the code
`pd.concat([df_time, df_float.iloc[:0]])`
has data type "object".
If I updated the codes to the following code:
` tm.assert_equal(
        pd.concat([df_time, df_float.iloc[:0]]),
        pd.DataFrame({"A": pd.array(["2000"], dtype="datetime64[ns]")}).astype({"A":"object"}),
    )`
The Ubuntu/Data manager will pass, but other 21 checks will fail since the code
`pd.concat([df_time, df_float.iloc[:0]]) `
may have data type "datetime64" in these 21 checks(I only checked some).
Looks like the `using_array_manager` fixture needs to be used to set the dtype for the last example
It seems that another numpy error in "Ubuntu / Numpy Dev(pull_request)".
```
>       warnings.warn("`product` is deprecated as of NumPy 1.25.0, and will be "
[134](https://github.com/pandas-dev/pandas/actions/runs/4379948504/jobs/7666450861#step:8:136)
                      "removed in NumPy 2.0. Please use `prod` instead.",
[135](https://github.com/pandas-dev/pandas/actions/runs/4379948504/jobs/7666450861#step:8:137)
                      DeprecationWarning, stacklevel=2)
[136](https://github.com/pandas-dev/pandas/actions/runs/4379948504/jobs/7666450861#step:8:138)
E       DeprecationWarning: `product` is deprecated as of NumPy 1.25.0, and will be removed in NumPy 2.0. Please use `prod` instead.
```
Thanks @liang3zy22 (that failure is unrelated)
REF: define arithmetic methods non-dynamically
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Avoid runtime imports, get stuff out of `ops.__init__`, which was the goal 4-5 years ago when ops was first madea  directory.

Move relevant helper methods from frame/series files .  Decided on a new mixin rather than dumping everything directly in DataFrame/Series bc those files are too bug as it is.  If others feel strongly i'd be OK putting it all in DataFrame/Series.

Labels:
name: Refactor
description: Internal refactoring of code

Comments:
The failing doctest looks like the doctest is wrong.  Must just not be getting in the status quo?

```
>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd'])
>>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e'])
>>> a.divmod(b, fill_value=0)
(a    1.0
- b    NaN
- c    NaN
+ b    inf
+ c    inf
 d    0.0
 e    NaN
 dtype: float64,
 a    0.0
 b    NaN
 c    NaN
 d    0.0
 e    NaN
 dtype: float64)
```
> The failing doctest looks like the doctest is wrong. Must just not be getting in the status quo?

Hmm is this the same for DataFrame too and/or a recent change?

For label b, I would expect this to follow:

```
In [10]: 1 / np.nan
Out[10]: nan
```

For label c, since there's no corresponding values in series `b` I would expect it to be filled with nan too?
> I would expect it to be filled with nan too?

At the reindex stage it is, but then `fill_value=0` part is done after the reindex.
Thanks @jbrockmendel
DEPR: observed=False default in groupby
State: closed

- [x] closes #43999 (Replace xxxx with the GitHub issue number)
- [x] closes #30552
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Groupby
name: Categorical
description: Categorical Data Type
name: Deprecate
description: Functionality to remove in pandas

Comments:
for users who work interaactively (ipython, jupyter,..), Is there way to avoid the `FutureWarning` coming repeatedly? It will probably get quite annoying after a while...

Otherwise LGTM.
Thanks @topper-123 

> for users who work interaactively (ipython, jupyter,..), Is there way to avoid the FutureWarning coming repeatedly? It will probably get quite annoying after a while...

Users can specify `observed=True|False`. Also, just to be sure, this will only warn if one of the groupings is categorical.

I think you're thinking of some global option? I would be -1 on that, and hope this will be rare enough that it won't pester users. They'll need to specify `observed=True` if they want to adopt future behavior or `observed=False` if they want to maintain current behavior eventually.




Thanks @rhshadrach
REF: Add ExtensionArray.map
State: closed

- [x] closes https://github.com/pandas-dev/pandas/issues/23179

Working with the `.map` method is very uncomfortable in main, because of issues with `ExtensionArray`:
* `ExtensionArray` is not required to  have a `.map` method, but may have it.
* Some `ExtensionArray` subclasses, that do have a `map` method have one function parameter (`mapper`), while other have two parameters (`mapper` and `na_action`).
* There is fallback functionality for calculation  `ExtensionArray.map` in `IndexOps.Mixin._map_values`. This means we can calculate `.map` for `ExtensionArrays` indirectly, if we wrap them in a `Series` or `Index`, but we cannot rely on the ability to do  it directly on the `ExtensionArray`.
* `ExtensionArray` subclasses often want create their own `.map` methods
* Because of all the above issues it is often very maze-like to follow code that uses `.map`

This PR solves this by:
1. moving the functionality in `IndexOpsMixin._map_values` to a function `map_array` in `algorithms`.
2.  adding a `.map` method to `ExtensionArray` that calls the new `map_array` function. Subclasses will override this method where needed.
3. Call the new `map_array` function inside `IndexOpsMixin._map_values`. 
4. Ensuring a common signature for the `.map` method for `ExtensionArray`, i.e. always two method parameters (`mapper` and `na_action`), so we can reliably call the `.map` method on any `ExtensionArray` subclass.

Note that this PR doesn't really add new functionality, because we're mostly just moving code around, though it is now possible to call the `map` method on any `Extensionarray` directly, instead of going through e.g. `Series.map`.

In the main branch, there are currently several `ExtensionArray` subclasses where `na_action='ignore'` doesn't work. I'm working on making it work with all of the `ExtensionArray` subclasses and there is already #51645, which handles `Categorical`/`CategoricalIndex` and more will follow.


Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Does this close any of #23179, #15706, #32815?
> Does this close any of #23179, #15706, #32815?

* #23179: Yes (adds `.map` to `ExtensionArray`)
* #15706: I don't think this issue is solveable while maintaining the performance benefits of a dedicated `Categorical.map` , (i.e. doing the op only once per category is intended).
* #32815: not in this PR, but that is fixed by #51645 for the `Categorical`/`CategoricalIndex` case. I'll take on `DateTimeArray`/`DateTimeIndex` etc. in a follow-up, so that will be fixed shortly.

Might need something in the extension tests?
> Might need something in the extension tests?

I've added some simple tests in the newest version.
I've updated.
Gentle ping.

I’m intending to follow up with fixes towards #51644, and it would be great if this could be merged.
Merge when ready @jbrockmendel 
thanks @topper-123 
@topper-123 does any of your recent work on this close #15706?
I'vew answered in #15706 to keep the discussion in one place.
STYLE Enable ruff TCH for pandas/core/util/*
State: closed

ref #51740 

This PR will enable ruff TCH for `pandas/core/util/*` and fix associated issues

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks - can you run `pre-commit run --all-files` please?
I ran `pre-commit run ruff --all-files` and it did pass the check. But since this also happened yesterday I have to rely on the CI...
given that it modified files, I doubt it worked yesterday

anyone, should be good, thanks
CLN: Enable codespell for all files
State: closed

- [x] closes #44045 (Replace xxxx with the GitHub issue number)

It seems works fine in current main branch.

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks

doesn't look like it
```python
 codespell...............................................................................................Failed
- hook id: codespell
- duration: 1.66s
- exit code: 65

pandas/tests/io/data/xml/cta_rail_lines.kml:14: fo ==> of, for, to, do, go
pandas/tests/io/data/xml/cta_rail_lines.kml:14: DESCRIPTIO ==> DESCRIPTION
pandas/tests/io/data/xml/cta_rail_lines.kml:27: fo ==> of, for, to, do, go
pandas/tests/io/data/xml/cta_rail_lines.kml:27: DESCRIPTIO ==> DESCRIPTION
pandas/tests/io/data/xml/cta_rail_lines.kml:40: fo ==> of, for, to, do, go
pandas/tests/io/data/xml/cta_rail_lines.kml:40: DESCRIPTIO ==> DESCRIPTION
pandas/tests/io/data/xml/cta_rail_lines.kml:53: fo ==> of, for, to, do, go
pandas/tests/io/data/xml/cta_rail_lines.kml:53: DESCRIPTIO ==> DESCRIPTION
pandas/tests/io/data/xml/cta_rail_lines.kml:66: fo ==> of, for, to, do, go
pandas/tests/io/data/xml/cta_rail_lines.kml:66: DESCRIPTIO ==> DESCRIPTION
pandas/_libs/src/ujson/lib/ultrajson.h:190: iteratable ==> iterable
pandas/_libs/src/ujson/lib/ultrajson.h:202: iteratable ==> iterable
pandas/_libs/testing.pyx:190: ambigous ==> ambiguous
pandas/_libs/src/parser/tokenizer.c:1831: housand ==> thousand
pandas/_libs/ops_dispatch.pyx:94: nin ==> inn, min, bin, nine
pandas/_libs/missing.pyx:55: ambigous ==> ambiguous
pandas/_libs/tslibs/strptime.pyx:3: TimeRE ==> timer
pandas/_libs/tslibs/strptime.pyx:11: TimeRE ==> timer
pandas/_libs/tslibs/strptime.pyx:28: TimeRE ==> timer
pandas/_libs/tslibs/strptime.pyx:236: TimeRE ==> timer
pandas/_libs/tslibs/strptime.pyx:536: TimeRE ==> timer
pandas/_libs/tslibs/strptime.pyx:552: TimeRE ==> timer
pandas/_libs/tslibs/strptime.pyx:562: TimeRE ==> timer
pandas/_libs/tslibs/strptime.pyx:571: TimeRE ==> timer
pandas/_libs/hashing.pyx:58: datas ==> data
pandas/_libs/hashing.pyx:100: datas ==> data
web/pandas/config.yml:161: dfine ==> define
WARNING: Decoding file using encoding=utf-8 failed: pandas/tests/io/data/xml/baby_names.xml
WARNING: Trying next encoding iso-8859-1
pandas/_libs/lib.pyx:2653: becase ==> because
pandas/_libs/tslibs/timestamps.pyx:2347: overriden ==> overridden
pandas/_libs/tslibs/timestamps.pyx:2357: overriden ==> overridden
pyproject.toml:276: indendation ==> indentation
pandas/_libs/tslibs/timedeltas.pyx:1109: overrided ==> overrode, overridden
```

how did you run it locally? it should be `pre-commit run codespell --all-files`
I forgot to run it with `--all-files` before.

I make some changes, but some of them may not suitable. I am open to any suggestions or advice.
BUG: ArrowDtype.numpy_dtype returning ns units for non-nano timestamp and duration types
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst` file if fixing a bug or adding a new feature.

`ArrowDtype.numpy_dtype` goes through the pyarrow `to_pandas_type` method which was added before pandas supported non-nano units. This PR ensures the correct units are passed along to the numpy dtype. This fixes 6 xfails.

pyarrow already preserves the units when going directly from pyarrow to numpy:
```
>>> import pyarrow as pa
>>> arr = pa.array([1, 2, 3], type=pa.duration("us"))
>>> arr.to_numpy()
array([1, 2, 3], dtype='timedelta64[us]')
```


Labels:
name: Bug
name: Arrow
description: pyarrow functionality
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
thanks @lukemanley
STYLE: Enable ruff TCH for pandas/core/common.py
State: closed

- [X] partly fixes #51740
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Hi everyone!
Thanks for this awesome package 👍 
This PR will enable `ruff TCH` for the `pandas/core/common.py` file, the necessary changes in said file have been implemented.


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Sorry for spamming small changes, the `pre-commit` check doesn't throw any errors locally..