BrandonFors/pandas-github-issues huggingface dataset:
DOC: Move notes to appropriate section in agg and aggregate docs
State: closed

- [x] closes #52253

Move a paragraph of notes from `Returns` to `Notes` in the docs of `pandas.DataFrame.agg` and `pandas.DataFrame.aggregate`, which removes the incorrect bolding.

Labels:
name: Docs

Comments:
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/52267/
thx @futureprogrammer360
DEPR: logical operations with dtype-less sequences
State: closed

- [x] closes #51521 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @jbrockmendel
STYLE: turn off PLW2901 (but keep some changes)
State: closed


Continuation of work started on closed issue #51708.
Didn't find a new issue to link this PR, may I ask you @MarcoGorelli if it is fine to link it to the closed one or shall I wait for a new issue to be opened?

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
> thanks
> 
> can you update the pyproject.toml file accordingly please?

I will update it when I finish all the files under  `pandas/core/*`. Give me few days for it, there are quite some to change :)
it's fine (and preferable) split it up into smaller chunks

we can split up the ` pandas/core/*` ignore into multiple ones
> looks like the "
> Code Checks / Docstring validation, typing, and other manual pre-commit hooks (pull_request) " is failing - code you give me a ping when that's one's fixed please?

Sure, I think I know which changes might have impacted it. I will ping you around Tuesday, unfortunately I won't make it earlier.
sure, no hurry, thanks!
Ok, I reverted most of the changes except on few files.
Under `# intentionally turned off ` I added `"redefined-loop-name"`, but I saw the errors still appearing when running `pre-commit run ruff --all-files`, so I kept `"PLW2901"` also under the to ignore section of ruff. Please let me know if this is the correct way. Also let me know if the few changes listed are fine.
BUG: DataFrame reductions losing EA dtypes
State: closed

- [x] closes #51446 (Replace xxxx with the GitHub issue number)
- [x] closes #42895
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

In the absence of 2D EAs, we need reductions to sometimes pretend the EA is 2D.  Enter "keepdims" adapted from numpy reductions.

This is still pretty ugly. I'm open to ideas to clean it up.

cc @rhshadrach any other particular cases need testing?

Labels:

Comments:
gentle ping @rhshadrach not looking to merge anytime soon but for thoughts on the approach
I've looked into this PR after you pointed it out. I had missed this PR unfortunately...
 
I like the keepdims on the arrays approach, it's a natural way to way to keep the dtype information that is being lost in reductions to scalars, rather than doing it on the frame as in my approach. I'm not a super fan of picking `"keepdims"` from the kwargs everywhere though, it seems brittle that it's necessary to add that to so many places.

I think adding a `keepdims=False` parameter to the signature of `ExtensionArray._reduce` and wrapping the returned scalar value there when `keepdims` is True before passing it on could be simpler? Then you wouldn't have to consider keepdims in all the reduction methods. That approach will also take care of wrapping the `NA` values that doesn't get wrapped currently.

Also, because I did #52707 from the point of solving #40669, I had a focus on frames with `min_count=1`. Many of the tests in #52707 fail in this PR currently (because NA result aren't wrapped in this current implementation), so those should be considered here also IMO.

I can take a closer look if the above approach works, if that's ok with you?
> I can take a closer look if the above approach works, if that's ok with you?

sounds good
Closing in favor of #52788
PERF: ArrowExtensionArray._from_sequence
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Avoid an unnecessary cast in `ArrowExtensionArray._from_sequence`. 

Noticed the time spent there when transposing and arrow-backed dataframe:

```
import pandas as pd
import numpy as np
import pyarrow as pa

data = np.random.randn(10_000, 10)
dtype = pd.ArrowDtype(pa.float64())
df = pd.DataFrame(data, dtype=dtype)

%timeit df.T

# 254 ms ± 14.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)    <- main
# 169 ms ± 3.69 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  <- PR
```


Labels:
name: Performance
description: Memory or execution speed performance
name: Arrow
description: pyarrow functionality

Comments:
Thanks @lukemanley
Correct some typos in the repository
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This PR corrects some typos that I've found in the repository.


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks @goggle
BUG #52197 proposed fix
State: closed

- [x] closes #52197
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
Thanks @zbreger
REGR: Revert GH51335
State: closed

Ref: https://github.com/pandas-dev/pandas/pull/51955#issuecomment-1478685300

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

@jbrockmendel - this only reverts the change to `_reduce`, but leaves other changes untouched. Would you prefer to revert all of #51335?

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
Does the whatsnew from #51335 need to be changed/reverted?
@jbrockmendel - yes, it did - thanks.
OK this looks fine.  Let's reiterate the plan for when i inevitably forget: merge this, backport to 2.0, then for 2.1 do either #51923, which might need to be updated to un-revert some or all of this?
I think you meant #51955 instead of #51923. #51955 can either go into 2.0 or 2.1 depending on whether there is time.
Is this ready for 2.0?
I believe this is ready, wanted to make sure @jbrockmendel is on board with the plan. I think both this and #51955 would be good for 2.0.1.
Ack - need to update the whatsnew, so no, this isn't ready anymore.
Yes I am on board with the plan.
I'm a bit more negative on this now. With 2.0.0 now released, this with #51955 will now reintroduce bad behavior (test_minmax_tzaware_skipna_axis_1 is xfailed). I'm going to see if I can get that test working in #51955, and if this is the case then I think we can go foward. 

Without being able to fix that test, I think this path should be abandoned and we should instead pursue (a) increase perf of transpose as @jorisvandenbossche has suggested (#52083) and (b) use online ops where possible as @jbrockmendel has suggested (https://github.com/pandas-dev/pandas/pull/52083#issuecomment-1476470581).
I like this plan; I'm excited to be a part of it.
I don't see any reasonable way to support axis=1 with tzaware dtypes by operating on NumPy arrays - so I think there is no way forward in this direction.
CLN: Assorted
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Clean

Comments:
Thanks @jbrockmendel
DOC: Add .map to ExtensionArray reference
State: closed


Labels:
name: Docs
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Thanks @topper-123 
On the docs build here, I'm seeing 

> /home/runner/work/pandas/pandas/doc/source/reference/api/pandas.api.extensions.ExtensionArray.map.rst: WARNING: document isn't included in any toctree

I _think_ this might be causing the docs build to "fail".
Oops yeah jumped the gun on the merge. Will revert for now.
@datapythonista - in this case the docs are still built successfully but I _think_ we can't use `/preview` because the step is skipped. Is that right? I wonder if there is an easy way still be able to view the docs if something is built (but not a big issue either way).
> @datapythonista - in this case the docs are still built successfully but I _think_ we can't use `/preview` because the step is skipped. Is that right? I wonder if there is an easy way still be able to view the docs if something is built (but not a big issue either way).

@rhshadrach we consider warnings as errors in the docs, for me the current behavior seems fine, even if I agree that the docs exist and we could preview them. The main problem I see is that it probably makes the CI quite complex to move forward with the publishing when the docs step is red. But I'm open to the change if things don't become too complicated.


Thanks @datapythonista - I suspect you're right, but will take a quick look into it to see if anything can be done.
ENH: semi joins
State: closed

- [x] closes #42784
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst` file if fixing a bug or adding a new feature.

This continues prior pull request #49661 which is currently closed.
I merged the changes from main into the development branch and moved the line added to the 2.0 release notes to the 2.1 release notes.


Labels:

Comments:
Besides moving the change to the v2.1.0 release notes, I only had to adjust the type definition of "MergeHow" compared to pull request #49661 to make the checks work. 

Can someone of you review this enhancement again so that we can merge it into the main branch?
@mroeschke @simonjayhawkins @rhshadrach
CI: Try running single_cpu/not single_cpu tests together
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

If I did this right, should get us from 20 jobs to around 14 jobs.

This'll probably slow down the current jobs by ~10 minutes (but they already take 1 hr), but we'll save at least the 5 minutes of compiling pandas on the single_cpu jobs + whatever it takes to setup the conda envs there.

Hopefully the shorter queue times during peak periods makes up for the extra time.

Labels:
name: CI
description: Continuous Integration

Comments:
This doesn't quite work yet (need to figure out how to propagate the PATTERN variable).
I would recommend bumping up the `timeout-minutes` since 1 job has to run more sets of tests now
> I would recommend bumping up the `timeout-minutes` since 1 job has to run more sets of tests now

It's already 3 hours which should be fine (all tests seem to take about 1ish hour).
Looks like on the latest commit that the single cpu task isn't running
Nice cleanup! Thanks @lithomas1. Hoping to follow up with https://github.com/pandas-dev/pandas/pull/52211 to reduce the number of queued workflows even further
COMPAT: Remove unnecessary memoryview workaround
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
I think Cython patched the memoryview bug now.

Probably safer to hold off merging for now until CI gets back to green.
Looks good. Could you merge in main once more?
Thanks @lithomas1
BUG: Removed bug from groupby/min on categoricals 
State: closed

- [ x] closes #52151 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Groupby

Comments:
Can you add a test for the fixed bug
Yes but where?
tests.groupby.test_min_max.py 
or
tests.groupby.text_categorical.py  ... maybe ? 
either of those would be fine, i guess the min_max one would be my preference.
@jbrockmendel I added the test. I think that the failings are not related with what I did, but I'm not sure. 
Could you merge the main branch once more?
Done. 
Thanks @DeaMariaLeon 
Thank you
read_fwf with urlopen test GH#26376
State: closed

- [ ] closes #26376 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

To avoid download file, I just compare the columns.

Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: IO Network
description: Local or Cloud (AWS, GCS, etc.) IO Issues

Comments:
If I added `check_before_test=True,` in the tm.network decorator then run the test case. Following error will be reported:
```
test_read_fwf.py::test_url_urlopen FAILED
pandas/tests/io/parser/test_read_fwf.py:1015 (test_url_urlopen)
args = (), kwargs = {}

    @wraps(t)
    def wrapper(*args, **kwargs):
        if (
            check_before_test
            and not raise_on_error
>           and not can_connect(url, error_classes)
        ):

../../../_testing/_io.py:222: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../_testing/_io.py:274: in can_connect
    if response.status != 200:
/Users/username/mambaforge/envs/pandas-dev/lib/python3.8/tempfile.py:468: in __getattr__
    a = getattr(file, name)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <addclosehook at 5811361200 whose fp = <_io.BufferedReader name=-1>>
name = 'status'

    def __getattr__(self, name):
        # Attribute lookups are delegated to the underlying file
        # and cached for non-numeric results
        # (i.e. methods are cached, closed and friends are not)
        file = self.__dict__['file']
>       a = getattr(file, name)
E       AttributeError: '_io.BufferedReader' object has no attribute 'status'

/Users/username/mambaforge/envs/pandas-dev/lib/python3.8/tempfile.py:468: AttributeError
```
It seems like a bug in the tm codes?
> It seems like a bug in the tm codes?

Could you diagnose how to check if ftp urls are connectable?
I done some diagnosis. url open with ftp will return a [urllib.response.addinfourl](https://docs.python.org/3/library/urllib.request.html#urllib.response.addinfourl) instance. There is a status method of addinfourl, but from python 3.9 only. And I have tried using python 3.10, the status only return None. 

It seems that if the ftp url can be open, then it is also connectable. So I added codes in can_connect to only check http/https url status.
Thanks for the test and nice investigation @liang3zy22
DataFrame transform with fillna test GH#26840
State: closed

- [ ] closes #26840 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Thanks @liang3zy22
REF: de-duplicate some test code
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Thanks @jbrockmendel
DOC: reshaping.rst Update
State: closed

- [x] closes #52142
- [x] Modified an entry in the latest `doc/source/user_guide/reshaping.rst` to include synonymous terms and explination
- [x] Modified image `doc/source/static/reshaping_pivot.png

Labels:
name: Docs
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Thanks @TravisGibbs
CI: replace flake8-pyi with ruff
State: closed

Not all rules are covered by ruff but, from my point of view, the important ones are covered.

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Slightly (un)-related: Ruff started to cover some of the autotyping parts. I would be fine replacing autotyping with ruff when ruff adds type annotations to int/float/str/... default parameters. I consider this rule "important" as it "gently encourages" people who add new arguments to also add type annotations.
sounds good

did you try upgrading to the latest autotyping? I tried, but got lots of error I couldn't resolve (though admittedly I didn't spend that long on it)
> did you try upgrading to the latest autotyping? I tried, but got lots of error I couldn't resolve (though admittedly I didn't spend that long on it)

I think these are triggered by the new `--guess-common-names`. I will go through them and see which ones are correct. But I think we probably have to exclude that option (or postpone until all incorrect guesses have correct annotations) .