BrandonFors/pandas-github-issues huggingface dataset:
BUG: REF: Groupby refactor
State: closed

This is a follow-up, scaled back version of https://github.com/pandas-dev/pandas/pull/46258, that hopefully should be easier to merge.

- [NA, this is small ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Groupby
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
I added a merge commit again because the CI run seemed to be failing because it was using an old version of the PR, before I updated the tests? Anyways, this PR seems to be innocent of any failing tests.
OK, some of the tests (https://github.com/pandas-dev/pandas/runs/6011207511?check_suite_focus=true) seem to still not be picking up the changed tests and are failing? But some of them have picked up the modified tests and are therefore passing? Not sure what this means, is the CI being weird or does this actually mean I need to adjust the tests more because I'm missing something?
@NickCrews - does the one failing test pass for you locally?
> @NickCrews - does the one failing test pass for you locally?

Poking into this showed that there was a second `assert` in the test case that I hadn't noticed and updated 🤦 . Looks good locally now, probably CI will pass.
little rebase to fix a merge conflict that appeared from the whatsnew changing in main
hmm i think this needs to merge master once more
@NickCrews 
oh nvm

i was looking at an older version

thanks @NickCrews 
Thanks @NickCrews!
BUG: Do not allow urllib to hijack all non-HTTP protocols
State: closed

Currently, many non HTTP protocols are routed to urllib to open, even when not supported by urllib.  In particular, opening files via SFTP will always fail, since the attempt to open the file is not sent to fsspec, which supports that protocol.

The following code demonstrates the issue. It will fail pre-patch, and succeed after.

```
pandas.read_csv('sftp://some.server/file.csv', storage_options={'username': 'my_username', 'password': 'my_password', 'allow_agent': False})
```

I apologize for not adding a test, as I didn't find any other sftp tests in test_fsspec.py, and I wasn't sure how to best handle a functional test for that protocol. Also, the real problem is misuse of `is_url`. There may be a different support function that would be correct to use in its place, but a quick search didn't find anything. However, please feel free to throw this pull request away and fix in a more appropriate manner if necessary.

Labels:

Comments:
Hello @maalgorium! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-04-12 18:23:35 UTC
BUG: Coerce to native types to_dict list and index (#46751)
State: closed

- [x] closes #46751 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: IO Data
description: IO issues that don't fit into a more specific label
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
@rhshadrach PR for fix
@rhshadrach looks like there are 2 tests failing that don't seem in any way related to my code, how should I proceed?
It's actually just one; the `pandas-dev.pandas` will fail anytime one of the Azure pipeline build fails, so it's just the py38_32bit. This build is failing on all PRs for reasons I haven't been able to figure out; okay to ignore.
Thanks @RogerThomas!
TST: Add a test for `series name should be preserved in pser.mode() `
State: closed

Add a test for pser.mode() should set series name

- [x] closes #46737
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
name: metadata
description: _metadata, .attrs

Comments:
@mroeschke Done, thanks!
Excellent, thanks @Yikun 

(Failure unrelated)
REF: stronger typing in libconversion
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code
name: Datetime
description: Datetime data dtype

Comments:
Thanks @jbrockmendel
Add a license file for klib (khash)
State: closed

File contents are exactly those of: https://raw.githubusercontent.com/attractivechaos/klib/9a063b33efd841fcc42d4b9f68cb78bb528bf75b/LICENSE.txt

- [ ] closes #xxxx (Replace xxxx with the Github issue number) **Not applicable**
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature **Not applicable**
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature. **Not applicable**


Labels:
name: Docs

Comments:
thanks @musicinmybrain
ENH: add validate as a param to join (#46622)
State: closed

- [ ] closes #46622

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature

- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/v1.5.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Does it matter for the change to be merged if one code check fails? The failure seems to come from a file I never touched, while the previous two commits were able to pass all checks.
> Does it matter for the change to be merged if one code check fails?

This was a known failure affecting all builds. Just waiting on if other maintainers would like to review
> > Does it matter for the change to be merged if one code check fails?
> 
> This was a known failure affecting all builds. Just waiting on if other maintainers would like to review

Sure. Thank you!
looks good, it might be worth adding something to https://pandas.pydata.org/pandas-docs/dev/user_guide/merging.html?highlight=join#joining-on-index (there is a lot of sections, so this is a *maybe*).
thanks @gaotian98 very nice!
REF: de-vendor _stprtime code
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
nice!
DOC: Improve .at indexer API docs
State: closed

- [x] closes #46722 

Not sure about the wording of the `ValueError` and whether it is worth adding the additional methods under `See Also` but the same doc is used for both DataFrame and Series so I thought that mentioning all the methods is good for the sake of completeness

Labels:
name: Docs

Comments:
> under See Also but the same doc is used for both DataFrame and Series so I thought that mentioning all the methods is good for the sake of completeness

this sounds fine
Hello @FactorizeD! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-05-10 18:54:56 UTC
Failing check seems not related tot his PR. Can this be merged in the current state after the check issue is resolved, or should I do something more?
Thanks @FactorizeD (sorry for the delay). Failure was unrelated
WARN: PerformanceWarning for non-pyarrow fallback
State: closed

- [x] closes #42613 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Warnings
description: Warnings that appear or should be added to pandas
name: Arrow
description: pyarrow functionality

Comments:
hmm lots of warnings not being caught
> pandas/tests/strings/test_find_replace.py::test_contains_moar[string[pyarrow]]
  /home/runner/work/pandas/pandas/pandas/tests/strings/test_find_replace.py:182: PerformanceWarning: Falling back on a non-pyarrow code path which may decrease performance. Upgrade to pyarrow >=4 to possibly suppress this warning.
    result = s.str.contains("a")
Hello @mroeschke! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:













There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-04-14 01:47:43 UTC
Test column addition using .at with different values
State: closed

- [x] closes #30649 

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Stale

Comments:
@jreback I am tagging you since you added a few labels to the PR a couple of days ago. Failing check seems not related to this PR. Is there something more that can be done here from my side at the moment or should I simply wait for the review?
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
BUG: to_json incorrectly localizes tz-naive datetimes to UTC
State: closed

- [ ] closes #38760 (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
Sorry for the low quality code. I was prototyping quickly and forgot to clean up afterwards.

Regarding https://github.com/pandas-dev/pandas/pull/46730/#discussion_r856421763, I think I'll stick to the current approach for now. Existing perf benchmarks surprisingly seem unaffected by this change, so not sure its worth pursuing at least for now(it might be worth doing for the tz-aware case later).
thanks @lithomas1 very nice!

I believe there is a de-duplication followup (values_for_json) if you can
ENH: Add numeric_only to certain groupby ops
State: closed

Part of #46560

- [x] Closes #46524  
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

For the ops that have numeric_only added, we have the option of dropping nuisance columns even when `numeric_only=False` is specified to be consistent with other groupby ops (e.g. sum). I decided to implement the correct behavior here instead; this required adding an argument to the cython/python paths. This argument can be removed in 2.0 when all ops would raise.

Labels:
name: Enhancement
name: Groupby
name: Nuisance Columns
description: Identifying/Dropping nuisance columns in reductions, groupby.add, DataFrame.apply

Comments:
@jreback - gentle ping.
thanks @rhshadrach very nice!
ENH: Use pyarrow.compute for unique, dropna
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).



Labels:
name: Arrow
description: pyarrow functionality

Comments:
Thanks @mroeschke for the PR. my understanding of #42613 is that we should no longer be implementing any fallback behavior. (there is not definitive policy on that there, so have not yet removed the fallbacks already implemented for StringArray)

That issue applies specifically to StringArray but with the work you have done/doing to have a common base class for pyarrow backed EAs, we may be adding fallback behaviour for the pyarrow backed StringArray?
> my understanding of #42613 is that ...

more of the discussion was actually in #42597
Ah thanks, I wasn't aware of this discussion.

My prior, related PR's so far have just been moving existing methods, so no fallback behavior should have been introduced I think.

For this PR, I will just remove the super calls for now and reintroduce them later if we decide on a different fallback policy
> Ah thanks, I wasn't aware of this discussion.
> 
> My prior, related PR's so far have just been moving existing methods, so no fallback behavior should have been introduced I think.
> 
> For this PR, I will just remove the super calls for now and reintroduce them later if we decide on a different fallback policy

comments on the other issues is we can show a PerformanceWarning if we are falling back. But let's do that as a pre-cursor 
actually if you can do that move in this PR and I think this also needs a whatsnew note. ping on green.
Here are the perf differences (@simonjayhawkins is correct that the array's (or any ExtentionArray's?) dropna is not hooked up to `Series/index/DataFrame.dropna()` yet)

PR
```
In [1]: import pyarrow as pa
   ...: pa.__version__
   ...:
Out[1]: '7.0.0'

In [2]: ser = pd.Series(["1", np.nan] * 100_000, dtype="string[pyarrow]")

In [4]: %timeit ser.unique()
   ...:
1.3 ms ± 6.59 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [3]: %timeit ser.array.dropna()
820 µs ± 1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```

main
```
In [1]: ser = pd.Series(["1", np.nan] * 100_000, dtype="string[pyarrow]")
   ...:

In [2]: %timeit ser.unique()
   ...:
24.4 ms ± 181 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [2]: %timeit ser.array.dropna()
1.21 ms ± 2.45 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```
thanks @mroeschke really nice
BUG: Fix __instancecheck__ and __subclasscheck__ of metaclass in pandas.core.dtypes.generic
State: closed

- [x] closes #46684 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

---

#46684 is caused by wrong `__instancecheck__` implementation, but `__subclasscheck__` implementation is also wrong. I fixed both.

```python
import pandas as pd
from pandas.core.dtypes.generic import ABCDataFrame

print(isinstance(pd.DataFrame, ABCDataFrame)) # Should have returned False, returned True instead
print(issubclass(pd.DataFrame(), ABCDataFrame)) # Should have raised, returned True instead
```

Labels:
name: Bug
name: Internals
description: Related to non-user accessible pandas implementation
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Alright done!
Thanks @leonarduschen!
Raise `FileNotFoundError` in `read_json` if input looks like file path but file is missing
State: closed

- [x] closes #29102
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/v1.5.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
> Also @twoertwein, for other file extensions with compression, it doesn't appear we manually raise a `FileNotFoundError`. Should this automatically be handled by the handler code?


Would maybe be better to raise here when the file doesn't exist:

https://github.com/pandas-dev/pandas/blob/b1b788fefcdc1f46cebbf6a0df4a9f321ca20086/pandas/io/common.py#L692
[Is this a known error](https://github.com/pandas-dev/pandas/runs/6754333579?check_suite_focus=true#step:9:62) in `TestiLocBaseIndependent.test_iloc_setitem_multicolumn_to_datetime`? Looks unrelated to my changes.


cc @mroeschke 
Great, thanks @janosh
EHN: add same value count for skew & kurt
State: closed

REF:
* https://github.com/pandas-dev/pandas/pull/46431#issuecomment-1092391777
* https://github.com/pandas-dev/pandas/pull/46431#issuecomment-1093798927

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Window
description: rolling, ewma, expanding

Comments:
@auderson can you merge master here
merged
There appears to be a bug in the main branch.
### Update
@jreback  All passed finally！
Thanks again @auderson! (Failure unrelated)
> Thanks again @auderson! (Failure unrelated)

My pleasure
ENH: Same val counts for roll sum mean
State: closed

REF: https://github.com/pandas-dev/pandas/pull/46431#issuecomment-1092391777

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.



Labels:
name: Window
description: rolling, ewma, expanding

Comments:
Hello @auderson! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-04-26 07:55:36 UTC
thanks @auderson very nice!
DOC/CLN: pd.test
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).



Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
hypothesis is imported unconditionally within the tests; won't that make pytest collection fail if it's not available? Similarly, pytest is still listed as a required dependency, why do optional import?
> hypothesis is imported unconditionally within the tests; won't that make pytest collection fail if it's not available?

I believe so, yes

> pytest is still listed as a required dependency, why do optional import?

I don't think it's listed as a required dependency to install pandas (not that I can find), but it is required to run `pd.test` so I think that makes pytest an optional dependency. 
yep this looks right.
DOC: sample variance -> population variance (#46482)
State: closed

- [x] closes #46482 (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
@[mroeschke](https://github.com/mroeschke) sorry i dont understand what you mean.... you mean another pull request?


> @[mroeschke](https://github.com/mroeschke) sorry i dont understand what you mean.... you mean another pull request?

no what you did is fine
thanks @kwhkim
DOC: removed .value_counts() and .nunique() from the docs for 'table …
State: closed

…methods available only for SeriesGroupBy' objects. GH46670

- [x] closes #46670 (Replace xxxx with the Github issue number)
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Labels:
name: Docs

Comments:
thanks @TarunRKaushik