BrandonFors/pandas-github-issues huggingface dataset:
REGR: groupby.sem with nuisance columns
State: closed

- [x] closes #38774
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

We were using the same integer locs in the result of `std` and `count` when computing the quotient for `sem`. However, `std` will drop nuisance columns and `count` will not.

I could only find two tests involving nuisance columns, and these only test sum/mean. Created #38815 for a followup.

Labels:
name: Groupby
name: Regression
description: Functionality that used to work in a prior pandas version
name: Nuisance Columns
description: Identifying/Dropping nuisance columns in reductions, groupby.add, DataFrame.apply

Comments:
Failure is unrelated.

    FAILED pandas/tests/tseries/offsets/test_offsets_properties.py::test_shift_across_dst
    = 1 failed, 96932 passed, 4400 skipped, 1090 xfailed, 63 xpassed, 82 warnings in 1150.49s (0:19:10) =

thanks @rhshadrach 
@meeseeksdev backport 1.2.x
CLN: Add typing for dtype argument in io directory (GH38808)
State: closed

incremental PR for issue #38808
- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
thanks @avinashpancham
DOC: fix sphinx directive error in 1.2.1 release notes
State: closed


Labels:
name: Docs

Comments:
Thanks!
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.2.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 812c3012cf3d3d027209798c3c3beeb7135807b8
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #38806: DOC: fix sphinx directive error in 1.2.1 release notes'
```

4. Push to a named branch :

```
git push YOURFORK 1.2.x:auto-backport-of-pr-38806-on-1.2.x
```

5. Create a PR against branch 1.2.x, I would have named this PR:

> "Backport PR #38806 on branch 1.2.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
TST: GH30999 add match=msg to pytest.raises in modules with one simple instance each
State: closed

This pull request partially addresses xref #30999 to remove bare pytest.raises by adding match=msg. It doesn't close that issue as I have only addressed the following three modules:

pandas/tests/dtypes/test_inference.py
pandas/tests/libs/test_hashtable.py
pandas/tests/resample/test_resampler_grouper.py

They are scattered around the test directory but each of them only had one instance of a bare pytest.raises which could be fixed with a simple addition of match=msg or change to tm.external_error_raised

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @moink
TST: GH30999 add match=msg to all pytest.raises in pandas/tests/window
State: closed

This pull request partially addresses xref #30999 to remove bare `pytest.raises` by adding `match=msg`. It doesn't close that issue as I have only addressed test modules in the pandas/tests/window/ directory.

I have added `match=msg` to 3 instances of `pytest.raises` and converted another to a `tm.external_error_raised`. Nothing complicated or controversial, imo.

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @moink
TST: GH30999 Add match=msg to all pytest.raises in pandas/tests/reshape
State: closed

This pull request partially addresses xref #30999 to remove bare pytest.raises by adding match=msg. It doesn't close that issue as I have only addressed test modules in the pandas/tests/reshape/ directory.

This one was super simple. Fixed 5 bare pytest.raises and didn't do anything complicated or controversial.

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Code Style
description: Code style, linting, code_checks

Comments:
Pretty sure the CI failure has nothing to do with the changes:
```
def _assert_caught_no_extra_warnings(
        *,
        caught_warnings: Sequence[warnings.WarningMessage],
        expected_warning: Optional[Union[Type[Warning], bool]],
    ) -> None:
        """Assert that no extra warnings apart from the expected ones are caught."""
        extra_warnings = []
    
        for actual_warning in caught_warnings:
            if _is_unexpected_warning(actual_warning, expected_warning):
                extra_warnings.append(
                    (
                        actual_warning.category.__name__,
                        actual_warning.message,
                        actual_warning.filename,
                        actual_warning.lineno,
                    )
                )
    
        if extra_warnings:
>           raise AssertionError(f"Caused unexpected warning(s): {repr(extra_warnings)}")
E           AssertionError: Caused unexpected warning(s): [('ResourceWarning', ResourceWarning("unclosed <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.1.0.4', 36820), raddr=('169.254.169.254', 80)>"), '/home/vsts/work/1/s/pandas/io/parsers.py', 2952)]
```
thanks @moink
TST: Add hook for Disallow bare pytest.raises
State: closed

xref #30999

Adding a hook with lots of excluded folders (for now), this can be made stricter as we go along, there's not that much left

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
looks reasonable, can you rebase once more
thanks @MarcoGorelli 
> We are now down to just extension and can merge this, and imo close #30999

@moink sorry I missed this comment - so, computation and io are covered now? If so, do you want to make a PR to remove them from the `exclude` section of this hook and write `closes #30999` in the PR body?
> > We are now down to just extension and can merge this, and imo close #30999
> 
> @moink sorry I missed this comment - so, computation and io are covered now? If so, do you want to make a PR to remove them from the `exclude` section of this hook and write `closes #30999` in the PR body?

#38973
BUG: GH38672 SeriesGroupBy.value_counts for categorical
State: closed

Unobserved categories in Series were being dropped in value_counts, which was inconsistent with Series.value_counts

- [x] closes #38672
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Groupby
name: Categorical
description: Categorical Data Type

Comments:
also pls add a whatsnew note to 1.3, bug fixes under categorical is fine.

so i guess this was just not tested before.
@jreback

> also pls add a whatsnew note to 1.3, bug fixes under categorical is fine.

No worries.  I think it might be more consistent with past whatsnew files to place it under Groupby/resample/rolling though.

Happy to do what I'm told though 😄 

thanks @venaturum 
might resolve https://github.com/pandas-dev/pandas/issues/32471 or https://github.com/pandas-dev/pandas/issues/25970
BUG: DataFrame(dt64data, dtype=td64) corner cases
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
what is the user facing change?
> what is the user facing change?

1)

```
dt = np.datetime64(4, "ns")
dtype = np.dtype("m8[ns]")

pd.DataFrame({"A": [dt, dt]}, dtype=dtype)  # <-- fails to raise on master (though it doesn't cast to td64, so maybe OK?)
```

2)

```
from datetime import datetime

scalar = datetime(9999, 1, 1)
pd.DataFrame(scalar, index=[0], columns=[0])  # <-- raises OutOfBoundsDatetime on master
```



kk change looks good, i think need a whatsnew note though as this is now working
whatsnew added + green
BUG: Fix precise_xstrtod segfault on long exponent
State: closed

- [x] closes #38753
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

@snowman2 can you check if this branch returns behavior to what you had before 1.2?

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: IO CSV
description: read_csv, to_csv
name: Segfault
description: Non-Recoverable Error

Comments:
cc @gfyoung 
> can you check if this branch returns behavior to what you had before 1.2?

`pandas.read_csv` successfully read in the CSV file with this branch without issue.
thanks @mzeitlin11 very nice, keep em coming!
@meeseeksdev backport 1.2.x
Backport PR #38737 on branch 1.2.x (BUG/REG: RollingGroupby MultiIndex levels dropped)
State: closed

Backport PR #38737: BUG/REG: RollingGroupby MultiIndex levels dropped

Labels:
name: Groupby
name: Window
description: rolling, ewma, expanding

Comments:
> Personally, I think we should first discuss this further before merging into 1.2.x, see [#38737 (comment)](https://github.com/pandas-dev/pandas/pull/38737#issuecomment-752205894)

ok. I actually think we *should* change this. I am also ok doing it for 1.2.x, but conversatively it might be better in 1.3
To be clear I am fine with doing this for 1.2.x, *if* we know this is the behaviour we actually want. But so we should probably first discuss, and based on that can still merge into 1.2.x.
Started a new discussion in https://github.com/pandas-dev/pandas/issues/38787.

Regardless, isn't it okay to still fix this perceived regression in 1.2.x and then possibly have a formal API behavior change in 1.3.0?
So we changed behaviour between 1.0.5 and 1.1.5, which is now also in 1.2.0 (for certain cases). So it's actually already a regression in 1.1.5, I think, and not only 1.2.0?

*Assume* that we decide that we actually want to keep the behaviour from 1.1.5/1.2.0 long-term, then I don't think we should revert back to the 1.0.5 behaviour in 1.2.1, to then change again in 1.3.0 to the behaviour of 1.1.5/1.2.0 (but then we could rather say: "OK, we accidentally changed behaviour in 1.1.5, but we actually think it is better, so let's now leave it as is"). 

That's a big *if* on what behaviour we want long term, and it might well be that we also long term want to change back to the 1.0.5 behavior. In which case this PR should certainly be merged for 1.2.x. But I would still have the discussion first to avoid the potential back and forth change in that example scenario above.
@jorisvandenbossche not merging a backport is not ideal as causes the releases notes to be out of sync and the potential for merge conflicts preventing auto backports

I think we should consider one of the following as a preference to leaving this in limbo 

1. merging the backport, and can then maybe revert before release if necessary
2. revert the PR to master, reopen, and only merge when ready
3. close this PR, move the release notes on master, and then do a manual backport when ready. 
added the blocker tag to #38787. so I think the best option is now to backport and revert if necessary.

will merge this shortly is no objections.
yep let's do it
BUG: inspect.getmembers(Series)
State: closed

Make `inspect.getmembers(Series)` work, previously raised an `AbstractMethodError`.

xref: #38740



Labels:
name: Bug
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
Note there was quite some discussion about this on https://github.com/pandas-dev/pandas/pull/31549 (it is also reported as a bug in python itself), and in the and this was fixed for `_constructor_expanddim` differently in https://github.com/pandas-dev/pandas/pull/33628

I don't care much about *how* it is folved, but probably good to be consistent for `_constructor_expanddim` and `_constructor_sliced` ?

> note we backported the other issue. ok for this too.

I would just leave it for 1.3 (it's not a regression like the other issue). You never know how some subclass might be using this method (or relying on the exact error class)


> I would just leave it for 1.3 (it's not a regression like the other issue). You never know how some subclass might be using this method (or relying on the exact error class)

makes sense.
> I don't care much about *how* it is folved, but probably good to be consistent for `_constructor_expanddim` and `_constructor_sliced` ?

I've made a new version, where they're the same: `_constructor_sliced`  is only for `DataFrame` (it doesn't make sense for `Series`) and `_constructor_expanddim` is only available on `Series` (doesn't make sense for `DataFrame`, now that NDFrame higher than 2 is gone).

Ping.
thanks @topper-123
CI,STYLE: add spell check?
State: closed

IIRC it was suggested in a comment some time ago to use a spell checker, so I thought I'd make a little PR showing how `codespell` would work (for now just applied to `pandas/core`).

Is this something we'd want? False positives can be ignored in `setup.cfg` in the `[codespell]` section

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
> i think we had this before but removed it

I just ran `git log -G codespell` and got nothing (other than this PR)

I did however find (with `git log -G spell`) #24299, which removed `sphinxcontrb.spelling`. That's a different tool - this one (codespell) is very fast and takes ~0.7 seconds on the whole codebase from my (non-exceptional) laptop
> > i think we had this before but removed it
> 
> I just ran `git log -G codespell` and got nothing (other than this PR)
> 
> I did however find (with `git log -G spell`) #24299, which removed `sphinxcontrb.spelling`. That's a different tool - this one (codespell) is very fast and takes ~0.7 seconds on the whole codebase from my (non-exceptional) laptop

ahh right that was for the docs. okie this looks fine to me (we *may* want to even clean up the exclude list at some point as well, e.g. narrow it) but that's for other isssues.
merge on green
BUG: Added test cases to check loc on multiindex with NaNs #29751
State: closed

- [x] closes #29751 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Added test cases to check loc on multiindex containing NaN values using `np.nan`, `pd.NA`, and `None`

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: MultiIndex

Comments:
thanks @kasim95
DOC: create shared includes for comparison docs
State: closed

This will help ensure consistency between the examples.

Similar to https://github.com/pandas-dev/pandas/pull/38735, but focusing on shared content in a different folder. Particularly interested to see if the build fails in the same way.

- [ ] ~~closes #xxxx~~
- [ ] tests added / passed
- [ ] ~~passes `black pandas`~~
- [ ] ~~passes `git diff upstream/master -u -- "*.py" | flake8 --diff`~~
- [ ] ~~whatsnew entry~~


Labels:
name: Docs

Comments:
Heh, this pull request is [failing differently](https://github.com/pandas-dev/pandas/pull/38771/checks?check_run_id=1620282550#step:4:67) than https://github.com/pandas-dev/pandas/pull/38735: `flake8-rst` is giving `F821 undefined name 'tips'`. I tried the `:flake8-add-ignore: F821` [Custom Role](https://github.com/kataev/flake8-rst#custom-roles), but that didn't work.

Sigh. Though I planned to use these includes for [the Excel page](https://pandas.pydata.org/pandas-docs/dev/), maybe not worth the trouble.

Thoughts?
> Heh, this pull request is failing differently than #38735: flake8-rst is giving F821 undefined name 'tips'. I tried the :flake8-add-ignore: F821 Custom Role, but that didn't work.

Do you have the commit that tried this? Did you add it to the list of extensions in conf.py?
Pushed it up in https://github.com/pandas-dev/pandas/pull/38837.
Closing in favor of https://github.com/pandas-dev/pandas/pull/38887.
BLD: fix build failure py3.9.1 on OSX
State: closed

Ran `brew upgrade` yesterday and ended up bumping from 3.9.0 to 3.9.1, after which I got build failures 

```
    LooseVersion(python_target) < "10.9"
  File "/usr/local/Cellar/python@3.9/3.9.1_2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/version.py", line 306, in __init__
    self.parse(vstring)
  File "/usr/local/Cellar/python@3.9/3.9.1_2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/version.py", line 314, in parse
    components = [x for x in self.component_re.split(vstring)
TypeError: expected string or bytes-like object
```

No idea why this just started breaking now.



Labels:
name: Build
description: Library building on various platforms

Comments:
can you add a whatsnew note (1.2.1) as should backport this
whatsnew added + green
@meeseeksdev backport 1.2.x
TYP: DataFrame.(dot, __matmul__)
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

xref https://github.com/pandas-dev/pandas/pull/38416/files#r549438292

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
cc @simonjayhawkins This currently fails with mypy for reasons I haven't figured out:
```
pandas/core/frame.py:1162: error: Overloaded function signatures 1 and 2 overlap with incompatible return types  [misc]
pandas/core/frame.py:1280: error: Overloaded function signatures 1 and 2 overlap with incompatible return types  [misc]
```
but I believe what I have is the correct logic for these methods:
- if `other` is `Series` we return Series
- if `other` is a `DataFrame`, `np.ndarray`, `ExtensionArray` or `Index` we return a `DataFrame` 
- otherwise we throw
> cc @simonjayhawkins This currently fails with mypy for reasons I haven't figured out:

from experience, `error: Overloaded function signatures 1 and 2 overlap with incompatible return types` occur since `np.ndarray` resolves to `Any`. 
Thanks @arw2019 for the PR. when adding overloads also add the types to the actual method.

The overloads are used by mypy when establishing the return type when checking functions with calls to this method. The typing on the actual method is used to check this method.
> > cc @simonjayhawkins This currently fails with mypy for reasons I haven't figured out:
> 
> from experience, `error: Overloaded function signatures 1 and 2 overlap with incompatible return types` occur since `np.ndarray` resolves to `Any`.

Right. Is it right there's no great way to fix that before `NumPy` 1.2? 

Currently I added a `type: ignore` which can be removed once we're checking against `NumPy` 1.2 (and I believe `mypy` will force that because the `type: ignore` will become unused). 

That said, I know `type: ignore` use is an anti-pattern, so alternatively I can just delete the overloads and create an issue in the tracker to be taken up once we've upgraded NumPy.

> Thanks @arw2019 for the PR. when adding overloads also add the types to the actual method.
> 
> The overloads are used by mypy when establishing the return type when checking functions with calls to this method. The typing on the actual method is used to check this method.

Thanks, that was the problem (in addition to the `ndarray` issue)
thanks @arw2019
BUG: Series construction with mismatched dt64 data vs td64 dtype
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Fixing this for DataFrame is much harder, so punting for now.

Labels:
name: Bug
name: Series
description: Series data structure

Comments:
looks fine. this is just a refactor right? doesn't change user facing?
> looks fine. this is just a refactor right? doesn't change user facing?

No, this fixes corner cases failing to raise:

```
dtype = np.dtype("m8[ns]")
val = np.datetime64(1234, "ns")

>>> pd.Series([val], dtype=dtype)

>>> pd.Series({0: val}, dtype=dtype)
```

and vice-versa for flipping tdt64/dt64
kk i guess then a whatsnew note
BUG: float-like string, trailing 0 truncation
State: closed

- [x] closes #38708
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
also pls rebase 
Moved tests, added issue number, fixed conflict
thanks @mzeitlin11 
@meeseeksdev backport 1.2.x
CLN: simplify tm.decompress_file
State: closed

Could re-use `get_handle` but I'm not sure whether the testing code should depend on the to-be tested code?

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
i think this is fine. we have enough tests to ensure that we are actually producing compressed files.
thanks @twoertwein
DOC: chat-ops
State: closed

- [x] closes #38726
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] Added documentation for `@githubactions pre-commit`
- [x] Added documentation for `@meeseeksdev backport`


Labels:
name: Docs

Comments:
@githubactions pre-commit
I'll make these changes and commit ASAP
> ASAP

:rofl: thanks, but there's no hurry
thanks @1nF0rmed 

if any more comments @rhshadrach @MarcoGorelli 
thanks @1nF0rmed
Iloc truncates single-column dataframe with extension arrays
State: closed

Added a test to ensure .iloc does not truncate data for a single-column dataframe with an extension array, as discussed in GH30263.

- [ ] closes #30263
- [x ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Stale

Comments:
Hello @luijkr! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-12-28 14:49:04 UTC
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
closing as stale. if you want to continue, pls ping and can re-open.