BrandonFors/pandas-github-issues huggingface dataset:
REGR: isin raising TypeError for RangeIndex
State: closed

- [x] closes #41151
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

This was easier than I've expected

cc @attack68 

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Index
description: Related to the Index class or subclasses
name: Upstream issue
description: Issue related to pandas dependency

Comments:
Regression on Master, so no note 
thanks @phofl
TST: Added iloc tests for casting from object to numeric dtypes for duplicate columns
State: closed

- [x] closes #22035
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
Hello @isaac-chung! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-04-25 22:57:11 UTC
You could install pre-commit to automatically run the linting checks before committing
I assume an extended test would not fall into the "whatsnew entry", so checking it off for completeness. Is there anything I can do about the failed checks?
> I assume an extended test would not fall into the "whatsnew entry", so checking it off for completeness. Is there anything I can do about the failed checks?

no
looks fine ping on green.
Thanks @isaac-chung! The CI failure was unrelated
BUG: retain ordered Categorical dtype in SeriesGroupBy aggregations
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

ATM we correctly wrap them for DataFrameGroupBy but not for SeriesGroupBy.  Doing this correctly for SeriesGroupBy has the benefit of avoiding the need to special-case it in DataFrameGroupBy


Labels:
name: Bug
name: Groupby
name: Categorical
description: Categorical Data Type

Comments:
@jbrockmendel lgtm. was there a precursor to merge first?
> lgtm. was there a precursor to merge first?

already got #41111
BUG: raise ValueError when sep and delimiter are defined in read_csv
State: closed

- [x] closes #39823
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO CSV
description: read_csv, to_csv

Comments:
cc @gfyoung if any comments
@phofl can you rebase and will look again
merged master
thanks @phofl
BUG: isin fails to correctly compare None values.
State: closed

- [x] closes #35565 
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Regression
description: Functionality that used to work in a prior pandas version
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
@DriesSchaumont can you respond to jreback comments
Hi @jbrockmendel, I am in the middle of moving to a new place and I currently have limited internet access. I will try to come back to this PR as soon as possible. I hope this is not blocking anything?  
> I hope this is not blocking anything?

no, not to worry
Seems I accidentally pushed a commit that was meant to be in another branch. Updating this PR now. I will add a comment when this is ready for review.
if possible we should reuse libmissing.is_matching_na
closing as stale, if you want to continue working, please ping.
BUG: Fix series clipping NA issue
State: closed

Series clipping method could not handle series with NA values.
Issue was fixed by first comparing values and only after then converted
to numpy array. Now the test code provided with the issue ticket provides an expected result.

- [x] closes #40581
- [x] tests added / passed
 - I did run test_fast.sh and 3 test failed but I think they are not related to my changes
   -  test_can_set_locale_valid_set 
   -  test_header_multi_index_blank_line[c_high]
   - test_header_multi_index_blank_line[c_high]
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry



Labels:
name: Bug
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Hello @Rasori! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-04-28 18:17:50 UTC
The test fail is not related to my changes.

E: I re-pushed the changes to see if the check passes this time. At least checks of other PRs are passing ...
thanks @Rasori 
This seemed to cause a performance regression. https://pandas.pydata.org/speed/pandas/index.html#series_methods.Clip.time_clip?python=3.8&Cython=0.29.21&p-n=50&commits=29512dc9ea2b066ab5f4a7cb80c1f8f5becef1c2
@TomAugspurger if u wouldn't mind creating a new issue about this perf regression
BUG: groupby sum, mean, var should always be floats
State: closed

- [x] closes #41137
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Built on top of #40790

Labels:
name: Bug
name: Groupby
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
looks good. can you add a whatsnew note. might consider a sub-section as this is a 'bigger' bug fix but either way.
@jreback Thanks - whatsnew section added.
thanks @rhshadrach
add type hint for 'data' argument in DataFrame's from_dict method: Un…
State: closed

…ion[dict, list]

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@GrossBetruger Thanks for the PR. closing as stale. ping if you want to continue.
BUG: pd.factorize should  not upconvert unique values unnecessarily
State: closed

The uniques returned from `pd.factorize` should keep their original dtype, where possible.

```python
>>> arr =np.array([1,2], dtype="int8")
>>> pd.factorize(arr)
(array([0, 1], dtype=int64), array([1, 2], dtype=int64))  # master
(array([0, 1], dtype=int64), array([1, 2], dtype=int8))  # this PR
```


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
thanks @topper-123
REF: mix NDArrayBacked into Categorical, PandasArray
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
does this have any perf implications?

pls rebase
> does this have any perf implications?

```
In [3]: cat = pd.Categorical(range(5))

In [4]: %timeit cat.copy()
315 ns ± 5.19 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  <-- PR
802 ns ± 7.08 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  <-- master
```
Rebased + green
great!
Deprecated usecols with out of bounds indices in read_csv with c engine
State: closed

- [x] closes #25623
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Deprecation for c parser

Labels:
name: IO CSV
description: read_csv, to_csv
name: Deprecate
description: Functionality to remove in pandas

Comments:
merged with #41130
TYP: resample
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @jbrockmendel
PERF/REF: Check use of numexpr earlier in the DataFrame operation
State: closed

This PR tries to gather some checks about whether numexpr can be used or not which can be done upfront on the DataFrame level, before dispatching to the array_ops. 
For example, for a scalar right operand, we can check it only once if it's compatible with numexpr usage. Or for ArrayManager, we can check the size constraint once upfront (since each array has the same size). For those cases, this avoids overhead in `expressions.evaluate`. 

To this end, I created a new `can_use_numexpr` that starts to gather some checks (there might be more that can be added), call this up front in `DataFrame._dispatch_frame_op` (this could IMO be bit cleaner with https://github.com/pandas-dev/pandas/pull/39772, but the if/else checks inside `_dispatch_frame_op` work as well). And then the result of `can_use_numexpr` is passed through to the array_ops. 

Labels:
name: Refactor
description: Internal refactoring of code
name: Performance
description: Memory or execution speed performance
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Stale

Comments:
There failing tests related to warning/error not being raised for boolean arithmetic ops, but (more easily) fixing those tests depends on https://github.com/pandas-dev/pandas/pull/41161. 
And I would also first like to see https://github.com/pandas-dev/pandas/pull/40463 merged for better test coverage.
any estimates for perf impact?
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
> There failing tests related to warning/error not being raised for boolean arithmetic ops, but (more easily) fixing those tests depends on #41161.
> And I would also first like to see #40463 merged for better test coverage.

#41161 and #40463 have been merged. can you address @jbrockmendel comments.
Thanks for the PR but appears to have gone stale. Closing to clear the queue, but feel free to reopen if you find the time to revisit this PR and the comments.
CLN: tests.indexes.test_setops.py
State: closed

I'm experimenting with subclassing NumericIndex and using the test suite to check if everything passes ok (by adding the new subclass to `indices_dict` in `conftest.py` and making a `makeXIndex` func in `_testing.py.

`COMPATIBLE_INCONSISTENT_PAIRS ` in `test_setops.py` is very tied to specific index classes, so is difficult to use on new index classes. So here's a PR to make this part more friendly to testing subclasses by making it test dtypes instead of index class.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Index
description: Related to the Index class or subclasses

Comments:
thanks @topper-123
Deprecate inplace in Categorical.add_categories.
State: closed

- [x] xref #37643
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Labels:
name: Categorical
description: Categorical Data Type
name: Deprecate
description: Functionality to remove in pandas

Comments:
thanks @OlehKSS
REF: move dispatch methods to WrappedCythonOp
State: closed

This gets us most of the way towards having everything in the correct class.

The sticking point that makes this not-a-slam-dunk is that a [check](https://github.com/pandas-dev/pandas/compare/master...jbrockmendel:ref-gbop-wrapped?expand=1#diff-17420aa54e4b74826eaaeedc6ef6b320c16691aa74c09420e6098854284b4874L833-L835) is removed:

```
            if self._filter_empty_groups and not counts.all():
                assert result.ndim != 2
                result = result[counts > 0]
```

This condition never holds in the tests, and the assertion would fail if it ever did.  I don't have a compelling reason to remove this check other than it being inconvenient to move it to WrappedCythonOp.

Labels:
name: Refactor
description: Internal refactoring of code
name: Groupby

Comments:
lgtm can you merge master
Rebased + green
Ci: start testing py310-dev
State: closed

This might not work now. Let's see.


Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies
name: Python 3.10

Comments:
nice,ping when ready (also happy to merge if we can xfail this build while in-progress)
pytest error due to https://github.com/pytest-dev/pytest/issues/8539, which is fixed on master.
looks great. can you rename this workflow similar to our others but just add dev e.g. actions-310-dev or similar. ping on green.
testing will timeout if using the xdist or cov plugin. not sure why.

```
============================= test session starts ==============================
platform linux -- Python 3.10.0b1, pytest-6.3.0.dev395+g1e2c2d972, py-1.10.0, pluggy-0.13.1
rootdir: /home/runner/work/pandas/pandas, configfile: pyproject.toml, testpaths: pandas
plugins: cov-2.11.1, forked-1.3.0, xdist-2.2.1, hypothesis-6.10.1
gw0 I / gw1 I
gw0 [160069] / gw1 [160069]

127.0.0.1 - - [27/Apr/2021 01:17:56] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [27/Apr/2021 01:17:56] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [27/Apr/2021 01:17:56] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [27/Apr/2021 01:17:56] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [27/Apr/2021 01:17:57] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [27/Apr/2021 01:17:57] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [27/Apr/2021 01:17:57] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [27/Apr/2021 01:17:57] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [27/Apr/2021 01:17:57] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [27/Apr/2021 01:17:57] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [27/Apr/2021 01:17:57] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [27/Apr/2021 01:17:58] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [27/Apr/2021 01:17:58] "GET / HTTP/1.1" 200 -
Error: The operation was canceled.
```
If we use coverage directly, the test would start running. But there were some errors in json tests and it freeze there.

```
pandas/tests/io/json/test_compression.py ......EEEEE....................ssss.........
pandas/tests/io/json/test_deprecated_kwargs.py ..
pandas/tests/io/json/test_json_table_schema.py .............................................................................................x...x...x...x...x...x...x...x...x...x.......................
pandas/tests/io/json/test_normalize.py ................................................
Error: The operation was canceled.
```
thanks @fangchenli very nice!

can you open some issues about the failures (and tag with 3.10, going to create it now). thanks!
BUG: DataFrameGroupBy agg with multi-column object block
State: closed

This is analogous to what we did recently-ish with DataFrame._reduce.  When we see an object block, split it and operate column-wise.  Side-bonus: this makes the BM behavior match the AM behavior.

Labels:
name: Bug
name: Groupby
name: Nuisance Columns
description: Identifying/Dropping nuisance columns in reductions, groupby.add, DataFrame.apply

Comments:
is this a user facing bug ? can you add a whatsnew
whatsnew added + green
gentle ping; will need to rebase #41147 after this
BUG: Check for null values when infering excel in read_clipboard
State: closed

GH41108
Stripping whitespace on read_clipboard causes data copied from
excel to lose the tab separator between the first column and
the nexts, shifting the data into the wrong column

This just removes the .lstrip() except for the first line
containing headers, to avoid breaking header-less index in csv
files

- [x] closes #41108
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: IO Data
description: IO issues that don't fit into a more specific label
name: IO Excel
description: read_excel, to_excel

Comments:
Hello @saucoide! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-05-22 17:07:04 UTC
Please add whatsnew and fix linting issue
Thanks - looks good! Can you resolve the merge conflict.
thanks @saucoide
BUG: read_json not reading in large ints properly 
State: closed

- [x] closes #20608
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
Casting to int64 / float64 is unfortunately not going to work generically here. I think @arw2019 was closer to a solution in #34984 - maybe something you can pick up?
OK, closing then. Python C API stuff is a little bit over my head.
BUG: rolling mean and sum not numerical stable for all nan window
State: closed

- [x] closes #41053
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

The problem is not numerical stable, so we are dividing -3e-18 / 0, which returns -inf running into line 177 setting to 0. Checking for nobs > 0 would solve these cases

cc @mroeschke 

Labels:
name: Window
description: rolling, ewma, expanding

Comments:
cc @mroeschke 
I've added the sum fix mentioned in the issue, the test returns

```
              0
0  2.000000e-03
1  1.000000e-02
2  1.500000e-02
3  1.300000e-02
4  5.000000e-03
5 -8.673617e-19
```

on master. Sum should be the only other case we have the problem. As long as all nan should return 0 we can do it this way
thanks @phofl
BUG: read_csv/table raising when sys.setprofile used
State: closed

- [x] closes #41069
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

According to https://docs.python.org/3/library/functions.html#locals, `locals()` should not be modified. AFAICT these are the only places `locals()` is used in this way without copying.

Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
good catch.
thanks @mzeitlin11 
With sys.setprofile, read_csv breaks when the data frame is saved using a different separator such as (sep='µ')

df.to_csv('df.csv', index=False, sep='µ')
pd.read_csv('df.csv', sep='µ', engine='python') #setprofile breaks this. separator not recognized.
CLN: avoid catching AttributeErorr in DataFrameGroupBy.aggregate
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Groupby
name: Clean
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
@jreback gentle ping for several groupby PRs; more in the pipeline and id like to avoid rebase trouble
BUG: any/all not returning booleans for object type
State: closed

- [x] closes #12863, closes #27709, closes #35450
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Also allows removing some of the any/all specific boolean conversion logic in frame reduction code, which (hopefully) also fixes part of #41079.

Labels:
name: Bug
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
thanks @mzeitlin11
CLN: remove split_and_operate in Block.fillna
State: closed

I didn't see any tests failing locally, so checking here with full CI.

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Clean

Comments:
@jbrockmendel do you have an idea of what case would need the `split_and_operate` for `Block.fillna`? Because if it's needed, it's clearly not covered by the tests. 



> do you have an idea of what case would need the split_and_operate for Block.fillna? Because if it's needed, it's clearly not covered by the tests.

None comes to mind.  Best guess is that for object dtype cases a splitting needs to occur at somewhere in the call stack, and in this case it happens later on even if it doesn't happen here.

Happy to see this go away!
lgtm. we have *nothing* hit that codepath?
Unfortunately, I just thought of a case:

```
df = pd.DataFrame({'a': [0.1, 0.1, np.nan], 'b': [0.1, 0.2, 0.3]})
df.fillna("string")
```

When splitting, this will only convert the column with missing values to object. But with this PR, it will convert the full block to object dtype to then infer and convert back to numeric the columns that didn't have missing values. So the splitting actually does serve some purpose here (but the end result is the same in both cases, so no tests failed).

should make a test case out of this?
The end result is the same, so I don't think we can test it with a normal test (a benchmark might catch it)
im not sure i understand.  isnt the `df.fillna("string")` case one that works in master and fails under this PR?  (and if not, why would finding it be a reason to close this PR?)
No, it doesn't fail here, because we will convert to object dtype but then afterwards infer again to float (for columns that didn't have any NaN). So as mentioned above, the *end result* is the same, but it would be achieved with an additional (unnecessary) conversion float -> object -> infer as float.