BrandonFors/pandas-github-issues huggingface dataset:
PERF: numexpr doesn't support floordiv, so don't try 
State: closed

See overview of supported operators: https://numexpr.readthedocs.io/projects/NumExpr3/en/latest/user_guide.html#supported-operators. By trying to use it, we were falling back to the (slower) `_masked_arith_op`

Discovered by investigating the arithmetic benchmarks at https://github.com/pandas-dev/pandas/issues/39146#issuecomment-799482699

For example using the `arithmetic.IntFrameWithScalar.time_frame_op_with_scalar(<class 'numpy.float64'>, 3.0, <built-in function floordiv>)` benchmark:

```python
import operator

dtype = np.float64
arr = np.random.randn(20000, 100)
df = pd.DataFrame(arr.astype(dtype))
scalar = 3.0
op = operator.floordiv
```

```
In [2]: %timeit op(df, scalar)
56.4 ms ± 723 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  <-- master
24.3 ms ± 236 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  <-- PR
```


Labels:
name: Performance
description: Memory or execution speed performance
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
There is still a failure in `test_expressions.py` to fix
@jreback Looks like this was red when merged and should be reverted. 

e.g. 

```
=================================== FAILURES ===================================
________ TestExpressions.test_bool_ops_raise_on_arithmetic[//-floordiv] ________
[gw1] linux -- Python 3.7.1 /usr/share/miniconda/envs/pandas-dev/bin/python

self = <pandas.tests.test_expressions.TestExpressions object at 0x7fc5e381b630>
op_str = '//', opname = 'floordiv'

    @pytest.mark.parametrize(
        "op_str,opname", [("/", "truediv"), ("//", "floordiv"), ("**", "pow")]
    )
    def test_bool_ops_raise_on_arithmetic(self, op_str, opname):
        df = DataFrame({"a": np.random.rand(10) > 0.5, "b": np.random.rand(10) > 0.5})
    
        msg = f"operator {repr(op_str)} not implemented for bool dtypes"
        f = getattr(operator, opname)
        err_msg = re.escape(msg)
    
        with pytest.raises(NotImplementedError, match=err_msg):
>           f(df, df)
E           Failed: DID NOT RAISE <class 'NotImplementedError'>
```
from https://github.com/pandas-dev/pandas/runs/2245866400
whoops

hard to tell sometimes

ok
reverted https://github.com/pandas-dev/pandas/pull/40742
There was even a comment that I still needed to fix the test .. ;)
BUG: Do not allow boolean values for indexing with .loc without a boolean index.
State: closed

- [x] closes #20432
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Please add whatsnew 
> Please add whatsnew

Added.
CI failure seems related
> CI failure seems related

Yes, indeed. I could fix most of the failing tests with specifying the dtypes. But for some reason specifying dtype="boolean" does not remove the warning. This is a separate issue though.
> > CI failure seems related
> 
> Yes, indeed. I could fix most of the failing tests with specifying the dtypes. But for some reason specifying dtype="boolean" does not remove the warning. This is a separate issue though.

I must have been tired. dtype="boolean" was not passed to the df or series, but to the index. Fixed and expect CI to be green.
Small comments, otherwise lgtm, pls merge master too
Hello @DriesSchaumont! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-04-06 15:29:28 UTC
lgtm cc @jreback @jbrockmendel 
@DriesSchaumont did you post the updated asv's? 
https://github.com/pandas-dev/pandas/pull/40726#discussion_r606390899

Here 
@phofl pls merge!
thanks @DriesSchaumont
ENH: MatplotlibDeprecationWarning: is_first_col function was deprecated
State: closed

Fix for [#40714](https://github.com/pandas-dev/pandas/issues/40714)

The is_first_col function was deprecated in Matplotlib 3.4 and will be removed two minor releases later. Used ax.get_subplotspec().is_first_col() instead.




Labels:
name: Visualization
description: plotting
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Deprecate
description: Functionality to remove in pandas

Comments:
closing in favor of #40718 
thanks @arnaujc91 this is a bit tricky as have multiple versions of mpl to support
BUG: fix comparison of NaT with numpy array
State: closed

Closes #40722


Labels:
name: Datetime
description: Datetime data dtype
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Regression
description: Functionality that used to work in a prior pandas version
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
Verified that the failing builds are matplotlib related
COMPAT: matplotlib 3.4.0
State: closed

- [ ] closes #40714
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
marking this for backport as well. @jbrockmendel ping when ready .
im kinda burned out on fixing-the-ci; can someone else pick up the baton on this?
> im kinda burned out on fixing-the-ci; can someone else pick up the baton on this?

I could pick this up sometime later today
ping @pandas-dev/pandas-core for merge. Thanks everyone!
thanks all!
@meeseeksdev backport 1.2.x
Hi, I just ran into this issue when running my matplotlib project based on pandas-input. Thanks for working on it. Please note, that matplotlib is now at 3.4.1 (3.4.0 was quickly superseded).

Hth,
Rob
REF: RangeIndex dont subclass Int64Index
State: closed

This makes it much simpler if we want to put constructors in cython.

Labels:
name: Refactor
description: Internal refactoring of code
name: Index
description: Related to the Index class or subclasses
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
Mypy problem @jbrockmendel 
nice
TYP: groupby.pyi
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Groupby
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@jbrockmendel you may have answered this, but how do we tests / keep these up to date?
> but how do we tests / keep these up to date?

@simonjayhawkins has a branch (though i dont see a PR) that implements tests, but im not sure exactly how that works.  as for keeping it up to date: manually
> @simonjayhawkins has a branch (though i dont see a PR) that implements tests

#40202 - closed until after arrow string array is further progressed.

> as for keeping it up to date: manually

related issue #28133
CLN: follow-up cleanups
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Clean

Comments:
lgtm ping on green
ping
conflict :-< ping on green.
ping
DOC: Add/Update Pandas-Genomics in Ecosystem files
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
thanks @jrm5100
REF: combine Block _can_hold_element methods
State: closed

which in turn lets us get rid of ObjectValuesExtensionBlock and HybridMixin.

Labels:
name: Refactor
description: Internal refactoring of code
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
Booyah!  #40614 gets rid of two more Block subclasses, then #40149 gets us within spitting distance of another after that
ENH: [ArrowStringArray] Enable the string methods for the arrow-backed StringArray
State: closed

from https://github.com/pandas-dev/pandas/issues/35169#issuecomment-712883859

> Enable the string methods for the arrow-backed StringArray. This might also need some additional changes in the string accessor code (eg to dispatch extract to the underlying array as well)



Labels:
name: Enhancement
name: Strings
description: String extension data type and string data

Comments:
@simonjayhawkins is this still draft? (it seems ready to me)
this one goes in after #40725, https://github.com/pandas-dev/pandas/pull/40708#discussion_r605101764, over 80% complete on the benchmarks.
To be able to move forward with the different PRs, I think it is perfectly fine to merge this with the ` "arrow_string": "string",` in infer_dtype (which *is* the name of the dtype on current master), and then this can be updated when the name gets changed or in #40725
@simonjayhawkins this is ready for review/merge now?
hopefully, will mark as ready for review once ci is green.
@jorisvandenbossche green
@jorisvandenbossche green. ok to merge?
Yep, thanks!
CLN: add property to check if NDFrame uses ArrayManager
State: closed

xref https://github.com/pandas-dev/pandas/pull/40482/files#r596222938

cc @jbrockmendel 

Labels:
name: Clean

Comments:
@jbrockmendel the property makes it more difficult for type checking, so not sure it's worth the change then for the few occurences
> the property makes it more difficult for type checking

ok, can revisit if the number of occurrences increases
BUG: RollingGroupby MultiIndex levels dropped
State: closed

- [x] closes #38523 
- [x] closes #38787
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Essentially https://github.com/pandas-dev/pandas/pull/38737 but targeted for 1.3 as it appears that the current behavior is largely unexpected

Labels:
name: Bug
name: MultiIndex
name: Window
description: rolling, ewma, expanding

Comments:
hmm should we target 1.2.4? e.g. how does this compare to that behavior (and how does that compare to 1.1.x)?
It looks like you mentioned before this would be okay for 1.3 and we reverted this before for 1.2.x

This gets us back to 1.1.3 behavior which was correct according to this bug report https://github.com/pandas-dev/pandas/issues/38523
> It looks like you mentioned before this would be okay for 1.3 and we reverted this before for 1.2.x
> 
> This gets us back to 1.1.3 behavior which was correct according to this bug report #38523

ok awesome
thanks @mroeschke
CLN: boolean->bool, string->str in docstrings
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs
name: Clean

Comments:
cool

cc @MarcoGorelli might be interesting to have a doc-string checker for 'boolean' and others (may not be easy though)
CI: update exception catching for np1.21
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: CI
description: Continuous Integration
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
@jreback this fixes npdev build (just codecov still failing)
thanks @jbrockmendel 

cc @simonjayhawkins we may need to backport something like this to 1.2.x
CI: unpin pymysql, sync min version, and skip failing test
State: closed

xref #36465



Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
thanks @fangchenli 
hmm

> FAILED pandas/tests/io/test_sql.py::TestMySQLAlchemy::test_to_sql_with_negative_npinf[load_iris_data0-input2]
FAILED pandas/tests/io/test_sql.py::TestMySQLAlchemyConn::test_to_sql_with_negative_npinf[load_iris_data0-input2]

on the database build
When one column is `infe0`, df that contains `np.nan` actually could be converted to SQL table. That's why we don't catch any error.

```python
import pandas
import numpy

from sqlalchemy import create_engine

engine = create_engine('mysql+pymysql://root:@localhost/foobar')

table = "foobar"
pandas.DataFrame({'foo': [-numpy.inf, 3], 'infe0': ['bar', 3]}).to_sql(table, engine, index=False)
import pandas.io.sql as sql
res = sql.read_sql_table(table, engine)
print(res)
```

```bash
   foo infe0
0  NaN   bar
1  3.0     3
```

But that column name must be `infe0`. I'm not sure why it's special.


thanks @fangchenli
REF: de-duplicate pad/backfill code
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
name: Clean

Comments:
lgtm if you can address comment.
updated + greenish
CLN/TST: Some cleanups in apply.test_invalid_arg
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
thanks @rhshadrach
BUG: round_trip_pickle for lib.no_default
State: closed

- [x] closes #40397
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

This will also un-block making a .pyi for lib.pyx

Labels:
name: Bug

Comments:
lgtm.
@jbrockmendel there was no release note here. should we have one?

(the reason I chase you on the release notes is that if regressions are reported, it's easier to justify a behavioral change if the change has been documented)
> the reason I chase you on the release notes

you're right to do so, and i should add them without prompting.  will add a note on my next follow-ups branch
Agg functions for df not respecting numeric only with level
State: closed

- [x] closes #40660
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

stumbled accross this one

Labels:
name: Bug
name: Timedelta
description: Timedelta data type
name: DataFrame
description: DataFrame data structure

Comments:
thanks @phofl
TST: [ArrowStringArray] more parameterised testing - part 1
State: closed

aim to reduce diff to address https://github.com/pandas-dev/pandas/pull/39908#discussion_r585573328, but good to do as follow-up to #35259 anyway.

This also includes a change to inference, which could be broken off if not wanted as part of a TST PR.

more tests to change once fixture in play and also some parameterisation to add for the str accessor testing (which could be a precursor with xfails for ArrowStringArray or combined with PR that adds the accessor.. will open a draft shortly for discussion)

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Strings
description: String extension data type and string data

Comments:
only the tests that test inference seem to break. so we currently have ArrowStringArray not infered as string in master. maybe not used but I'll look in detail and test perf tomorrow.

This will slow down this PR and I want to get the fixture in play so that I can parameterise many other tests that test for only `dtype="string"` so I will remove the inference change and pararmeterisation of the inference test in this PR and do that as a separate PR. (after this PR with the fixture in conftest is merged)
on master

```
>>> pd.__version__
'1.3.0.dev0+1180.gdd697e1cca'
>>> 
>>> from pandas.core.arrays.string_arrow import ArrowStringDtype
>>> 
>>> arr = ArrowStringDtype.construct_array_type()._from_sequence(["B", pd.NA, "A"])
>>> 
>>> arr
<ArrowStringArray>
['B', <NA>, 'A']
Length: 3, dtype: arrow_string
>>> 
>>> pd._libs.lib.infer_dtype(arr)
'unknown-array'
>>> ```
OK have reverted the inference change... so can open a PR on the str accessor and parameterise existing str accessor tests for that, but nbd if this is not ready as can just dup the fixture and resolve conflicts if necessary.
Failures are the matplotlib ones
TST: [ArrowStringArray] remove xfail from test_repr
State: closed

follow-up to #35259, xref https://github.com/pandas-dev/pandas/issues/35169#issuecomment-727217462

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Strings
description: String extension data type and string data

Comments:
thanks @simonjayhawkins 

i think the mypy issue is fixed on master
TST: [ArrowStringArray] add dtype parameterisation to test_astype_float and test_fillna_args
State: closed

#35259 added dtype parameterisation to existing tests in pandas/tests/arrays/string_/test_string.py and xfailed for ArrowStringArray where needed.

test_astype_float and test_fillna_args were later added in #37974 and #37987 respectively.

This PR doesn't fix (although will be looking at the all the xfailed tests in this module shortly, https://github.com/pandas-dev/pandas/issues/35169#issuecomment-727217462) just xfails to reduce the diff (slightly) for #39908

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Strings
description: String extension data type and string data

Comments:
thanks @simonjayhawkins
REF: _cython_operation handle values.ndim==1 case up-front
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code
name: Groupby

Comments:
neutralish on perf?
Yes
This will give a slowdown for ArrayManager, because you are calling itself (`_cython_operation`) again, which means that everything in the beginning of `_cython_operation` is repeated, and thus done twice per column (check for numeric dtype, for disallow_invalid_ops, check for extension arrays -> those are already costly when done repeatedly in wide dataframes).
> and thus done twice per column (check for numeric dtype, for disallow_invalid_ops, check for extension arrays -> those are already costly when done repeatedly in wide dataframes).

could move these to after the 1d check
I am not sure that's possible with the dispatching to `_ea_wrap_cython_operation`, which needs the 1D EA as is, and not reshaped to a 2D array
REF: move most of Block.interpolate to an array method
State: closed

Not _quite_ at the point where it can be called directly from ArrayManager, but close.

Labels:
name: Refactor
description: Internal refactoring of code
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
@jorisvandenbossche have you gotten a chance to look at the Categorical-ArrayManager failure on master?
any issues with AM tests @jbrockmendel ?
> any issues with AM tests

nothing showing up in the CI.  AM doesnt (yet) call this directly
> Not quite at the point where it can be called directly from ArrayManager, but close.

What's needed additionally to have it usable for ArrayManager?
> What's needed additionally to have it usable for ArrayManager?

the short-circuiting on L1113 and L1123 need some special attention (particularly the one on 1123).  Then maybe_downcast at the end needs porting (ive got a branch but am not quite happy with it yet)
can you merge master and ping on green
ping
REF: better use of fused_types for group_ohlc
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@jreback updated to fix the mypy failure on master (newly released numpy version AFAICT) (doesnt fix the new npdev failure)
thanks
DOC: Trivial typo fix in user_guide/window (#40666)
State: closed

Trivial string change: `min_periods` corrected from `min_peridos`

Fixes Issue #40666 

Labels:

Comments:
thanks @mdengler 

for future reference, if you fill in the PR template like

```
- [x] closed #40666
- [ ] ...
```

then the issue will automatically be closed when the PR is merged
> for future reference, if you fill in the PR template like
> 
> ```
> - [x] closed #40666
> - [ ] ...
> ```
> 
> then the issue will automatically be closed when the PR is merged

Thanks -- will do.
BUG: Update Styler.clear method to clear all
State: closed

- [ x ] closes #40484 
- [ x ] tests added / passed
- [ x ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ x ] whatsnew entry updated in docs


Labels:
name: Styler
description: conditional formatting using DataFrame.style

Comments:
Thank you for the review guys! I'll make the change above.

Curious fails in the CI tests yesterday though - was going through some of the logs last night; the errors arose from modules/code lines I didn't touch (pulled from repo :S)
thanks @bsun94 
thank you guys for the review!
BUG: rank treating min int as NaN
State: closed

- [x] closes #32859
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

ASV's look unaffected:
<details>

```
before           after         ratio
     [029907c9]       [b45f9193]
     <master>         <bug/rank_datetimelike>
       6.90±0.2ms       6.76±0.6ms     0.98  frame_methods.Rank.time_rank('float')
       2.63±0.3ms       2.39±0.2ms    ~0.91  frame_methods.Rank.time_rank('int')
         45.7±1ms         45.3±1ms     0.99  frame_methods.Rank.time_rank('object')
       2.47±0.2ms       2.22±0.1ms    ~0.90  frame_methods.Rank.time_rank('uint')
      10.8±0.09ms       10.8±0.8ms     1.01  series_methods.Rank.time_rank('float')
       8.10±0.2ms       8.26±0.3ms     1.02  series_methods.Rank.time_rank('int')
         51.3±2ms         51.2±1ms     1.00  series_methods.Rank.time_rank('object')
       8.76±0.6ms       8.30±0.2ms     0.95  series_methods.Rank.time_rank('uint')
```

</details>

Labels:
name: Bug
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
thanks @mzeitlin11 

keep em coming on closing older issues!