BrandonFors/pandas-github-issues huggingface dataset:
CLN: remove CategoricalBlock.to_native_types
State: closed

The implementation of `to_native_types` on CategoricalBlock seems duplicated with the one of ExtensionBlock (the default for `na_rep` is different, but internally we always pass a value for that and never use the default).

Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Clean

Comments:
> The implementation of `to_native_types` on CategoricalBlock seems duplicated with the one of ExtensionBlock

is `np.array(values, dtype="object")` equivalent to `values.astype(str)` for Categorical?

> (the default for `na_rep` is different, but internally we always pass a value for that and never use the default)

also could preserve the signature and call super or would we not expect CategoricalBlock to be used by downstream projects?


> is `np.array(values, dtype="object")` equivalent to `values.astype(str)` for Categorical?

@jorisvandenbossche you can ignore that, I was on an older commit. it's now `np.array(values, dtype="object")` vs `np.asarray(values.astype(object))`
> also could preserve the signature and call super or would we not expect CategoricalBlock to be used by downstream projects?

I don't expect anybody to use `to_native_types` (downstream projects use blocks, but typically for construction or accessing the values block-wise).
thanks
ERR: Raise NotImplementedError with BaseIndexer and certain rolling operations
State: closed

- [x] xref #32865
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

As a temporary solution for returning incorrect results when using a BaseIndexer subclass with certain rolling operations, a `NotImplementedError` will now raise with instructions to use `apply` instead.

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Window
description: rolling, ewma, expanding

Comments:
Hello @mroeschke! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-03-28 20:52:23 UTC
thanks @mroeschke
[WIP] Make Index.values read-only
State: closed

- [ ] closes #33001
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This is a PR for looking around enforcing immutability of `pd.Index` objects. It will be a draft, but the main idea is to show what tests fail and what would need to change for this to be implemented. Because of this, I'm going to add some commits incrementally, so there is a history in the build logs.

Labels:

Comments:
@ivirshup are you still interested in working on this?
I've gotten a little stuck on the cython part. Using the cython 3.0 alpha, the errors about using a `const` fused type go away, but are replaced by parse errors of the generated c code. This seems to only be the case if the types are imported from numpy.

I think it could make sense for someone who has more experience with cython to take over, or at least figure out that part. I'm still interested, but I'm not sure I have the time to learn that much about cython at the moment.
see if you can merge master and get this passing
@jreback, IIRC this requires cython 3. Since master can't compile with cython 3 (https://github.com/pandas-dev/pandas/issues/34014), I think this is stuck.
closing as inactive, we can reopen if you like after cy3 is available.
CLN: remove BlockManager.get
State: closed

This should not be merged until after #33047 and #33045, since the assertions made here will not always hold until then.

Labels:
name: Refactor
description: Internal refactoring of code
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Internals
description: Related to non-user accessible pandas implementation
name: Clean

Comments:
Looks good - any perf implications?
> any perf implications?

_getitem_multilevel calling _ixs instead of _get_item_cache is slightly higher-level, and we lost a little ground on the columns.is_unique assertions, then make up some ground with unnecessary checks in BlockManager.get being removed.  It's about a wash.
Updated so as to not be blocked by #33047.
BUG: DataFrame.at with non-unique axes
State: closed

- [x] closes #33041
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Closes #33041 in conjunction with #33045.

Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
i think consensus on the call was to move forward with this?
gentle ping
> i think consensus on the call was to move forward with this?

Yes, as also summarized on the issue (#33153)

gentle ping
Thanks! 

(BTW, if you have 2 approvals and no others commenting/requesting changes, I think it is fine to merge yourself)
> BTW, if you have 2 approvals and no others commenting/requesting changes, I think it is fine to merge yourself

Good rule of thumb, thanks
BUG: frame.lookup with non-unique axes
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

xref #33041, one more PR coming up to close that.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
thanks
Bug: TypeError: ufunc 'sqrt' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
State: closed

- [x] closes #32904 
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
@gkaku pls don't open up multiple PRs; if you have a patch, just push to the *same* one
There's also a PR for this at https://github.com/pandas-dev/pandas/pull/32937.
closing in favor of #32937
PERF: remove large-array-creating path in fast_xs
State: closed

When frame.columns is non-unique, `frame.iloc[n]` goes through an unnecessary path that effectively creates `frame.values` and looking up `[n]` on that.  That's a lot of casting to access just one row.

Luckily, that case is obsolete, so this rips it right out.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Performance
description: Memory or execution speed performance

Comments:
this *only* hits the non-unique case I think. do we have any benchmarks?
Just added a benchmark:

```
In [3]: arr = np.arange(10**7).reshape(-1, 10) 
In [4]: df = pd.DataFrame(arr)
In [5]: dtypes = ['u1', 'u2', 'u4', 'u8', 'i1', 'i2', 'i4', 'i8', 'f8', 'f4']                                                                              
In [6]: for i, d in enumerate(dtypes): 
   ...:         df[i] = df[i].astype(d) 

In [8]: %timeit df.iloc[10000]                                                                                                                             
126 µs ± 1.29 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)  # <-- both

In [9]: df.columns = ["A", "A"] + list(df.columns[2:])     
                                                                                                
In [11]: %timeit df.iloc[10000]                                                                                                                            
17.5 ms ± 161 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)   # <-- master
124 µs ± 1.21 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)   # <-- PR

```
thanks
DOC: Fixing links to the contributing page
State: closed

Not 100% sure on this changes.

---

Feel free to close this at anytime.

Labels:
name: Docs

Comments:
@MomIsBestFriend the PR fixing the deployment of the dev docs has been merged. There is a problem in statmodels breaking the docs build atm, but the dev docs should be available at  https://pandas.pydata.org/docs/dev/ very soon. If you want to update the links here, that would be great. Thanks!
Thanks @MomIsBestFriend
CLN: de-duplicate Block.should_store and related
State: closed

FooBlock.should_store is equivalent to `is_dtype_equal(self.dtype, other.dtype)` for almost all of our Block subclasses.  This makes that the implementation for the base class, so we define it in fewer places.  xref #32878 on remaining cases.

Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Clean

Comments:
wow, cleans it up
@jreback thoughts on using the new default should_store implementation (return is_dtype_equal(self.dtype, other.dtype)` for the remaining places that special case it (ExtensionBlock, ComplexBlock)?  For ExtensionBlock im pretty sure what we have now is just wrong, while for ComplexBlock it becomes equivalent to the new default if we were to add an itemsize check, which seems reasonable
> @jreback thoughts on using the new default should_store implementation (return is_dtype_equal(self.dtype, other.dtype)` for the remaining places that special case it (ExtensionBlock, ComplexBlock)? For ExtensionBlock im pretty sure what we have now is just wrong, while for ComplexBlock it becomes equivalent to the new default if we were to add an itemsize check, which seems reasonable

yes i think you can
OK next pass
Thanks!
ERR: Raise a better error for numpy singletons in Index
State: closed

- [x] closes #33017
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Index
description: Related to the Index class or subclasses

Comments:
lgtm - can you add a whatsnew?
thanks @dsaxton 
Many thanks! That was a quick fix! :)
Added RuntimeWarning to lib.fast_unique_multiple
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: API - Consistency
description: Internal Consistency of API/Behavior

Comments:
Looks like at ```L340```, it triggers the warning, but only at the AP ```Linux py37_np_dev```

https://github.com/pandas-dev/pandas/blob/42ef40971261b286b53ddfc85d2e399067b7c72d/pandas/tests/indexes/multi/test_setops.py#L334-L345

---

@jbrockmendel Does having something like this:

```python
from pandas.compat.numpy import _is_numpy_dev

...

if _is_numpy_dev:
    with tm.assert_produces_warning(RuntimeWarning):
        result = idx.union(idx[:1]) 
else:
    result = idx.union(idx[:1]) 
```

is the correct approach?
Usually we would write

```
warn = RuntimeWarning if _is_numpy_dev else None
with tm.assert_produces_warning(warn):
    ...
```
So apparently I never needed the ```_is_numpy_dev```. \*facepalm\*
ping @jreback 
thanks @MomIsBestFriend
add match to bare pyteset.raises() - arrays
State: closed

- [x] ref #30999
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
>     * [x]  closes #30999
> 
>     * [x]  tests added / passed
> 
>     * [x]  passes `black pandas`
> 
>     * [x]  passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
> 
>     * [ ]  whatsnew entry

@sathyz Can you please change the word "closes" to the word "ref"?
thanks @sathyz
STY: Boolean values for bint variables
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Same comment as #33008 - a lot of the Cython stuff is performance critical so really want to be sure
can you just verify that the same code is generated here as well, otherwise lgtm.
@jreback I checked and the generated C code, is identical. (Only diff is in the comments)
thanks @MomIsBestFriend 

I suppose if we had a code check to catch this would be great, but might not be so easy to check
STY: Boolean values for bint variables
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Looks fine.  Any reason to suspect a performance impact?  (for this one id look at the generated C code and see if it suddenly became more verbose)
> Looks fine. Any reason to suspect a performance impact? (for this one id look at the generated C code and see if it suddenly became more verbose)

I looked a the generated C-code, there is no difference (except for the comments that show the equivalent code at the ```pyx``` file), this change is purely a stylistic change.
> I looked a the generated C-code, there is no difference (except for the comments that show the equivalent code at the `pyx` file), this change is purely a stylistic change.

Can you run benchmarks to be sure? 
> > I looked a the generated C-code, there is no difference (except for the comments that show the equivalent code at the `pyx` file), this change is purely a stylistic change.
> 
> Can you run benchmarks to be sure?

Sure!
@WillAyd Can you please guide to which asv benchmarks you'd like to see for this PR?
also you can run `cython -a` to generate the annotate code to see if there is any difference (alt just recompile); if the c-code is the same then we are good. 
> also you can run `cython -a` to generate the annotate code to see if there is any difference (alt just recompile); if the c-code is the same then we are good.

@jreback I have done both, and the result are the same.

---

I also did a small POC, what I did was:

Created  a file called ```foo.pyx``` and it's contents:

```cython
def f(bint VAR_NAME=0):
    pass
```

then ran:

```bash
cython -3 foo.pyx
mv foo.c foo_ZERO.c   # So future diff won't have diffs related to the file name.
rm -f foo.*                     # Was afraid that cython uses same file name as cache or something crazy like that
```

then I rewrote ```foo.pyx```, and it's contents:

```cython
def f(bint VAR_NAME=False):
    pass
```

and then I ran:

```bash
cython -3 foo.pyx
mv foo.c foo_FALSE.c
```

and then I checked via ```vimdiff```:

```bash
vimdiff foo_ZERO.c foo_FALSE.c
```

![generated_c_code](https://user-images.githubusercontent.com/50263213/77597561-ebd0b800-6f07-11ea-990d-afbe9cc227cc.png)

---

I have also included the two files generated in the POC

[foo.zip](https://github.com/pandas-dev/pandas/files/4384372/foo.zip)


Thanks!
BUG: query uses numexpr even when 'compute.use_numexpr' is False
State: closed

- [ ] closes #32556
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: expressions
description: pd.eval, query

Comments:
If i understand correctly, the issue #32556 is about df.query does not care about the pd.set_option option, not about the extension dtype (which is still open at #25369)
@teto used ```pd.set_option('compute.use_numexpr', False)``` as a workaround for the original dtype issue but turned out it is also a bug on its own.
@quangngd  that's an accurate description of the issue. Thanks for taking care of this !
@quangngd you need to have a test which fails on master but works here. can you construct one.
there is one in my issue (might not be usable out of the box though).
@teto I don't think it's a good idea using a bug as a test as it would be fixed in the future.
So to my understanding, the test must be able to differentiate the used engine during the ```df.query```/ ```df.eval``` call (yah, just found out the bug is in ```df.eval``` too, and is too fixed by this pr). The thing is we could't use any "non-parse-able"  behaviour since it could be a bug and could be fixed in the future.
The best I could come up with is the following based on [numexpr' note](https://numexpr.readthedocs.io/en/latest/user_guide.html#notes).
```py
import pandas as pd
df = pd.DataFrame(data=[4+1j,5,6], columns=["col_test"])
s = "abs(col_test)"
display(df.eval(s, engine='python').dtype)
display(df.eval(s, engine='numexpr').dtype)
```
output:
```py
dtype('float64')
dtype('complex128')
```
But so far I can't think of how to make a test for ```df.query```
@quangngd I am not sure of the status here. please merge master and address comments.
@jreback the fix is already there but I cannot think of a sensible test
@quangngd: What about this?

    from pandas.core.computation.eval import _check_engine
    with pd.option_context('compute.use_numexpr', False):
        result = _check_engine(None)
        assert result == 'python'
@rhshadrach thankyou, the test indeed fails on master and pass on my branch. Where should be the proper place to place the test?
> Where should be the proper place to place the test?

`tests.computation.test_eval`
closing, this needs a reproducible tests that fails before any patch. (it can certainly be part of the patch, but demonstrate on the issue)
wait what ? it's not easy to reproduce, you would need to disable one package or monkeypatch one package I suppose. The absence of a test should not prevent fixing an issue/document it/remove the feature.
w/o a test how do you know it was fixed?
Is this test not sufficient?

https://github.com/pandas-dev/pandas/pull/33006#issuecomment-654424734
sure if it fails on master
STY: Boolean values for bint variables
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

---

```asv``` benchmarks: to show that performance is not changed with the stylistic changes)

```
All benchmarks:

       before           after         ratio
     [28e0f18a]       [6f1fd78d]
     <master>         <STY-bint-boolean-not-int-join>
         21.4±1ms       19.8±0.4ms     0.92  join_merge.Join.time_join_dataframe_index_multi(False)
       24.9±0.3ms       23.6±0.3ms     0.95  join_merge.Join.time_join_dataframe_index_multi(True)
       14.5±0.3ms       14.0±0.3ms     0.97  join_merge.Join.time_join_dataframe_index_shuffle_key_bigger_sort(False)
       17.6±0.6ms       16.7±0.5ms     0.95  join_merge.Join.time_join_dataframe_index_shuffle_key_bigger_sort(True)
       14.4±0.3ms       13.8±0.2ms     0.95  join_merge.Join.time_join_dataframe_index_single_key_bigger(False)
       17.8±0.7ms       16.4±0.5ms     0.92  join_merge.Join.time_join_dataframe_index_single_key_bigger(True)
       13.5±0.3ms       13.1±0.3ms     0.97  join_merge.Join.time_join_dataframe_index_single_key_small(False)
       14.5±0.1ms       13.9±0.5ms     0.96  join_merge.Join.time_join_dataframe_index_single_key_small(True)
       2.86±0.08s       2.94±0.05s     1.03  join_merge.JoinIndex.time_left_outer_join_index
         433±70μs         428±50μs     0.99  join_merge.JoinNonUnique.time_join_non_unique_equal
```

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
I am not sure asv is always appropriate for changes that might only give a small change in performance. Can you maybe check the generated code to ensure that no more work is being done (unless this is documented in the cython docs to work fine?)
> I am not sure asv is always appropriate for changes that might only give a small change in performance. Can you maybe check the generated code to ensure that no more work is being done (unless this is documented in the cython docs to work fine?)

@jorisvandenbossche I have checked the sources of the generated C-files. there is no code change at all (except for the comments displaying the equivalent code at the ```.pyx``` file), this is purely stylistic change.
Super, thanks for checking!
Hah, I just made the same suggestion in the next PR
Thanks @MomIsBestFriend
DOC: Fix PR01, PR02, RT03 on cumulative function
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Related to #27977.

Labels:
name: Docs

Comments:
Thanks
WIP/PERF: block-wise ops for frame-with-series axis=1
State: closed

xref #32779 which does the same thing for frame-with-frame ops.  This one is turning out to be the most difficult of the cases to get right.

Posting this for comments, in large part because there are still two failing tests locally, and these may just be behaviors we want to change.

```
# test_scalar_na_logical_ops_corners
s = Series([2, 3, 4, 5, 6, 7, 8, 9, datetime(2005, 1, 1)])
s[::2] = np.nan
d = DataFrame({"A": s})

with pytest.raises(TypeError):
    d.__and__(s, axis="columns")   # <-- when done block-wise, this does not raise
```

The other one is harder to post an example of because it uses fixtures making it a PITA to copy/paste from the test, but the upshot is that `op(frame[mixed-float-dtypes], series[float64])` downcasts to the original dtypes ATM (since each column is effectively operating against a scalar) but when done blockwise they all come back float64.

Labels:
name: Performance
description: Memory or execution speed performance
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
mothballing to clear the queue
BUG: DataFrame.diff(axis=1) with mixed (or EA) dtypes
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
> Can we instead skip doing things blockwise for axis=1 and do it columnwise instead?

That would also allow us to avoid consolidating, which would be nice.
updated to operate column-wise.

DatetimeTZBlock.diff could have its NotImplementedError case removed in a follow-up
Do we have anything approaching consensus here?  I'm OK with either column-wise or transpose
I'm pretty strongly in favor of columnwise when there's more than 1 block. While I don't think sparse support should ever be a primary motivator, the difference in performance of a columnwise diff vs. a `.T.diff().T` for something like

```python
In [6]: a = pd.arrays.SparseArray([1] + [0] * 1000)

In [7]: df = pd.DataFrame({"A": a, "B": a})
```

will be huge.
@jreback is column-wise vs transpose a deal breaker?  if not, i think this is ready
if you really want to do columnwise can you move impl to another location as indicated

the code is pretty complex and should be separated from inline in the function 
thanks, IIRC there *might* be some issues this closes if you'd have a look