BrandonFors/pandas-github-issues huggingface dataset:
BUG: fix IntegerArray astype with copy=True/False
State: closed

xref https://github.com/pandas-dev/pandas/pull/34307#discussion_r443370993

Right now, we were not copying the mask correctly with `copy=True`, and also not when the data was actually copied (then the mask also needs to be copied, even with `copy=False`)

Labels:
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
One additional API question that pops up: in numpy, `astype(.., copy=False)` seems to return itself if it doesn't need to copy (I am actually also relying on this in the `if data is self._data` check).  
But in the EA implementation, we always return a new array *object*, potentially with the same backing data. But we could in principle also do something like `if dtype == self.dtype and not copy: return self` (and that should already catch the one case where a copy can be avoided I think, and we could simply always copy in the other cases)
> Does this need a whatsnew, or is it only on master?

Not only master it seems, will add a whatsnew.
I don't have a strong opinion on returning a new object backed by the data vs. just returning `self`.
Any preferences on what I do with `copy=False`? (returning a new object vs returning self) 
Matching NumPy (returning the same object) seems OK.
Updated to return same object
lgtm, does this need a whatsnew? e.g. changed in 1.1?
Added a bug fix note
thanks @jorisvandenbossche
CLN: Update Cython data pointers for rolling apply
State: closed

- [x] xref #34014
- [x] tests passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Performance looks pretty comparable between master and this PR

```
N = 10 ** 3
arr = 100 * np.random.random(N)
roll = pd.DataFrame(arr).rolling(10)
%timeit roll.apply(lambda x: np.sum(x) + 5, raw=True)

4.48 ms ± 14.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) <--PR
4.34 ms ± 29.6 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) <--master
```

Labels:
name: Clean
name: Window
description: rolling, ewma, expanding

Comments:
cc @WillAyd & @jbrockmendel since I think you have some experience here.

It's good to see that performance is comparable.
This looks great! I assume memory impact is roughly the same? Also do you know if this will work in reductions.pyx as well?
LGTM.  

> Also do you know if this will work in reductions.pyx as well?

My understanding is that these are pretty different usages.  #34009 addresses one of the usages in reductions.pyx; local attempts to address the other usages are proving more troublesome.
>  I assume memory impact is roughly the same?

My ASVs are botched, but memit shows comparable memusage. Looks like this pointer dance was to avoid copying data, and I think the slicing in this PR should be operating on views.

```
N = 10 ** 3
arr = 100 * np.random.random(N)
roll = pd.DataFrame(arr).rolling(10)
%memit roll.apply(lambda x: np.sum(x) + 5, raw=True)

peak memory: 74.94 MiB, increment: 0.00 MiB <--PR

peak memory: 74.72 MiB, increment: 0.53 MiB <--master
```
Alrighty, thanks!
Fix DataFrame/Series stack/unstack docs
State: closed

- [x] closes #21675
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

As discussed in the issue, the sorted level is an implementation detail that does not always happen. Thus I think we should remove the misleading documentation.

Labels:
name: Docs

Comments:
Great thanks for the cleanup @pdnm
DOC: Fix language style
State: closed

- Ref: https://9to5google.com/2020/06/12/google-android-chrome-blacklist-blocklist-more-inclusive/
- Ref: https://android-review.googlesource.com/c/platform/frameworks/ml/+/970739

cc @pandas-dev/pandas-core 

Labels:
name: Docs
name: Clean

Comments:
Thanks for doing this 
Lgtm
LGTM. These are all internal, so no need for a deprecation correct?

Thoughts on adding a linting rule to catch these in the future?
> LGTM. These are all internal, so no need for a deprecation correct?
> 
> Thoughts on adding a linting rule to catch these in the future?

Yep all internals. Good shout if ppl think a lint check is helpful here i'm happy to add.

P.S Any further thoughts/further ideas in this area welcomed.
Thanks @alimcmaster1 
> Thanks @alimcmaster1

Thanks @WillAyd
TST : Added test for creating empty dataframe with column of type str…
State: closed

- [x] closes #34915
- [x] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Strings
description: String extension data type and string data
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
For now I have placed the test in ```tests/frame/test_constructors.py``` but maybe we don't want tests for extension dtypes in that file, in which case i can move it elsewhere.
lgtm - @TomAugspurger any thoughts on location of test before merge?
This looks good, thanks.
TST: Verify whether non writable numpy array is shiftable (21049)
State: closed

- [x ] closes #21049
- [x ] tests added / passed
- [x ] passes `black pandas`
- [x ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks @avinashpancham
BUG: DataFrame.(any|all) inconsistency
State: closed


Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Clean
name: Stale

Comments:
AFAICT there are 3 concerns to be balanced here:

1) internal consistency w/r/t sub-frames, i.e `df[subset].all(bool_only=True)` should never be _larger_ than `df.all(bool_only=True)` (bug in master)
2) independence of consolidation status (in the status quo we consolidate, so this is not an issue)
3) backwards compatibility

I see two possible ways to accomplish get 1) consistent with 2):

a) drop object-dtype blocks in `get_bool_data` (i.e. what this PR does ATM).  This will produce some cases where `df.all(bool_only=True)` is smaller than it is in master.
b) operate column-wise on object-dtype blocks.  This will produce some cases where `df.all(bool_only=True)` is larger than it is in master.

I lean towards a) because it is simpler to implement and describe/document.  Since it is fixing a bug in a corner case, I don't think a deprecation cycle is needed
I am also in favor of your option a). Although b) is technically possible (and something we would also get with 1D blocks), I think long term we should simply not regard object dtype columns as boolean or infer if they might be boolean (the same goes for indexing with object dtype bools). 

But, I still think we can deprecate this first instead of directly changing. Yes, it's quite a specific case, but it doesn't seem that complicated to deprecate? (the logic is only in `_get_bool_data`?)
> (the logic is only in _get_bool_data?)

AFAIK the only things affected are `DataFrame.all(bool_only=True)` and `DataFrame.any(bool_only=True)`

> But, I still think we can deprecate this first instead of directly changing. Yes, it's quite a specific case, but it doesn't seem that complicated to deprecate? 

The corner-ness of it makes me not care _that_ much about deprecate vs change.  I marginally lean towards getting it over with because there is a bug fix involved and it will make it easier to simplify `DataFrame._reduce`.
Went through the issues and added Reduction and Nuisance Column labels where appropriate.  Reinforced my belief that numeric_only needs to be thrown into the sun.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
rebased.  i maintain this bug merits ripping off the bandaid.
BUG: indexing regression with datetime index
State: closed

The bug is already fixed on master. I have just added a unit test.

- [x] closes #34860
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Datetime
description: Datetime data dtype
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
Hello @ketanarlulkar! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-06-24 06:31:47 UTC
thanks @ketanarlulkar
CLN: GH29547 format with f-strings
State: closed

xref #29547 
replace .format() for f-strings in the following:

- pandas/tests/series/indexing/test_numeric.py


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Hello @DanBasson! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-06-21 17:03:13 UTC
hi @jreback, 

do you understand the error ?
any help will be appriciated.
CI is successful.
REF: dont use compute_reduction
State: closed

Full asv run shows no change.

This gets rid of one usage of libreduction.compute_reduction; #34909 gets rid of the other.

Labels:
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
any perf impact?
> any perf impact?

ran a full asv run and it said results were unchanged.
@jbrockmendel this closed https://github.com/pandas-dev/pandas/issues/34506 ?
> this closed #34506 ?

Looks like it addresses the example in the OP, not sure if it solves the general case.  Needs a test before closing the issue.
ENH: specificy missing labels in loc calls GH34272
State: closed

- [ ] closes #34272
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
This should address #34272. 

Would it be useful to add max limit to the number of missing labels it would print? I think this will be handled better by the __str__ method of the Series class, already.
thanks @timhunderwood very nice
thanks, Happy to help!
DOC: explain EWM 
State: closed

- [ ] closes #34867
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Clean
name: Window
description: rolling, ewma, expanding

Comments:
Hello @KrishnaSai2020! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:











There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-06-20 20:06:07 UTC
@charlesdong1991 believe this is done please have a look when you can.
thanks @KrishnaSai2020 
@jreback np and @charlesdong1991 thanks for your help!
PERF: avoid creating many Series in apply_standard
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This avoids going through perennial-problem-causing libreduction code (xref #34014, #34080) and instead does the same trick in python-space by re-assigning block.values instead of creating new Series objects.

If we avoid libreduction but _dont_ do this optimization, the most-affected asv is `time_apply_ref_by_name` that clocks in at 6.92x slower.  This achieves parity on that asv.


ATM I'm still getting 4 test failures locally, need to troubleshoot.
 Update: passing



Labels:
name: Performance
description: Memory or execution speed performance
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
does this fix #34506 ?
> does this fix #34506 ?

The example from the OP there was fixed by #34913, but i still need to double-check the general case and add a test.
The groupby.ops analogue of this is continuing to be a PITA, so im exploring other avenues there (xred #34982).  This should be good to go though, and as a follow-up we can rip out libreduction.Reducer.
thanks, yeah incremental refactors are good. 
This commit caused some regressions, eg https://pandas.pydata.org/speed/pandas/#frame_methods.Apply.time_apply_np_mean,
https://pandas.pydata.org/speed/pandas/#frame_methods.Apply.time_apply_ref_by_name?x-axis-scale=date

Was that expected? (above you mention "achieves parity on that asv.")
Ah, I suppose this is just fixed by https://github.com/pandas-dev/pandas/pull/35166
BUG: incorrect type when indexing sparse dataframe with iterable
State: closed

closes #34526
closes #34540 
- [x] 2 tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

The problem arose when indexing with an iterable.  If the result consisted of columns which originally only had sparse values, the `dtype` was solely inferred from the `fill_value`, which defaults to `0`.  Hence, the resulting columns have the dtype ` Sparse[int64, 0]`.   This commit changes the type inference logic to use the numpy type promotion rules between the underlying subtype of the `SparseArray.dtype` and the type of the fill value.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Sparse
description: Sparse Data Type

Comments:
@jreback Thanks for the review!  I'll try fix the bug in the next few days.
For the type inference logic, I wasn't sure which to use, `SparseArray.dtype.subtype` or the sparse values themselves (`SparseArray.sp_values`).  I went with the former because it seemed conceptually cleaner.
Hi @jreback, just bringing this to your attention. I wasn't sure if you noticed it's ready for review (I used the "request review" option in the side panel).  Apologies if this is already on your list.
@suvayu can you add a whatsnew to 1.1.0.rst in the bug fixes section?
Hi @TomAugspurger, I added the the whatsnew entry.  There was a merge conflict with master (another entry in the same section), so I had to force push after resolving it.
Thanks @suvayu!
TST: pandas/test/window/ changes for #30999
State: closed

- [ ] xref  #30999 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

I've made changes for:
- tests/window/moments/test_moments_rolling.py
- tests/window/test_dtypes.py
- tests/window/test_ewm.py
- tests/window/test_expanding.py
- tests/window/test_timeseries_windows.py

In a couple of places there were comments stating the raised error's output. The `msg` variable contains the same information. Should I remove these coments?

Belongs to issue: https://github.com/pandas-dev/pandas/issues/30999

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Window
description: rolling, ewma, expanding

Comments:
lgtm. ping on green.
looks like a code check failed if u can address
pls ping on green 
@jreback the checks came back green :)
Thanks @boweyism
BUG: Groupby with as_index=True causes incorrect summarization
State: closed

- [x] closes #26321 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby

Comments:
Given that this is my first commit to pandas, I'm not quite sure whether what I'm doing is what is expected. I just added the example from the bugreport as a test, is that ok? Should additional things be tested? Tested differently? Please, let me know.
I addressed the comments, could you please have a look again?
I don't know why there suddenly are test failures, the particular test that failed (pandas/tests/base/test_unique.py::test_unique_bad_unicode), passes on my setup and is nowhere near the test that I added.
@vivikelapoutre can you try merging master & repushing to see if that fixes it?
> 
> 
> @vivikelapoutre can you try merging master & repushing to see if that fixes it?

Thank you, that worked indeed.
Thanks @vivikelapoutre!
BUG: plotting layout patch
State: closed

This fixes a layout problem.  


Labels:
name: Visualization
description: plotting

Comments:
hmm can you show the before / after. also please add a whatsnew note in plotting / bug fixes / 1.1.
![before](https://user-images.githubusercontent.com/15750539/85206000-eae8d000-b316-11ea-8bd1-cfa7199e03be.png)
![after](https://user-images.githubusercontent.com/15750539/85206006-efad8400-b316-11ea-9ea8-38584c5d5b13.png)


kk great, pls add a whatsnew (e.g. when a user would care about this) and ping on green.
thanks @lyashevska
TST:add test for df replace GH34871
State: closed

- [x] improves upon #34871
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry <---- is that helpful for this specific regression test? happy to write this if you think this helps! :)

add tiny regression test for not failing on calling df.replace() with Period column

working on this revealed other quirks of df.replace(): it now does not raise an exception anymore, and the dataframe returned by replace() has the right values. However, the dtype of a Period column changes to Object, which is surprising. 

I've added the observations to GH34871

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Period
description: Period data type

Comments:
thanks @nwweber 

happy for you to tackle https://github.com/pandas-dev/pandas/issues/34871 :->
> thanks @nwweber
> 
> happy for you to tackle #34871 :->

i'll have a look over the weekend :)
Replace old string formatting syntax with f-strings #29547
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Hello @ynorouzz! Thanks for opening this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/tests/series/indexing/test_where.py`](https://github.com/pandas-dev/pandas/blob/f993356973fcd24cdfa6934c6baef36fd2ff7a21/pandas/tests/series/indexing/test_where.py):

> [Line 226:89](https://github.com/pandas-dev/pandas/blob/f993356973fcd24cdfa6934c6baef36fd2ff7a21/pandas/tests/series/indexing/test_where.py#L226): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (92 > 88 characters)


there are some linting issues
see https://github.com/pandas-dev/pandas/pull/34903/checks?check_run_id=790893140
@ynorouzz is this still active? Can you address the comments / failures?
closing since these were changed to f-strings in #34502 which has been merged, hence the conflict.
DOC: explain EWM
State: closed

- [ ] closes #34867
- [ ] tests  passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew - Made docs more clear and renamed files


Labels:
name: Docs
name: Window
description: rolling, ewma, expanding

Comments:
Hello @KrishnaSai2020! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-06-20 14:03:50 UTC
right I'll fix that pep 8 issue and reopen this pr
thanks for the PR @KrishnaSai2020 

can you pls fix it though? Not sure about what the previous contributor did, but unlikely his/her PR got merged with the PEP8 issue. 

Looks like your expanded `EM` to `Exponential Weighted` caused it.
Ah I see! should i close this, fix and reopen or is there an easier way to edit the PR??
you can fix it in this PR, no need to close it and reopen a new one, thanks!
@charlesdong1991  fixed the pep 8 problem
I _think_ what was meant by @jreback is to rename `EWM` to `ExponentialMoving` or `ExpontentialMovingWindow` other than to `EWF`.

so you might also need to replace all your `EWF` to the new class name you choose.
@charlesdong1991 I think that's all instances refactored let me know if there more.
https://github.com/pandas-dev/pandas/pull/34910
DOC: add note about the values of unit for pd.to_datetime
State: closed

The `unit` option for `pandas.to_datetime` takes different format strings than the `format` option,
which makes sense, but caught me off-guard as a new user. It would be helpful if the documentation
mentioned this.

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs
name: Datetime
description: Datetime data dtype

Comments:
k looks reasonable
thanks @aidanmontare-edu 
No problem!

On Sat, Jun 20, 2020 at 4:12 PM Jeff Reback <notifications@github.com>
wrote:

> thanks @aidanmontare-edu <https://github.com/aidanmontare-edu>
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/34899#issuecomment-647040616>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ALMCGOPQ6H33O473MN6JGALRXUJZTANCNFSM4ODMJOLA>
> .
>


-- 
Sincerely,

Aidan Montare
CWRU Class of 2021
TST: groupby apply called multiple times
State: closed

closes #31111 
xref https://github.com/pandas-dev/pandas/pull/24748
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ Added test for testing func should be called len(set(groupby_column)) times when passed to groupby.apply(func) ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Hello @Rohith295! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-06-20 14:58:22 UTC
this is very similar to these issues
GH2936, GH7739, GH10519, GH2656, GH12155, GH20084, GH21417
which are now fixed
lgtm ping on green.
thanks @Rohith295